Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 827386 Details for
Bug 1032598
Unable to activate 6.3 RHEV-Hypervisor after upgrading to RHEV-Manager 3.2
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
vdsm.log
vdsm.log (text/plain), 18.64 MB, created by
Pratik Pravin Bandarkar
on 2013-11-21 17:20:50 UTC
(
hide
)
Description:
vdsm.log
Filename:
MIME Type:
Creator:
Pratik Pravin Bandarkar
Created:
2013-11-21 17:20:50 UTC
Size:
18.64 MB
patch
obsolete
>Dummy-15909965::DEBUG::2013-11-19 21:01:01,753::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.04418 s, 23.2 MB/s\n'; <rc> = 0 >Thread-15909988::DEBUG::2013-11-19 21:01:02,171::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15909988::INFO::2013-11-19 21:01:02,172::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15909988::DEBUG::2013-11-19 21:01:02,173::task::495::TaskManager.Task::(_debug) Task 993b65de-5d01-4045-a3d2-4b44e46b13a9: moving from state init -> state preparing >Thread-15909988::DEBUG::2013-11-19 21:01:02,173::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15909988::DEBUG::2013-11-19 21:01:02,173::task::495::TaskManager.Task::(_debug) Task 993b65de-5d01-4045-a3d2-4b44e46b13a9: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15909988::DEBUG::2013-11-19 21:01:02,174::task::495::TaskManager.Task::(_debug) Task 993b65de-5d01-4045-a3d2-4b44e46b13a9: moving from state preparing -> state finished >Thread-15909988::DEBUG::2013-11-19 21:01:02,175::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15909988::DEBUG::2013-11-19 21:01:02,175::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15909988::DEBUG::2013-11-19 21:01:02,175::task::495::TaskManager.Task::(_debug) Task 993b65de-5d01-4045-a3d2-4b44e46b13a9: ref 0 aborting False >Thread-15909988::INFO::2013-11-19 21:01:02,176::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15909989::DEBUG::2013-11-19 21:01:02,181::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15909989::INFO::2013-11-19 21:01:02,183::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15909989::DEBUG::2013-11-19 21:01:02,184::task::495::TaskManager.Task::(_debug) Task 613006dc-c03a-4bb1-a590-7c175598d210: moving from state init -> state preparing >Thread-15909989::DEBUG::2013-11-19 21:01:02,184::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`13a1dfd1-3f3a-442b-8cdc-a2c8668a7cf1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15909989::DEBUG::2013-11-19 21:01:02,185::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15909989::DEBUG::2013-11-19 21:01:02,185::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15909989::DEBUG::2013-11-19 21:01:02,186::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`13a1dfd1-3f3a-442b-8cdc-a2c8668a7cf1`::Granted request >Thread-15909989::DEBUG::2013-11-19 21:01:02,186::task::495::TaskManager.Task::(_debug) Task 613006dc-c03a-4bb1-a590-7c175598d210: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15909989::DEBUG::2013-11-19 21:01:02,187::task::495::TaskManager.Task::(_debug) Task 613006dc-c03a-4bb1-a590-7c175598d210: ref 1 aborting False >Thread-15909989::WARNING::2013-11-19 21:01:02,188::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15909989::DEBUG::2013-11-19 21:01:02,189::task::495::TaskManager.Task::(_debug) Task 613006dc-c03a-4bb1-a590-7c175598d210: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15909989::DEBUG::2013-11-19 21:01:02,189::task::495::TaskManager.Task::(_debug) Task 613006dc-c03a-4bb1-a590-7c175598d210: moving from state preparing -> state finished >Thread-15909989::DEBUG::2013-11-19 21:01:02,189::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15909989::DEBUG::2013-11-19 21:01:02,190::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15909989::DEBUG::2013-11-19 21:01:02,190::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15909989::DEBUG::2013-11-19 21:01:02,191::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15909989::DEBUG::2013-11-19 21:01:02,191::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15909989::DEBUG::2013-11-19 21:01:02,191::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15909989::DEBUG::2013-11-19 21:01:02,191::task::495::TaskManager.Task::(_debug) Task 613006dc-c03a-4bb1-a590-7c175598d210: ref 0 aborting False >Thread-15909989::INFO::2013-11-19 21:01:02,192::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15909990::INFO::2013-11-19 21:01:02,482::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15909990::DEBUG::2013-11-19 21:01:02,483::task::495::TaskManager.Task::(_debug) Task 6c8b3899-81bc-4fdc-ba48-48e8cdb5b30b: moving from state init -> state preparing >Thread-15909990::DEBUG::2013-11-19 21:01:02,483::task::495::TaskManager.Task::(_debug) Task 6c8b3899-81bc-4fdc-ba48-48e8cdb5b30b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000282049179077', 'lastCheck': 1384894855.904762, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00129199028015', 'lastCheck': 1384894852.394912, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894857.41732, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000365018844604', 'lastCheck': 1384894854.3821671, 'valid': True, 'code': 0}} >Thread-15909990::DEBUG::2013-11-19 21:01:02,484::task::495::TaskManager.Task::(_debug) Task 6c8b3899-81bc-4fdc-ba48-48e8cdb5b30b: moving from state preparing -> state finished >Thread-15909990::DEBUG::2013-11-19 21:01:02,484::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15909990::DEBUG::2013-11-19 21:01:02,485::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15909990::DEBUG::2013-11-19 21:01:02,485::task::495::TaskManager.Task::(_debug) Task 6c8b3899-81bc-4fdc-ba48-48e8cdb5b30b: ref 0 aborting False >Thread-15909990::INFO::2013-11-19 21:01:02,485::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000282049179077', 'lastCheck': 1384894855.904762, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00129199028015', 'lastCheck': 1384894852.394912, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894857.41732, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000365018844604', 'lastCheck': 1384894854.3821671, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:01:03,764::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:01:04,180::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0443889 s, 23.1 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:01:06,190::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:01:06,682::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0772103 s, 13.3 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:01:07,527::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:01:08,697::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:01:09,197::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0623932 s, 16.4 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:01:11,211::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:01:11,788::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0828814 s, 12.4 MB/s\n'; <rc> = 0 >Thread-15909996::DEBUG::2013-11-19 21:01:12,435::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15909996::INFO::2013-11-19 21:01:12,436::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15909996::DEBUG::2013-11-19 21:01:12,437::task::495::TaskManager.Task::(_debug) Task 59171e4f-397b-492b-9b6f-3236be8e6bf4: moving from state init -> state preparing >Thread-15909996::DEBUG::2013-11-19 21:01:12,446::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15909996::DEBUG::2013-11-19 21:01:12,447::task::495::TaskManager.Task::(_debug) Task 59171e4f-397b-492b-9b6f-3236be8e6bf4: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15909996::DEBUG::2013-11-19 21:01:12,447::task::495::TaskManager.Task::(_debug) Task 59171e4f-397b-492b-9b6f-3236be8e6bf4: moving from state preparing -> state finished >Thread-15909996::DEBUG::2013-11-19 21:01:12,448::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15909996::DEBUG::2013-11-19 21:01:12,448::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15909996::DEBUG::2013-11-19 21:01:12,449::task::495::TaskManager.Task::(_debug) Task 59171e4f-397b-492b-9b6f-3236be8e6bf4: ref 0 aborting False >Thread-15909996::INFO::2013-11-19 21:01:12,450::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15909997::DEBUG::2013-11-19 21:01:12,462::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15909997::INFO::2013-11-19 21:01:12,463::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15909997::DEBUG::2013-11-19 21:01:12,464::task::495::TaskManager.Task::(_debug) Task 560e81b2-e9ed-4b17-96c1-eddca5cf97cc: moving from state init -> state preparing >Thread-15909997::DEBUG::2013-11-19 21:01:12,465::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`38cab960-17df-479c-ae61-27f84ed5dd2b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15909997::DEBUG::2013-11-19 21:01:12,466::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15909997::DEBUG::2013-11-19 21:01:12,468::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15909997::DEBUG::2013-11-19 21:01:12,468::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`38cab960-17df-479c-ae61-27f84ed5dd2b`::Granted request >Thread-15909997::DEBUG::2013-11-19 21:01:12,470::task::495::TaskManager.Task::(_debug) Task 560e81b2-e9ed-4b17-96c1-eddca5cf97cc: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15909997::DEBUG::2013-11-19 21:01:12,471::task::495::TaskManager.Task::(_debug) Task 560e81b2-e9ed-4b17-96c1-eddca5cf97cc: ref 1 aborting False >Thread-15909997::WARNING::2013-11-19 21:01:12,474::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15909997::DEBUG::2013-11-19 21:01:12,475::task::495::TaskManager.Task::(_debug) Task 560e81b2-e9ed-4b17-96c1-eddca5cf97cc: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15909997::DEBUG::2013-11-19 21:01:12,476::task::495::TaskManager.Task::(_debug) Task 560e81b2-e9ed-4b17-96c1-eddca5cf97cc: moving from state preparing -> state finished >Thread-15909997::DEBUG::2013-11-19 21:01:12,477::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15909997::DEBUG::2013-11-19 21:01:12,477::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15909997::DEBUG::2013-11-19 21:01:12,478::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15909997::DEBUG::2013-11-19 21:01:12,479::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15909997::DEBUG::2013-11-19 21:01:12,480::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15909997::DEBUG::2013-11-19 21:01:12,480::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15909997::DEBUG::2013-11-19 21:01:12,481::task::495::TaskManager.Task::(_debug) Task 560e81b2-e9ed-4b17-96c1-eddca5cf97cc: ref 0 aborting False >Thread-15909997::INFO::2013-11-19 21:01:12,482::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15909998::INFO::2013-11-19 21:01:12,688::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15909998::DEBUG::2013-11-19 21:01:12,688::task::495::TaskManager.Task::(_debug) Task ff581736-df30-4ea6-87ae-90afb178060d: moving from state init -> state preparing >Thread-15909998::DEBUG::2013-11-19 21:01:12,689::task::495::TaskManager.Task::(_debug) Task ff581736-df30-4ea6-87ae-90afb178060d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000279188156128', 'lastCheck': 1384894865.9636569, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00115513801575', 'lastCheck': 1384894872.5390949, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894867.5280011, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000637054443359', 'lastCheck': 1384894864.5238931, 'valid': True, 'code': 0}} >Thread-15909998::DEBUG::2013-11-19 21:01:12,690::task::495::TaskManager.Task::(_debug) Task ff581736-df30-4ea6-87ae-90afb178060d: moving from state preparing -> state finished >Thread-15909998::DEBUG::2013-11-19 21:01:12,690::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15909998::DEBUG::2013-11-19 21:01:12,691::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15909998::DEBUG::2013-11-19 21:01:12,691::task::495::TaskManager.Task::(_debug) Task ff581736-df30-4ea6-87ae-90afb178060d: ref 0 aborting False >Thread-15909998::INFO::2013-11-19 21:01:12,692::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000279188156128', 'lastCheck': 1384894865.9636569, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00115513801575', 'lastCheck': 1384894872.5390949, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894867.5280011, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000637054443359', 'lastCheck': 1384894864.5238931, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:01:13,803::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:01:14,379::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0804903 s, 12.7 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:01:16,392::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:01:16,945::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0796164 s, 12.9 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:01:17,624::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:01:18,960::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:01:19,607::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0765105 s, 13.4 MB/s\n'; <rc> = 0 >Thread-4922911::INFO::2013-11-19 21:01:21,196::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-4922911::DEBUG::2013-11-19 21:01:21,197::task::495::TaskManager.Task::(_debug) Task 5bc39542-2cfa-490d-8151-6a459e8a4c98: moving from state init -> state preparing >Thread-4922911::DEBUG::2013-11-19 21:01:21,199::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`ad7e2cd9-7ffd-4b6b-a244-5aacf97e0c71`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-4922911::DEBUG::2013-11-19 21:01:21,199::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-4922911::DEBUG::2013-11-19 21:01:21,200::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-4922911::DEBUG::2013-11-19 21:01:21,201::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`ad7e2cd9-7ffd-4b6b-a244-5aacf97e0c71`::Granted request >Thread-4922911::DEBUG::2013-11-19 21:01:21,202::task::495::TaskManager.Task::(_debug) Task 5bc39542-2cfa-490d-8151-6a459e8a4c98: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-4922911::DEBUG::2013-11-19 21:01:21,202::task::495::TaskManager.Task::(_debug) Task 5bc39542-2cfa-490d-8151-6a459e8a4c98: ref 1 aborting False >Thread-4922911::DEBUG::2013-11-19 21:01:21,203::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-4922911::DEBUG::2013-11-19 21:01:21,215::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:01:21,627::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-4922911::DEBUG::2013-11-19 21:01:22,013::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-4922911::DEBUG::2013-11-19 21:01:22,093::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-4922911::DEBUG::2013-11-19 21:01:22,104::task::495::TaskManager.Task::(_debug) Task 5bc39542-2cfa-490d-8151-6a459e8a4c98: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-4922911::DEBUG::2013-11-19 21:01:22,105::task::495::TaskManager.Task::(_debug) Task 5bc39542-2cfa-490d-8151-6a459e8a4c98: moving from state preparing -> state finished >Thread-4922911::DEBUG::2013-11-19 21:01:22,106::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-4922911::DEBUG::2013-11-19 21:01:22,106::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4922911::DEBUG::2013-11-19 21:01:22,108::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-4922911::DEBUG::2013-11-19 21:01:22,109::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-4922911::DEBUG::2013-11-19 21:01:22,110::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-4922911::DEBUG::2013-11-19 21:01:22,110::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-4922911::DEBUG::2013-11-19 21:01:22,112::task::495::TaskManager.Task::(_debug) Task 5bc39542-2cfa-490d-8151-6a459e8a4c98: ref 0 aborting False >Thread-4922911::INFO::2013-11-19 21:01:22,112::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-4922911::INFO::2013-11-19 21:01:22,114::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-4922911::DEBUG::2013-11-19 21:01:22,115::task::495::TaskManager.Task::(_debug) Task d2c8edc6-d7ef-41a3-9b50-257972b687fd: moving from state init -> state preparing >Thread-4922911::DEBUG::2013-11-19 21:01:22,116::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`3c0ecf2d-3d46-449f-84dc-3e55cf0cfc0e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-4922911::DEBUG::2013-11-19 21:01:22,117::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-4922911::DEBUG::2013-11-19 21:01:22,118::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-4922911::DEBUG::2013-11-19 21:01:22,119::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`3c0ecf2d-3d46-449f-84dc-3e55cf0cfc0e`::Granted request >Thread-4922911::DEBUG::2013-11-19 21:01:22,120::task::495::TaskManager.Task::(_debug) Task d2c8edc6-d7ef-41a3-9b50-257972b687fd: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-4922911::DEBUG::2013-11-19 21:01:22,121::task::495::TaskManager.Task::(_debug) Task d2c8edc6-d7ef-41a3-9b50-257972b687fd: ref 1 aborting False >Thread-4922911::DEBUG::2013-11-19 21:01:22,122::task::495::TaskManager.Task::(_debug) Task d2c8edc6-d7ef-41a3-9b50-257972b687fd: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-4922911::DEBUG::2013-11-19 21:01:22,122::task::495::TaskManager.Task::(_debug) Task d2c8edc6-d7ef-41a3-9b50-257972b687fd: moving from state preparing -> state finished >Thread-4922911::DEBUG::2013-11-19 21:01:22,123::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-4922911::DEBUG::2013-11-19 21:01:22,124::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4922911::DEBUG::2013-11-19 21:01:22,125::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-4922911::DEBUG::2013-11-19 21:01:22,126::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-4922911::DEBUG::2013-11-19 21:01:22,126::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-4922911::DEBUG::2013-11-19 21:01:22,127::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-4922911::DEBUG::2013-11-19 21:01:22,128::task::495::TaskManager.Task::(_debug) Task d2c8edc6-d7ef-41a3-9b50-257972b687fd: ref 0 aborting False >Thread-4922911::INFO::2013-11-19 21:01:22,129::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-4922911::INFO::2013-11-19 21:01:22,130::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-4922911::DEBUG::2013-11-19 21:01:22,131::task::495::TaskManager.Task::(_debug) Task 7464a467-3e8f-4a2d-a8e1-dbff00120867: moving from state init -> state preparing >Thread-4922911::DEBUG::2013-11-19 21:01:22,132::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`f9461eb8-3a82-47ba-b0e4-41deaf87f8c0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-4922911::DEBUG::2013-11-19 21:01:22,133::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-4922911::DEBUG::2013-11-19 21:01:22,134::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-4922911::DEBUG::2013-11-19 21:01:22,134::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`f9461eb8-3a82-47ba-b0e4-41deaf87f8c0`::Granted request >Thread-4922911::DEBUG::2013-11-19 21:01:22,136::task::495::TaskManager.Task::(_debug) Task 7464a467-3e8f-4a2d-a8e1-dbff00120867: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-4922911::DEBUG::2013-11-19 21:01:22,137::task::495::TaskManager.Task::(_debug) Task 7464a467-3e8f-4a2d-a8e1-dbff00120867: ref 1 aborting False >Thread-4922911::DEBUG::2013-11-19 21:01:22,138::task::495::TaskManager.Task::(_debug) Task 7464a467-3e8f-4a2d-a8e1-dbff00120867: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-4922911::DEBUG::2013-11-19 21:01:22,138::task::495::TaskManager.Task::(_debug) Task 7464a467-3e8f-4a2d-a8e1-dbff00120867: moving from state preparing -> state finished >Thread-4922911::DEBUG::2013-11-19 21:01:22,139::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-4922911::DEBUG::2013-11-19 21:01:22,140::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4922911::DEBUG::2013-11-19 21:01:22,141::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-4922911::DEBUG::2013-11-19 21:01:22,142::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-4922911::DEBUG::2013-11-19 21:01:22,142::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-4922911::DEBUG::2013-11-19 21:01:22,143::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-4922911::DEBUG::2013-11-19 21:01:22,144::task::495::TaskManager.Task::(_debug) Task 7464a467-3e8f-4a2d-a8e1-dbff00120867: ref 0 aborting False >Thread-4922911::INFO::2013-11-19 21:01:22,145::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-4922911::INFO::2013-11-19 21:01:22,146::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-4922911::DEBUG::2013-11-19 21:01:22,146::task::495::TaskManager.Task::(_debug) Task 9a21f246-cb53-4233-a17c-d6454d07d8ee: moving from state init -> state preparing >Thread-4922911::DEBUG::2013-11-19 21:01:22,148::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`24a3070b-1fb7-4dcd-8645-98f01a10c769`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-4922911::DEBUG::2013-11-19 21:01:22,149::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-4922911::DEBUG::2013-11-19 21:01:22,149::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-4922911::DEBUG::2013-11-19 21:01:22,150::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`24a3070b-1fb7-4dcd-8645-98f01a10c769`::Granted request >Thread-4922911::DEBUG::2013-11-19 21:01:22,152::task::495::TaskManager.Task::(_debug) Task 9a21f246-cb53-4233-a17c-d6454d07d8ee: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-4922911::DEBUG::2013-11-19 21:01:22,152::task::495::TaskManager.Task::(_debug) Task 9a21f246-cb53-4233-a17c-d6454d07d8ee: ref 1 aborting False >Thread-4922911::DEBUG::2013-11-19 21:01:22,153::task::495::TaskManager.Task::(_debug) Task 9a21f246-cb53-4233-a17c-d6454d07d8ee: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-4922911::DEBUG::2013-11-19 21:01:22,154::task::495::TaskManager.Task::(_debug) Task 9a21f246-cb53-4233-a17c-d6454d07d8ee: moving from state preparing -> state finished >Thread-4922911::DEBUG::2013-11-19 21:01:22,155::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-4922911::DEBUG::2013-11-19 21:01:22,155::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4922911::DEBUG::2013-11-19 21:01:22,156::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-4922911::DEBUG::2013-11-19 21:01:22,157::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-4922911::DEBUG::2013-11-19 21:01:22,158::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-4922911::DEBUG::2013-11-19 21:01:22,158::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-4922911::DEBUG::2013-11-19 21:01:22,159::task::495::TaskManager.Task::(_debug) Task 9a21f246-cb53-4233-a17c-d6454d07d8ee: ref 0 aborting False >Thread-4922911::INFO::2013-11-19 21:01:22,160::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15909965::DEBUG::2013-11-19 21:01:22,290::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.188912 s, 5.4 MB/s\n'; <rc> = 0 >Thread-15909535::DEBUG::2013-11-19 21:01:22,567::guestIF::97::vm.Vm::(_handleMessage) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::u'AOS1Batch@brd-klee.dk' >Thread-15910004::DEBUG::2013-11-19 21:01:22,610::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910004::INFO::2013-11-19 21:01:22,611::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910004::DEBUG::2013-11-19 21:01:22,612::task::495::TaskManager.Task::(_debug) Task 04c3b15b-5f11-4801-9334-b7c02ec9353e: moving from state init -> state preparing >Thread-15910004::DEBUG::2013-11-19 21:01:22,612::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910004::DEBUG::2013-11-19 21:01:22,613::task::495::TaskManager.Task::(_debug) Task 04c3b15b-5f11-4801-9334-b7c02ec9353e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910004::DEBUG::2013-11-19 21:01:22,614::task::495::TaskManager.Task::(_debug) Task 04c3b15b-5f11-4801-9334-b7c02ec9353e: moving from state preparing -> state finished >Thread-15910004::DEBUG::2013-11-19 21:01:22,614::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910004::DEBUG::2013-11-19 21:01:22,615::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910004::DEBUG::2013-11-19 21:01:22,615::task::495::TaskManager.Task::(_debug) Task 04c3b15b-5f11-4801-9334-b7c02ec9353e: ref 0 aborting False >Thread-15910004::INFO::2013-11-19 21:01:22,616::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910005::DEBUG::2013-11-19 21:01:22,621::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910005::INFO::2013-11-19 21:01:22,621::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15868524::INFO::2013-11-19 21:01:22,630::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=e0628852-41b1-4535-8dda-ebc96cb00952 volUUID=cd4f9324-c91d-405b-983f-62afa4daa8ad) >Thread-15910005::DEBUG::2013-11-19 21:01:22,632::task::495::TaskManager.Task::(_debug) Task 212cb393-ef4e-4c5d-b925-b91acd034e46: moving from state init -> state preparing >Thread-15868524::DEBUG::2013-11-19 21:01:22,633::task::495::TaskManager.Task::(_debug) Task a2100830-05f0-4377-add4-d1f6895dab03: moving from state init -> state preparing >Thread-15910005::DEBUG::2013-11-19 21:01:22,634::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b71149ba-a62c-47e1-885b-fe53799e6680`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15868524::DEBUG::2013-11-19 21:01:22,636::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`737867fc-820c-42ed-af00-580a718a4926`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15910005::DEBUG::2013-11-19 21:01:22,637::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15868524::DEBUG::2013-11-19 21:01:22,638::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15910005::DEBUG::2013-11-19 21:01:22,639::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910005::DEBUG::2013-11-19 21:01:22,640::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b71149ba-a62c-47e1-885b-fe53799e6680`::Granted request >Thread-15868524::DEBUG::2013-11-19 21:01:22,641::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15910005::DEBUG::2013-11-19 21:01:22,641::task::495::TaskManager.Task::(_debug) Task 212cb393-ef4e-4c5d-b925-b91acd034e46: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15868524::DEBUG::2013-11-19 21:01:22,642::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`737867fc-820c-42ed-af00-580a718a4926`::Granted request >Thread-15910005::DEBUG::2013-11-19 21:01:22,643::task::495::TaskManager.Task::(_debug) Task 212cb393-ef4e-4c5d-b925-b91acd034e46: ref 1 aborting False >Thread-15868524::DEBUG::2013-11-19 21:01:22,644::task::495::TaskManager.Task::(_debug) Task a2100830-05f0-4377-add4-d1f6895dab03: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15910005::WARNING::2013-11-19 21:01:22,647::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15868524::DEBUG::2013-11-19 21:01:22,647::task::495::TaskManager.Task::(_debug) Task a2100830-05f0-4377-add4-d1f6895dab03: ref 1 aborting False >Thread-15910005::DEBUG::2013-11-19 21:01:22,648::task::495::TaskManager.Task::(_debug) Task 212cb393-ef4e-4c5d-b925-b91acd034e46: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15868524::DEBUG::2013-11-19 21:01:22,649::task::495::TaskManager.Task::(_debug) Task a2100830-05f0-4377-add4-d1f6895dab03: finished: {'truesize': '32346472448', 'apparentsize': '32346472448'} >Thread-15910005::DEBUG::2013-11-19 21:01:22,650::task::495::TaskManager.Task::(_debug) Task 212cb393-ef4e-4c5d-b925-b91acd034e46: moving from state preparing -> state finished >Thread-15868524::DEBUG::2013-11-19 21:01:22,651::task::495::TaskManager.Task::(_debug) Task a2100830-05f0-4377-add4-d1f6895dab03: moving from state preparing -> state finished >Thread-15910005::DEBUG::2013-11-19 21:01:22,651::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15868524::DEBUG::2013-11-19 21:01:22,652::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15910005::DEBUG::2013-11-19 21:01:22,653::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15868524::DEBUG::2013-11-19 21:01:22,653::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910005::DEBUG::2013-11-19 21:01:22,654::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15868524::DEBUG::2013-11-19 21:01:22,655::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15910005::DEBUG::2013-11-19 21:01:22,656::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910005::DEBUG::2013-11-19 21:01:22,657::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910005::DEBUG::2013-11-19 21:01:22,657::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15868524::DEBUG::2013-11-19 21:01:22,658::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15910005::DEBUG::2013-11-19 21:01:22,658::task::495::TaskManager.Task::(_debug) Task 212cb393-ef4e-4c5d-b925-b91acd034e46: ref 0 aborting False >Thread-15868524::DEBUG::2013-11-19 21:01:22,659::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15910005::INFO::2013-11-19 21:01:22,660::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15868524::DEBUG::2013-11-19 21:01:22,660::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15868524::DEBUG::2013-11-19 21:01:22,662::task::495::TaskManager.Task::(_debug) Task a2100830-05f0-4377-add4-d1f6895dab03: ref 0 aborting False >Thread-15868524::INFO::2013-11-19 21:01:22,663::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '32346472448', 'apparentsize': '32346472448'} >Thread-15910006::INFO::2013-11-19 21:01:22,915::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910006::DEBUG::2013-11-19 21:01:22,916::task::495::TaskManager.Task::(_debug) Task 3106f792-1c67-4842-8057-2e9bc26517f9: moving from state init -> state preparing >Thread-15910006::DEBUG::2013-11-19 21:01:22,917::task::495::TaskManager.Task::(_debug) Task 3106f792-1c67-4842-8057-2e9bc26517f9: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000410079956055', 'lastCheck': 1384894876.0406599, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00194406509399', 'lastCheck': 1384894882.6341989, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894877.6248951, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000679016113281', 'lastCheck': 1384894874.5350571, 'valid': True, 'code': 0}} >Thread-15910006::DEBUG::2013-11-19 21:01:22,918::task::495::TaskManager.Task::(_debug) Task 3106f792-1c67-4842-8057-2e9bc26517f9: moving from state preparing -> state finished >Thread-15910006::DEBUG::2013-11-19 21:01:22,919::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910006::DEBUG::2013-11-19 21:01:22,919::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910006::DEBUG::2013-11-19 21:01:22,920::task::495::TaskManager.Task::(_debug) Task 3106f792-1c67-4842-8057-2e9bc26517f9: ref 0 aborting False >Thread-15910006::INFO::2013-11-19 21:01:22,920::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000410079956055', 'lastCheck': 1384894876.0406599, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00194406509399', 'lastCheck': 1384894882.6341989, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894877.6248951, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000679016113281', 'lastCheck': 1384894874.5350571, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:01:24,305::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:01:24,874::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0715697 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:01:26,889::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:01:27,399::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0796909 s, 12.8 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:01:27,639::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:01:29,415::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:01:30,014::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0822195 s, 12.5 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:01:32,024::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:01:32,540::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0831492 s, 12.3 MB/s\n'; <rc> = 0 >Thread-15910012::DEBUG::2013-11-19 21:01:32,789::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910012::INFO::2013-11-19 21:01:32,792::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910012::DEBUG::2013-11-19 21:01:32,792::task::495::TaskManager.Task::(_debug) Task df93d138-ce77-4017-902e-7f2033781f67: moving from state init -> state preparing >Thread-15910012::DEBUG::2013-11-19 21:01:32,793::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910012::DEBUG::2013-11-19 21:01:32,794::task::495::TaskManager.Task::(_debug) Task df93d138-ce77-4017-902e-7f2033781f67: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910012::DEBUG::2013-11-19 21:01:32,794::task::495::TaskManager.Task::(_debug) Task df93d138-ce77-4017-902e-7f2033781f67: moving from state preparing -> state finished >Thread-15910012::DEBUG::2013-11-19 21:01:32,795::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910012::DEBUG::2013-11-19 21:01:32,795::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910012::DEBUG::2013-11-19 21:01:32,796::task::495::TaskManager.Task::(_debug) Task df93d138-ce77-4017-902e-7f2033781f67: ref 0 aborting False >Thread-15910012::INFO::2013-11-19 21:01:32,796::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910013::DEBUG::2013-11-19 21:01:32,801::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910013::INFO::2013-11-19 21:01:32,802::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910013::DEBUG::2013-11-19 21:01:32,803::task::495::TaskManager.Task::(_debug) Task 386691f5-daa8-49dc-b888-f0cd8c4fc3f3: moving from state init -> state preparing >Thread-15910013::DEBUG::2013-11-19 21:01:32,804::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`033e88b2-9ecc-4e27-93e9-fe4ccd906126`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910013::DEBUG::2013-11-19 21:01:32,805::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910013::DEBUG::2013-11-19 21:01:32,805::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910013::DEBUG::2013-11-19 21:01:32,806::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`033e88b2-9ecc-4e27-93e9-fe4ccd906126`::Granted request >Thread-15910013::DEBUG::2013-11-19 21:01:32,807::task::495::TaskManager.Task::(_debug) Task 386691f5-daa8-49dc-b888-f0cd8c4fc3f3: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910013::DEBUG::2013-11-19 21:01:32,807::task::495::TaskManager.Task::(_debug) Task 386691f5-daa8-49dc-b888-f0cd8c4fc3f3: ref 1 aborting False >Thread-15910013::WARNING::2013-11-19 21:01:32,809::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910013::DEBUG::2013-11-19 21:01:32,810::task::495::TaskManager.Task::(_debug) Task 386691f5-daa8-49dc-b888-f0cd8c4fc3f3: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910013::DEBUG::2013-11-19 21:01:32,811::task::495::TaskManager.Task::(_debug) Task 386691f5-daa8-49dc-b888-f0cd8c4fc3f3: moving from state preparing -> state finished >Thread-15910013::DEBUG::2013-11-19 21:01:32,812::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910013::DEBUG::2013-11-19 21:01:32,812::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910013::DEBUG::2013-11-19 21:01:32,813::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910013::DEBUG::2013-11-19 21:01:32,813::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910013::DEBUG::2013-11-19 21:01:32,814::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910013::DEBUG::2013-11-19 21:01:32,814::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910013::DEBUG::2013-11-19 21:01:32,815::task::495::TaskManager.Task::(_debug) Task 386691f5-daa8-49dc-b888-f0cd8c4fc3f3: ref 0 aborting False >Thread-15910013::INFO::2013-11-19 21:01:32,816::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910014::INFO::2013-11-19 21:01:33,193::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910014::DEBUG::2013-11-19 21:01:33,193::task::495::TaskManager.Task::(_debug) Task 9576e14f-d870-49b0-9d7a-fbf1513bb616: moving from state init -> state preparing >Thread-15910014::DEBUG::2013-11-19 21:01:33,194::task::495::TaskManager.Task::(_debug) Task 9576e14f-d870-49b0-9d7a-fbf1513bb616: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000364065170288', 'lastCheck': 1384894886.2032509, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00187611579895', 'lastCheck': 1384894892.6852951, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894887.640872, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000503063201904', 'lastCheck': 1384894884.6285191, 'valid': True, 'code': 0}} >Thread-15910014::DEBUG::2013-11-19 21:01:33,195::task::495::TaskManager.Task::(_debug) Task 9576e14f-d870-49b0-9d7a-fbf1513bb616: moving from state preparing -> state finished >Thread-15910014::DEBUG::2013-11-19 21:01:33,195::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910014::DEBUG::2013-11-19 21:01:33,196::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910014::DEBUG::2013-11-19 21:01:33,197::task::495::TaskManager.Task::(_debug) Task 9576e14f-d870-49b0-9d7a-fbf1513bb616: ref 0 aborting False >Thread-15910014::INFO::2013-11-19 21:01:33,198::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000364065170288', 'lastCheck': 1384894886.2032509, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00187611579895', 'lastCheck': 1384894892.6852951, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894887.640872, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000503063201904', 'lastCheck': 1384894884.6285191, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:01:34,556::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:01:35,123::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0795763 s, 12.9 MB/s\n'; <rc> = 0 >Thread-4922912::DEBUG::2013-11-19 21:01:36,722::guestIF::97::vm.Vm::(_handleMessage) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::u'BRD-KLEE\\Administrator' >Dummy-15909965::DEBUG::2013-11-19 21:01:37,138::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:01:37,710::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0857649 s, 11.9 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:01:37,790::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:01:39,726::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:01:40,211::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0474575 s, 21.6 MB/s\n'; <rc> = 0 >Thread-15909534::INFO::2013-11-19 21:01:40,233::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15909534::DEBUG::2013-11-19 21:01:40,234::task::495::TaskManager.Task::(_debug) Task 93216059-81a6-473b-901d-9c2dda12ef29: moving from state init -> state preparing >Thread-15909534::DEBUG::2013-11-19 21:01:40,236::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`aefbb1e0-5b34-4a18-8ad4-950ccf811034`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15909534::DEBUG::2013-11-19 21:01:40,236::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15909534::DEBUG::2013-11-19 21:01:40,237::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15909534::DEBUG::2013-11-19 21:01:40,238::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`aefbb1e0-5b34-4a18-8ad4-950ccf811034`::Granted request >Thread-15909534::DEBUG::2013-11-19 21:01:40,239::task::495::TaskManager.Task::(_debug) Task 93216059-81a6-473b-901d-9c2dda12ef29: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15909534::DEBUG::2013-11-19 21:01:40,240::task::495::TaskManager.Task::(_debug) Task 93216059-81a6-473b-901d-9c2dda12ef29: ref 1 aborting False >Thread-15909534::DEBUG::2013-11-19 21:01:40,241::task::495::TaskManager.Task::(_debug) Task 93216059-81a6-473b-901d-9c2dda12ef29: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15909534::DEBUG::2013-11-19 21:01:40,241::task::495::TaskManager.Task::(_debug) Task 93216059-81a6-473b-901d-9c2dda12ef29: moving from state preparing -> state finished >Thread-15909534::DEBUG::2013-11-19 21:01:40,242::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15909534::DEBUG::2013-11-19 21:01:40,243::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15909534::DEBUG::2013-11-19 21:01:40,244::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15909534::DEBUG::2013-11-19 21:01:40,244::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15909534::DEBUG::2013-11-19 21:01:40,245::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15909534::DEBUG::2013-11-19 21:01:40,245::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15909534::DEBUG::2013-11-19 21:01:40,246::task::495::TaskManager.Task::(_debug) Task 93216059-81a6-473b-901d-9c2dda12ef29: ref 0 aborting False >Thread-15909534::INFO::2013-11-19 21:01:40,247::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15909965::DEBUG::2013-11-19 21:01:42,221::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:01:42,781::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0753163 s, 13.6 MB/s\n'; <rc> = 0 >Thread-15910020::DEBUG::2013-11-19 21:01:42,932::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910020::INFO::2013-11-19 21:01:42,933::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910020::DEBUG::2013-11-19 21:01:42,933::task::495::TaskManager.Task::(_debug) Task efc1a099-4439-4c60-861c-bc0969c313f3: moving from state init -> state preparing >Thread-15910020::DEBUG::2013-11-19 21:01:42,934::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910020::DEBUG::2013-11-19 21:01:42,934::task::495::TaskManager.Task::(_debug) Task efc1a099-4439-4c60-861c-bc0969c313f3: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910020::DEBUG::2013-11-19 21:01:42,934::task::495::TaskManager.Task::(_debug) Task efc1a099-4439-4c60-861c-bc0969c313f3: moving from state preparing -> state finished >Thread-15910020::DEBUG::2013-11-19 21:01:42,935::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910020::DEBUG::2013-11-19 21:01:42,935::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910020::DEBUG::2013-11-19 21:01:42,935::task::495::TaskManager.Task::(_debug) Task efc1a099-4439-4c60-861c-bc0969c313f3: ref 0 aborting False >Thread-15910020::INFO::2013-11-19 21:01:42,936::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910021::DEBUG::2013-11-19 21:01:42,940::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910021::INFO::2013-11-19 21:01:42,941::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910021::DEBUG::2013-11-19 21:01:42,941::task::495::TaskManager.Task::(_debug) Task dc12650e-2b17-45bc-b50e-d430b33f713c: moving from state init -> state preparing >Thread-15910021::DEBUG::2013-11-19 21:01:42,943::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e4745502-798a-4569-a3d2-8037c6d095a4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910021::DEBUG::2013-11-19 21:01:42,943::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910021::DEBUG::2013-11-19 21:01:42,944::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910021::DEBUG::2013-11-19 21:01:42,945::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e4745502-798a-4569-a3d2-8037c6d095a4`::Granted request >Thread-15910021::DEBUG::2013-11-19 21:01:42,946::task::495::TaskManager.Task::(_debug) Task dc12650e-2b17-45bc-b50e-d430b33f713c: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910021::DEBUG::2013-11-19 21:01:42,946::task::495::TaskManager.Task::(_debug) Task dc12650e-2b17-45bc-b50e-d430b33f713c: ref 1 aborting False >Thread-15910021::WARNING::2013-11-19 21:01:42,948::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910021::DEBUG::2013-11-19 21:01:42,949::task::495::TaskManager.Task::(_debug) Task dc12650e-2b17-45bc-b50e-d430b33f713c: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910021::DEBUG::2013-11-19 21:01:42,950::task::495::TaskManager.Task::(_debug) Task dc12650e-2b17-45bc-b50e-d430b33f713c: moving from state preparing -> state finished >Thread-15910021::DEBUG::2013-11-19 21:01:42,951::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910021::DEBUG::2013-11-19 21:01:42,951::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910021::DEBUG::2013-11-19 21:01:42,952::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910021::DEBUG::2013-11-19 21:01:42,953::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910021::DEBUG::2013-11-19 21:01:42,953::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910021::DEBUG::2013-11-19 21:01:42,954::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910021::DEBUG::2013-11-19 21:01:42,954::task::495::TaskManager.Task::(_debug) Task dc12650e-2b17-45bc-b50e-d430b33f713c: ref 0 aborting False >Thread-15910021::INFO::2013-11-19 21:01:42,955::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910022::INFO::2013-11-19 21:01:43,409::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910022::DEBUG::2013-11-19 21:01:43,410::task::495::TaskManager.Task::(_debug) Task ce249973-9c45-4bdf-8947-15a7324c08a7: moving from state init -> state preparing >Thread-15910022::DEBUG::2013-11-19 21:01:43,410::task::495::TaskManager.Task::(_debug) Task ce249973-9c45-4bdf-8947-15a7324c08a7: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000366926193237', 'lastCheck': 1384894896.230176, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0010130405426', 'lastCheck': 1384894902.931289, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894897.791348, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000730037689209', 'lastCheck': 1384894894.7851319, 'valid': True, 'code': 0}} >Thread-15910022::DEBUG::2013-11-19 21:01:43,411::task::495::TaskManager.Task::(_debug) Task ce249973-9c45-4bdf-8947-15a7324c08a7: moving from state preparing -> state finished >Thread-15910022::DEBUG::2013-11-19 21:01:43,411::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910022::DEBUG::2013-11-19 21:01:43,411::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910022::DEBUG::2013-11-19 21:01:43,412::task::495::TaskManager.Task::(_debug) Task ce249973-9c45-4bdf-8947-15a7324c08a7: ref 0 aborting False >Thread-15910022::INFO::2013-11-19 21:01:43,412::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000366926193237', 'lastCheck': 1384894896.230176, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0010130405426', 'lastCheck': 1384894902.931289, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894897.791348, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000730037689209', 'lastCheck': 1384894894.7851319, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:01:44,798::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:01:45,316::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0788518 s, 13.0 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:01:47,332::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:01:47,918::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.12047 s, 8.5 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:01:48,015::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:01:49,935::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:01:50,539::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0804068 s, 12.7 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:01:52,559::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:01:53,103::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0793667 s, 12.9 MB/s\n'; <rc> = 0 >Thread-15910028::DEBUG::2013-11-19 21:01:53,189::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910028::INFO::2013-11-19 21:01:53,191::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910028::DEBUG::2013-11-19 21:01:53,191::task::495::TaskManager.Task::(_debug) Task a29ad9fe-2b4d-4560-b678-a7c8ddd2c95e: moving from state init -> state preparing >Thread-15910028::DEBUG::2013-11-19 21:01:53,192::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910028::DEBUG::2013-11-19 21:01:53,192::task::495::TaskManager.Task::(_debug) Task a29ad9fe-2b4d-4560-b678-a7c8ddd2c95e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910028::DEBUG::2013-11-19 21:01:53,193::task::495::TaskManager.Task::(_debug) Task a29ad9fe-2b4d-4560-b678-a7c8ddd2c95e: moving from state preparing -> state finished >Thread-15910028::DEBUG::2013-11-19 21:01:53,194::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910028::DEBUG::2013-11-19 21:01:53,194::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910028::DEBUG::2013-11-19 21:01:53,195::task::495::TaskManager.Task::(_debug) Task a29ad9fe-2b4d-4560-b678-a7c8ddd2c95e: ref 0 aborting False >Thread-15910028::INFO::2013-11-19 21:01:53,195::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910029::DEBUG::2013-11-19 21:01:53,200::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910029::INFO::2013-11-19 21:01:53,201::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910029::DEBUG::2013-11-19 21:01:53,202::task::495::TaskManager.Task::(_debug) Task c1921c03-3c39-4aaf-bfb6-7e3fc78d7b26: moving from state init -> state preparing >Thread-15910029::DEBUG::2013-11-19 21:01:53,203::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ab305262-e39a-4e0d-a11b-4d1165aa9184`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910029::DEBUG::2013-11-19 21:01:53,204::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910029::DEBUG::2013-11-19 21:01:53,205::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910029::DEBUG::2013-11-19 21:01:53,205::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ab305262-e39a-4e0d-a11b-4d1165aa9184`::Granted request >Thread-15910029::DEBUG::2013-11-19 21:01:53,206::task::495::TaskManager.Task::(_debug) Task c1921c03-3c39-4aaf-bfb6-7e3fc78d7b26: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910029::DEBUG::2013-11-19 21:01:53,207::task::495::TaskManager.Task::(_debug) Task c1921c03-3c39-4aaf-bfb6-7e3fc78d7b26: ref 1 aborting False >Thread-15910029::WARNING::2013-11-19 21:01:53,209::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910029::DEBUG::2013-11-19 21:01:53,210::task::495::TaskManager.Task::(_debug) Task c1921c03-3c39-4aaf-bfb6-7e3fc78d7b26: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910029::DEBUG::2013-11-19 21:01:53,211::task::495::TaskManager.Task::(_debug) Task c1921c03-3c39-4aaf-bfb6-7e3fc78d7b26: moving from state preparing -> state finished >Thread-15910029::DEBUG::2013-11-19 21:01:53,211::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910029::DEBUG::2013-11-19 21:01:53,212::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910029::DEBUG::2013-11-19 21:01:53,213::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910029::DEBUG::2013-11-19 21:01:53,213::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910029::DEBUG::2013-11-19 21:01:53,214::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910029::DEBUG::2013-11-19 21:01:53,214::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910029::DEBUG::2013-11-19 21:01:53,215::task::495::TaskManager.Task::(_debug) Task c1921c03-3c39-4aaf-bfb6-7e3fc78d7b26: ref 0 aborting False >Thread-15910029::INFO::2013-11-19 21:01:53,216::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910030::INFO::2013-11-19 21:01:53,602::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910030::DEBUG::2013-11-19 21:01:53,603::task::495::TaskManager.Task::(_debug) Task dd39d70b-2b6b-4dd4-82dd-32c0b30bd2e3: moving from state init -> state preparing >Thread-15910030::DEBUG::2013-11-19 21:01:53,604::task::495::TaskManager.Task::(_debug) Task dd39d70b-2b6b-4dd4-82dd-32c0b30bd2e3: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000334024429321', 'lastCheck': 1384894906.4503391, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00132894515991', 'lastCheck': 1384894913.170774, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894908.0167711, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00113487243652', 'lastCheck': 1384894905.011519, 'valid': True, 'code': 0}} >Thread-15910030::DEBUG::2013-11-19 21:01:53,605::task::495::TaskManager.Task::(_debug) Task dd39d70b-2b6b-4dd4-82dd-32c0b30bd2e3: moving from state preparing -> state finished >Thread-15910030::DEBUG::2013-11-19 21:01:53,605::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910030::DEBUG::2013-11-19 21:01:53,606::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910030::DEBUG::2013-11-19 21:01:53,607::task::495::TaskManager.Task::(_debug) Task dd39d70b-2b6b-4dd4-82dd-32c0b30bd2e3: ref 0 aborting False >Thread-15910030::INFO::2013-11-19 21:01:53,607::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000334024429321', 'lastCheck': 1384894906.4503391, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00132894515991', 'lastCheck': 1384894913.170774, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894908.0167711, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00113487243652', 'lastCheck': 1384894905.011519, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:01:55,119::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-42840::DEBUG::2013-11-19 21:01:55,329::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:01:55,806::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.086326 s, 11.9 MB/s\n'; <rc> = 0 >Thread-42840::DEBUG::2013-11-19 21:01:56,100::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:01:57,822::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:01:58,337::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:01:58,366::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0712587 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:02:00,381::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:02:00,919::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0780567 s, 13.1 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:02:02,935::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:02:03,488::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0758178 s, 13.5 MB/s\n'; <rc> = 0 >Thread-15910036::DEBUG::2013-11-19 21:02:03,663::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910036::INFO::2013-11-19 21:02:03,664::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910036::DEBUG::2013-11-19 21:02:03,665::task::495::TaskManager.Task::(_debug) Task f5b6fd4c-f6b4-4bcc-83e1-4d472d56bed1: moving from state init -> state preparing >Thread-15910036::DEBUG::2013-11-19 21:02:03,666::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910036::DEBUG::2013-11-19 21:02:03,666::task::495::TaskManager.Task::(_debug) Task f5b6fd4c-f6b4-4bcc-83e1-4d472d56bed1: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910036::DEBUG::2013-11-19 21:02:03,667::task::495::TaskManager.Task::(_debug) Task f5b6fd4c-f6b4-4bcc-83e1-4d472d56bed1: moving from state preparing -> state finished >Thread-15910036::DEBUG::2013-11-19 21:02:03,667::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910036::DEBUG::2013-11-19 21:02:03,668::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910036::DEBUG::2013-11-19 21:02:03,668::task::495::TaskManager.Task::(_debug) Task f5b6fd4c-f6b4-4bcc-83e1-4d472d56bed1: ref 0 aborting False >Thread-15910036::INFO::2013-11-19 21:02:03,669::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910037::DEBUG::2013-11-19 21:02:03,674::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910037::INFO::2013-11-19 21:02:03,674::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910037::DEBUG::2013-11-19 21:02:03,675::task::495::TaskManager.Task::(_debug) Task 59ef5f50-a15e-4e13-ba6f-80bec36a97e5: moving from state init -> state preparing >Thread-15910037::DEBUG::2013-11-19 21:02:03,676::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`239f6a66-d27b-4a30-ad74-75bfa6108018`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910037::DEBUG::2013-11-19 21:02:03,677::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910037::DEBUG::2013-11-19 21:02:03,678::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910037::DEBUG::2013-11-19 21:02:03,678::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`239f6a66-d27b-4a30-ad74-75bfa6108018`::Granted request >Thread-15910037::DEBUG::2013-11-19 21:02:03,679::task::495::TaskManager.Task::(_debug) Task 59ef5f50-a15e-4e13-ba6f-80bec36a97e5: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910037::DEBUG::2013-11-19 21:02:03,680::task::495::TaskManager.Task::(_debug) Task 59ef5f50-a15e-4e13-ba6f-80bec36a97e5: ref 1 aborting False >Thread-15910037::WARNING::2013-11-19 21:02:03,682::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910037::DEBUG::2013-11-19 21:02:03,683::task::495::TaskManager.Task::(_debug) Task 59ef5f50-a15e-4e13-ba6f-80bec36a97e5: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910037::DEBUG::2013-11-19 21:02:03,684::task::495::TaskManager.Task::(_debug) Task 59ef5f50-a15e-4e13-ba6f-80bec36a97e5: moving from state preparing -> state finished >Thread-15910037::DEBUG::2013-11-19 21:02:03,684::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910037::DEBUG::2013-11-19 21:02:03,685::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910037::DEBUG::2013-11-19 21:02:03,686::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910037::DEBUG::2013-11-19 21:02:03,687::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910037::DEBUG::2013-11-19 21:02:03,687::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910037::DEBUG::2013-11-19 21:02:03,688::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910037::DEBUG::2013-11-19 21:02:03,689::task::495::TaskManager.Task::(_debug) Task 59ef5f50-a15e-4e13-ba6f-80bec36a97e5: ref 0 aborting False >Thread-15910037::INFO::2013-11-19 21:02:03,689::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910038::INFO::2013-11-19 21:02:04,133::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910038::DEBUG::2013-11-19 21:02:04,134::task::495::TaskManager.Task::(_debug) Task f64ca708-9ea8-4ab6-b682-7fc58ed19260: moving from state init -> state preparing >Thread-15910038::DEBUG::2013-11-19 21:02:04,134::task::495::TaskManager.Task::(_debug) Task f64ca708-9ea8-4ab6-b682-7fc58ed19260: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000420808792114', 'lastCheck': 1384894916.673322, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00155305862427', 'lastCheck': 1384894923.343523, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894918.3400691, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00031304359436', 'lastCheck': 1384894916.1024711, 'valid': True, 'code': 0}} >Thread-15910038::DEBUG::2013-11-19 21:02:04,135::task::495::TaskManager.Task::(_debug) Task f64ca708-9ea8-4ab6-b682-7fc58ed19260: moving from state preparing -> state finished >Thread-15910038::DEBUG::2013-11-19 21:02:04,136::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910038::DEBUG::2013-11-19 21:02:04,136::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910038::DEBUG::2013-11-19 21:02:04,136::task::495::TaskManager.Task::(_debug) Task f64ca708-9ea8-4ab6-b682-7fc58ed19260: ref 0 aborting False >Thread-15910038::INFO::2013-11-19 21:02:04,137::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000420808792114', 'lastCheck': 1384894916.673322, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00155305862427', 'lastCheck': 1384894923.343523, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894918.3400691, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00031304359436', 'lastCheck': 1384894916.1024711, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:02:05,502::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:02:05,936::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0474992 s, 21.6 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:02:07,951::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:02:08,353::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:02:08,478::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0803388 s, 12.7 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:02:10,493::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:02:11,054::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.088026 s, 11.6 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:02:13,072::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:02:13,588::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0767494 s, 13.3 MB/s\n'; <rc> = 0 >Thread-15910044::DEBUG::2013-11-19 21:02:13,789::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910044::INFO::2013-11-19 21:02:13,791::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910044::DEBUG::2013-11-19 21:02:13,791::task::495::TaskManager.Task::(_debug) Task 15d007fe-ee35-428e-b8e3-748f4c8cfc06: moving from state init -> state preparing >Thread-15910044::DEBUG::2013-11-19 21:02:13,792::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910044::DEBUG::2013-11-19 21:02:13,793::task::495::TaskManager.Task::(_debug) Task 15d007fe-ee35-428e-b8e3-748f4c8cfc06: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910044::DEBUG::2013-11-19 21:02:13,794::task::495::TaskManager.Task::(_debug) Task 15d007fe-ee35-428e-b8e3-748f4c8cfc06: moving from state preparing -> state finished >Thread-15910044::DEBUG::2013-11-19 21:02:13,795::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910044::DEBUG::2013-11-19 21:02:13,796::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910044::DEBUG::2013-11-19 21:02:13,796::task::495::TaskManager.Task::(_debug) Task 15d007fe-ee35-428e-b8e3-748f4c8cfc06: ref 0 aborting False >Thread-15910044::INFO::2013-11-19 21:02:13,797::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910045::DEBUG::2013-11-19 21:02:13,802::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910045::INFO::2013-11-19 21:02:13,803::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910045::DEBUG::2013-11-19 21:02:13,803::task::495::TaskManager.Task::(_debug) Task ae4a902b-0b1c-45ce-bc7a-c060a36e5290: moving from state init -> state preparing >Thread-15910045::DEBUG::2013-11-19 21:02:13,805::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`328d4193-1369-4730-b0bf-36ded5d047c8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910045::DEBUG::2013-11-19 21:02:13,805::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910045::DEBUG::2013-11-19 21:02:13,806::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910045::DEBUG::2013-11-19 21:02:13,807::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`328d4193-1369-4730-b0bf-36ded5d047c8`::Granted request >Thread-15910045::DEBUG::2013-11-19 21:02:13,808::task::495::TaskManager.Task::(_debug) Task ae4a902b-0b1c-45ce-bc7a-c060a36e5290: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910045::DEBUG::2013-11-19 21:02:13,808::task::495::TaskManager.Task::(_debug) Task ae4a902b-0b1c-45ce-bc7a-c060a36e5290: ref 1 aborting False >Thread-15910045::WARNING::2013-11-19 21:02:13,810::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910045::DEBUG::2013-11-19 21:02:13,811::task::495::TaskManager.Task::(_debug) Task ae4a902b-0b1c-45ce-bc7a-c060a36e5290: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910045::DEBUG::2013-11-19 21:02:13,812::task::495::TaskManager.Task::(_debug) Task ae4a902b-0b1c-45ce-bc7a-c060a36e5290: moving from state preparing -> state finished >Thread-15910045::DEBUG::2013-11-19 21:02:13,813::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910045::DEBUG::2013-11-19 21:02:13,813::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910045::DEBUG::2013-11-19 21:02:13,814::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910045::DEBUG::2013-11-19 21:02:13,815::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910045::DEBUG::2013-11-19 21:02:13,815::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910045::DEBUG::2013-11-19 21:02:13,816::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910045::DEBUG::2013-11-19 21:02:13,816::task::495::TaskManager.Task::(_debug) Task ae4a902b-0b1c-45ce-bc7a-c060a36e5290: ref 0 aborting False >Thread-15910045::INFO::2013-11-19 21:02:13,817::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910046::INFO::2013-11-19 21:02:14,283::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910046::DEBUG::2013-11-19 21:02:14,284::task::495::TaskManager.Task::(_debug) Task 3529a14c-9fc4-41fe-8924-0fab27f50e1c: moving from state init -> state preparing >Thread-15910046::DEBUG::2013-11-19 21:02:14,284::task::495::TaskManager.Task::(_debug) Task 3529a14c-9fc4-41fe-8924-0fab27f50e1c: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000343084335327', 'lastCheck': 1384894926.70735, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137591362', 'lastCheck': 1384894933.360806, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894928.354538, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000740051269531', 'lastCheck': 1384894926.1610529, 'valid': True, 'code': 0}} >Thread-15910046::DEBUG::2013-11-19 21:02:14,285::task::495::TaskManager.Task::(_debug) Task 3529a14c-9fc4-41fe-8924-0fab27f50e1c: moving from state preparing -> state finished >Thread-15910046::DEBUG::2013-11-19 21:02:14,286::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910046::DEBUG::2013-11-19 21:02:14,286::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910046::DEBUG::2013-11-19 21:02:14,287::task::495::TaskManager.Task::(_debug) Task 3529a14c-9fc4-41fe-8924-0fab27f50e1c: ref 0 aborting False >Thread-15910046::INFO::2013-11-19 21:02:14,288::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000343084335327', 'lastCheck': 1384894926.70735, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137591362', 'lastCheck': 1384894933.360806, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894928.354538, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000740051269531', 'lastCheck': 1384894926.1610529, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:02:15,603::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:02:16,136::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0752066 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:02:18,151::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:02:18,394::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:02:18,606::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0455872 s, 22.5 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:02:20,616::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:02:21,142::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0827538 s, 12.4 MB/s\n'; <rc> = 0 >Thread-4922911::INFO::2013-11-19 21:02:22,838::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-4922911::DEBUG::2013-11-19 21:02:22,839::task::495::TaskManager.Task::(_debug) Task 8210e323-d7c1-4275-a9f2-e9de178208a9: moving from state init -> state preparing >Thread-4922911::DEBUG::2013-11-19 21:02:22,840::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`087fca4a-3fdf-497e-ae98-ed3d823d11b6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-4922911::DEBUG::2013-11-19 21:02:22,841::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-4922911::DEBUG::2013-11-19 21:02:22,842::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-4922911::DEBUG::2013-11-19 21:02:22,842::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`087fca4a-3fdf-497e-ae98-ed3d823d11b6`::Granted request >Thread-4922911::DEBUG::2013-11-19 21:02:22,843::task::495::TaskManager.Task::(_debug) Task 8210e323-d7c1-4275-a9f2-e9de178208a9: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-4922911::DEBUG::2013-11-19 21:02:22,844::task::495::TaskManager.Task::(_debug) Task 8210e323-d7c1-4275-a9f2-e9de178208a9: ref 1 aborting False >Thread-4922911::DEBUG::2013-11-19 21:02:22,845::task::495::TaskManager.Task::(_debug) Task 8210e323-d7c1-4275-a9f2-e9de178208a9: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-4922911::DEBUG::2013-11-19 21:02:22,845::task::495::TaskManager.Task::(_debug) Task 8210e323-d7c1-4275-a9f2-e9de178208a9: moving from state preparing -> state finished >Thread-4922911::DEBUG::2013-11-19 21:02:22,846::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-4922911::DEBUG::2013-11-19 21:02:22,846::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4922911::DEBUG::2013-11-19 21:02:22,847::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-4922911::DEBUG::2013-11-19 21:02:22,847::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-4922911::DEBUG::2013-11-19 21:02:22,848::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-4922911::DEBUG::2013-11-19 21:02:22,848::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-4922911::DEBUG::2013-11-19 21:02:22,849::task::495::TaskManager.Task::(_debug) Task 8210e323-d7c1-4275-a9f2-e9de178208a9: ref 0 aborting False >Thread-4922911::INFO::2013-11-19 21:02:22,850::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-4922911::INFO::2013-11-19 21:02:22,850::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-4922911::DEBUG::2013-11-19 21:02:22,851::task::495::TaskManager.Task::(_debug) Task c42eb4e8-df32-4c13-8338-a6b197446d10: moving from state init -> state preparing >Thread-4922911::DEBUG::2013-11-19 21:02:22,852::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`d66e814d-59a7-492c-8e10-8997e272fce1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-4922911::DEBUG::2013-11-19 21:02:22,852::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-4922911::DEBUG::2013-11-19 21:02:22,853::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-4922911::DEBUG::2013-11-19 21:02:22,853::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`d66e814d-59a7-492c-8e10-8997e272fce1`::Granted request >Thread-4922911::DEBUG::2013-11-19 21:02:22,854::task::495::TaskManager.Task::(_debug) Task c42eb4e8-df32-4c13-8338-a6b197446d10: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-4922911::DEBUG::2013-11-19 21:02:22,855::task::495::TaskManager.Task::(_debug) Task c42eb4e8-df32-4c13-8338-a6b197446d10: ref 1 aborting False >Thread-4922911::DEBUG::2013-11-19 21:02:22,855::task::495::TaskManager.Task::(_debug) Task c42eb4e8-df32-4c13-8338-a6b197446d10: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-4922911::DEBUG::2013-11-19 21:02:22,856::task::495::TaskManager.Task::(_debug) Task c42eb4e8-df32-4c13-8338-a6b197446d10: moving from state preparing -> state finished >Thread-4922911::DEBUG::2013-11-19 21:02:22,856::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-4922911::DEBUG::2013-11-19 21:02:22,857::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4922911::DEBUG::2013-11-19 21:02:22,858::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-4922911::DEBUG::2013-11-19 21:02:22,858::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-4922911::DEBUG::2013-11-19 21:02:22,859::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-4922911::DEBUG::2013-11-19 21:02:22,859::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-4922911::DEBUG::2013-11-19 21:02:22,860::task::495::TaskManager.Task::(_debug) Task c42eb4e8-df32-4c13-8338-a6b197446d10: ref 0 aborting False >Thread-4922911::INFO::2013-11-19 21:02:22,860::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-4922911::INFO::2013-11-19 21:02:22,861::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-4922911::DEBUG::2013-11-19 21:02:22,862::task::495::TaskManager.Task::(_debug) Task 74bc7a8f-e2d8-4598-b287-06594999dc50: moving from state init -> state preparing >Thread-4922911::DEBUG::2013-11-19 21:02:22,863::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`e31dccc7-daad-4ab2-bd70-106c81ba36b8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-4922911::DEBUG::2013-11-19 21:02:22,863::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-4922911::DEBUG::2013-11-19 21:02:22,864::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-4922911::DEBUG::2013-11-19 21:02:22,864::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`e31dccc7-daad-4ab2-bd70-106c81ba36b8`::Granted request >Thread-4922911::DEBUG::2013-11-19 21:02:22,865::task::495::TaskManager.Task::(_debug) Task 74bc7a8f-e2d8-4598-b287-06594999dc50: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-4922911::DEBUG::2013-11-19 21:02:22,866::task::495::TaskManager.Task::(_debug) Task 74bc7a8f-e2d8-4598-b287-06594999dc50: ref 1 aborting False >Thread-4922911::DEBUG::2013-11-19 21:02:22,866::task::495::TaskManager.Task::(_debug) Task 74bc7a8f-e2d8-4598-b287-06594999dc50: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-4922911::DEBUG::2013-11-19 21:02:22,867::task::495::TaskManager.Task::(_debug) Task 74bc7a8f-e2d8-4598-b287-06594999dc50: moving from state preparing -> state finished >Thread-4922911::DEBUG::2013-11-19 21:02:22,867::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-4922911::DEBUG::2013-11-19 21:02:22,868::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4922911::DEBUG::2013-11-19 21:02:22,868::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-4922911::DEBUG::2013-11-19 21:02:22,869::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-4922911::DEBUG::2013-11-19 21:02:22,870::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-4922911::DEBUG::2013-11-19 21:02:22,870::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-4922911::DEBUG::2013-11-19 21:02:22,871::task::495::TaskManager.Task::(_debug) Task 74bc7a8f-e2d8-4598-b287-06594999dc50: ref 0 aborting False >Thread-4922911::INFO::2013-11-19 21:02:22,871::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-4922911::INFO::2013-11-19 21:02:22,872::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-4922911::DEBUG::2013-11-19 21:02:22,873::task::495::TaskManager.Task::(_debug) Task 0e604aac-817f-4d4c-bebe-d3a3322605b3: moving from state init -> state preparing >Thread-4922911::DEBUG::2013-11-19 21:02:22,874::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`d5c203e5-c864-4f86-9866-95e57fc3e88e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-4922911::DEBUG::2013-11-19 21:02:22,874::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-4922911::DEBUG::2013-11-19 21:02:22,875::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-4922911::DEBUG::2013-11-19 21:02:22,875::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`d5c203e5-c864-4f86-9866-95e57fc3e88e`::Granted request >Thread-4922911::DEBUG::2013-11-19 21:02:22,876::task::495::TaskManager.Task::(_debug) Task 0e604aac-817f-4d4c-bebe-d3a3322605b3: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-4922911::DEBUG::2013-11-19 21:02:22,877::task::495::TaskManager.Task::(_debug) Task 0e604aac-817f-4d4c-bebe-d3a3322605b3: ref 1 aborting False >Thread-4922911::DEBUG::2013-11-19 21:02:22,877::task::495::TaskManager.Task::(_debug) Task 0e604aac-817f-4d4c-bebe-d3a3322605b3: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-4922911::DEBUG::2013-11-19 21:02:22,878::task::495::TaskManager.Task::(_debug) Task 0e604aac-817f-4d4c-bebe-d3a3322605b3: moving from state preparing -> state finished >Thread-4922911::DEBUG::2013-11-19 21:02:22,879::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-4922911::DEBUG::2013-11-19 21:02:22,879::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4922911::DEBUG::2013-11-19 21:02:22,880::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-4922911::DEBUG::2013-11-19 21:02:22,880::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-4922911::DEBUG::2013-11-19 21:02:22,881::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-4922911::DEBUG::2013-11-19 21:02:22,881::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-4922911::DEBUG::2013-11-19 21:02:22,882::task::495::TaskManager.Task::(_debug) Task 0e604aac-817f-4d4c-bebe-d3a3322605b3: ref 0 aborting False >Thread-4922911::INFO::2013-11-19 21:02:22,883::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15909965::DEBUG::2013-11-19 21:02:23,157::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15868524::INFO::2013-11-19 21:02:23,411::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=e0628852-41b1-4535-8dda-ebc96cb00952 volUUID=cd4f9324-c91d-405b-983f-62afa4daa8ad) >Thread-15868524::DEBUG::2013-11-19 21:02:23,413::task::495::TaskManager.Task::(_debug) Task db0baf9b-0bc3-4c3f-b629-bb1c876e27f5: moving from state init -> state preparing >Thread-15868524::DEBUG::2013-11-19 21:02:23,423::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`434ccaff-4288-4c11-89b5-545956124eb8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15868524::DEBUG::2013-11-19 21:02:23,424::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15868524::DEBUG::2013-11-19 21:02:23,425::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15868524::DEBUG::2013-11-19 21:02:23,425::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`434ccaff-4288-4c11-89b5-545956124eb8`::Granted request >Thread-15868524::DEBUG::2013-11-19 21:02:23,427::task::495::TaskManager.Task::(_debug) Task db0baf9b-0bc3-4c3f-b629-bb1c876e27f5: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15868524::DEBUG::2013-11-19 21:02:23,428::task::495::TaskManager.Task::(_debug) Task db0baf9b-0bc3-4c3f-b629-bb1c876e27f5: ref 1 aborting False >Thread-15868524::DEBUG::2013-11-19 21:02:23,429::task::495::TaskManager.Task::(_debug) Task db0baf9b-0bc3-4c3f-b629-bb1c876e27f5: finished: {'truesize': '32346472448', 'apparentsize': '32346472448'} >Thread-15868524::DEBUG::2013-11-19 21:02:23,429::task::495::TaskManager.Task::(_debug) Task db0baf9b-0bc3-4c3f-b629-bb1c876e27f5: moving from state preparing -> state finished >Thread-15868524::DEBUG::2013-11-19 21:02:23,430::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15868524::DEBUG::2013-11-19 21:02:23,431::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15868524::DEBUG::2013-11-19 21:02:23,432::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15868524::DEBUG::2013-11-19 21:02:23,432::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15868524::DEBUG::2013-11-19 21:02:23,433::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15868524::DEBUG::2013-11-19 21:02:23,433::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15868524::DEBUG::2013-11-19 21:02:23,434::task::495::TaskManager.Task::(_debug) Task db0baf9b-0bc3-4c3f-b629-bb1c876e27f5: ref 0 aborting False >Thread-15868524::INFO::2013-11-19 21:02:23,435::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '32346472448', 'apparentsize': '32346472448'} >Dummy-15909965::DEBUG::2013-11-19 21:02:23,718::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0747645 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15910052::DEBUG::2013-11-19 21:02:23,928::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910052::INFO::2013-11-19 21:02:23,929::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910052::DEBUG::2013-11-19 21:02:23,929::task::495::TaskManager.Task::(_debug) Task c7b1d90e-ede8-4c9b-9758-fc0c23632976: moving from state init -> state preparing >Thread-15910052::DEBUG::2013-11-19 21:02:23,930::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910052::DEBUG::2013-11-19 21:02:23,930::task::495::TaskManager.Task::(_debug) Task c7b1d90e-ede8-4c9b-9758-fc0c23632976: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910052::DEBUG::2013-11-19 21:02:23,931::task::495::TaskManager.Task::(_debug) Task c7b1d90e-ede8-4c9b-9758-fc0c23632976: moving from state preparing -> state finished >Thread-15910052::DEBUG::2013-11-19 21:02:23,932::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910052::DEBUG::2013-11-19 21:02:23,932::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910052::DEBUG::2013-11-19 21:02:23,933::task::495::TaskManager.Task::(_debug) Task c7b1d90e-ede8-4c9b-9758-fc0c23632976: ref 0 aborting False >Thread-15910052::INFO::2013-11-19 21:02:23,933::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910053::DEBUG::2013-11-19 21:02:23,938::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910053::INFO::2013-11-19 21:02:23,939::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910053::DEBUG::2013-11-19 21:02:23,940::task::495::TaskManager.Task::(_debug) Task be2ecf13-ac9a-44dc-a9c7-6ec94b2dcb92: moving from state init -> state preparing >Thread-15910053::DEBUG::2013-11-19 21:02:23,941::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6db061ef-503b-4376-89f1-1acebec59819`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910053::DEBUG::2013-11-19 21:02:23,942::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910053::DEBUG::2013-11-19 21:02:23,942::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910053::DEBUG::2013-11-19 21:02:23,943::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6db061ef-503b-4376-89f1-1acebec59819`::Granted request >Thread-15910053::DEBUG::2013-11-19 21:02:23,944::task::495::TaskManager.Task::(_debug) Task be2ecf13-ac9a-44dc-a9c7-6ec94b2dcb92: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910053::DEBUG::2013-11-19 21:02:23,945::task::495::TaskManager.Task::(_debug) Task be2ecf13-ac9a-44dc-a9c7-6ec94b2dcb92: ref 1 aborting False >Thread-15910053::WARNING::2013-11-19 21:02:23,947::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910053::DEBUG::2013-11-19 21:02:23,948::task::495::TaskManager.Task::(_debug) Task be2ecf13-ac9a-44dc-a9c7-6ec94b2dcb92: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910053::DEBUG::2013-11-19 21:02:23,949::task::495::TaskManager.Task::(_debug) Task be2ecf13-ac9a-44dc-a9c7-6ec94b2dcb92: moving from state preparing -> state finished >Thread-15910053::DEBUG::2013-11-19 21:02:23,949::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910053::DEBUG::2013-11-19 21:02:23,950::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910053::DEBUG::2013-11-19 21:02:23,951::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910053::DEBUG::2013-11-19 21:02:23,951::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910053::DEBUG::2013-11-19 21:02:23,952::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910053::DEBUG::2013-11-19 21:02:23,953::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910053::DEBUG::2013-11-19 21:02:23,953::task::495::TaskManager.Task::(_debug) Task be2ecf13-ac9a-44dc-a9c7-6ec94b2dcb92: ref 0 aborting False >Thread-15910053::INFO::2013-11-19 21:02:23,954::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910054::INFO::2013-11-19 21:02:24,428::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910054::DEBUG::2013-11-19 21:02:24,429::task::495::TaskManager.Task::(_debug) Task fced4c13-5ec8-4956-9fcb-2f8398b3de43: moving from state init -> state preparing >Thread-15910054::DEBUG::2013-11-19 21:02:24,429::task::495::TaskManager.Task::(_debug) Task fced4c13-5ec8-4956-9fcb-2f8398b3de43: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000293016433716', 'lastCheck': 1384894936.820327, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00141501426697', 'lastCheck': 1384894943.4160161, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894938.396374, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000478029251099', 'lastCheck': 1384894936.274322, 'valid': True, 'code': 0}} >Thread-15910054::DEBUG::2013-11-19 21:02:24,430::task::495::TaskManager.Task::(_debug) Task fced4c13-5ec8-4956-9fcb-2f8398b3de43: moving from state preparing -> state finished >Thread-15910054::DEBUG::2013-11-19 21:02:24,431::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910054::DEBUG::2013-11-19 21:02:24,432::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910054::DEBUG::2013-11-19 21:02:24,432::task::495::TaskManager.Task::(_debug) Task fced4c13-5ec8-4956-9fcb-2f8398b3de43: ref 0 aborting False >Thread-15910054::INFO::2013-11-19 21:02:24,433::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000293016433716', 'lastCheck': 1384894936.820327, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00141501426697', 'lastCheck': 1384894943.4160161, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894938.396374, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000478029251099', 'lastCheck': 1384894936.274322, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:02:25,733::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:02:26,284::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0757299 s, 13.5 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:02:28,299::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:02:28,526::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:02:28,787::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0575639 s, 17.8 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:02:30,798::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:02:31,360::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0736069 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:02:33,375::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:02:33,908::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0713555 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15910060::DEBUG::2013-11-19 21:02:34,053::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910060::INFO::2013-11-19 21:02:34,054::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910060::DEBUG::2013-11-19 21:02:34,055::task::495::TaskManager.Task::(_debug) Task 5931f955-0efb-4587-a59c-1d7ecc4279f6: moving from state init -> state preparing >Thread-15910060::DEBUG::2013-11-19 21:02:34,055::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910060::DEBUG::2013-11-19 21:02:34,056::task::495::TaskManager.Task::(_debug) Task 5931f955-0efb-4587-a59c-1d7ecc4279f6: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910060::DEBUG::2013-11-19 21:02:34,056::task::495::TaskManager.Task::(_debug) Task 5931f955-0efb-4587-a59c-1d7ecc4279f6: moving from state preparing -> state finished >Thread-15910060::DEBUG::2013-11-19 21:02:34,056::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910060::DEBUG::2013-11-19 21:02:34,057::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910060::DEBUG::2013-11-19 21:02:34,057::task::495::TaskManager.Task::(_debug) Task 5931f955-0efb-4587-a59c-1d7ecc4279f6: ref 0 aborting False >Thread-15910060::INFO::2013-11-19 21:02:34,057::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910061::DEBUG::2013-11-19 21:02:34,062::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910061::INFO::2013-11-19 21:02:34,062::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910061::DEBUG::2013-11-19 21:02:34,063::task::495::TaskManager.Task::(_debug) Task ecac23a4-3ee5-40d3-9987-61ec58adb60b: moving from state init -> state preparing >Thread-15910061::DEBUG::2013-11-19 21:02:34,064::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`05bf7975-b615-4f40-8fad-d262d30de26c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910061::DEBUG::2013-11-19 21:02:34,065::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910061::DEBUG::2013-11-19 21:02:34,066::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910061::DEBUG::2013-11-19 21:02:34,066::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`05bf7975-b615-4f40-8fad-d262d30de26c`::Granted request >Thread-15910061::DEBUG::2013-11-19 21:02:34,067::task::495::TaskManager.Task::(_debug) Task ecac23a4-3ee5-40d3-9987-61ec58adb60b: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910061::DEBUG::2013-11-19 21:02:34,068::task::495::TaskManager.Task::(_debug) Task ecac23a4-3ee5-40d3-9987-61ec58adb60b: ref 1 aborting False >Thread-15910061::WARNING::2013-11-19 21:02:34,070::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910061::DEBUG::2013-11-19 21:02:34,070::task::495::TaskManager.Task::(_debug) Task ecac23a4-3ee5-40d3-9987-61ec58adb60b: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910061::DEBUG::2013-11-19 21:02:34,071::task::495::TaskManager.Task::(_debug) Task ecac23a4-3ee5-40d3-9987-61ec58adb60b: moving from state preparing -> state finished >Thread-15910061::DEBUG::2013-11-19 21:02:34,072::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910061::DEBUG::2013-11-19 21:02:34,072::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910061::DEBUG::2013-11-19 21:02:34,073::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910061::DEBUG::2013-11-19 21:02:34,073::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910061::DEBUG::2013-11-19 21:02:34,074::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910061::DEBUG::2013-11-19 21:02:34,074::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910061::DEBUG::2013-11-19 21:02:34,075::task::495::TaskManager.Task::(_debug) Task ecac23a4-3ee5-40d3-9987-61ec58adb60b: ref 0 aborting False >Thread-15910061::INFO::2013-11-19 21:02:34,076::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910062::INFO::2013-11-19 21:02:34,568::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910062::DEBUG::2013-11-19 21:02:34,569::task::495::TaskManager.Task::(_debug) Task f4441c1c-f5ca-419c-b207-0ec73b1625ac: moving from state init -> state preparing >Thread-15910062::DEBUG::2013-11-19 21:02:34,569::task::495::TaskManager.Task::(_debug) Task f4441c1c-f5ca-419c-b207-0ec73b1625ac: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000393152236938', 'lastCheck': 1384894946.965574, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00206685066223', 'lastCheck': 1384894953.59971, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894948.5350389, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000555992126465', 'lastCheck': 1384894946.400502, 'valid': True, 'code': 0}} >Thread-15910062::DEBUG::2013-11-19 21:02:34,570::task::495::TaskManager.Task::(_debug) Task f4441c1c-f5ca-419c-b207-0ec73b1625ac: moving from state preparing -> state finished >Thread-15910062::DEBUG::2013-11-19 21:02:34,571::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910062::DEBUG::2013-11-19 21:02:34,571::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910062::DEBUG::2013-11-19 21:02:34,572::task::495::TaskManager.Task::(_debug) Task f4441c1c-f5ca-419c-b207-0ec73b1625ac: ref 0 aborting False >Thread-15910062::INFO::2013-11-19 21:02:34,572::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000393152236938', 'lastCheck': 1384894946.965574, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00206685066223', 'lastCheck': 1384894953.59971, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894948.5350389, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000555992126465', 'lastCheck': 1384894946.400502, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:02:35,924::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:02:36,466::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0713031 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:02:38,481::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:02:38,701::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:02:39,028::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0878445 s, 11.7 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:02:41,043::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15909534::INFO::2013-11-19 21:02:41,300::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15909534::DEBUG::2013-11-19 21:02:41,303::task::495::TaskManager.Task::(_debug) Task 0c76e5fb-4faf-4f4e-aef9-99a5dae4aebd: moving from state init -> state preparing >Thread-15909534::DEBUG::2013-11-19 21:02:41,306::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f209bd64-ddba-4cfb-8858-04f3c851bfe1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15909534::DEBUG::2013-11-19 21:02:41,307::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15909534::DEBUG::2013-11-19 21:02:41,308::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15909534::DEBUG::2013-11-19 21:02:41,309::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f209bd64-ddba-4cfb-8858-04f3c851bfe1`::Granted request >Thread-15909534::DEBUG::2013-11-19 21:02:41,310::task::495::TaskManager.Task::(_debug) Task 0c76e5fb-4faf-4f4e-aef9-99a5dae4aebd: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15909534::DEBUG::2013-11-19 21:02:41,311::task::495::TaskManager.Task::(_debug) Task 0c76e5fb-4faf-4f4e-aef9-99a5dae4aebd: ref 1 aborting False >Thread-15909534::DEBUG::2013-11-19 21:02:41,311::task::495::TaskManager.Task::(_debug) Task 0c76e5fb-4faf-4f4e-aef9-99a5dae4aebd: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15909534::DEBUG::2013-11-19 21:02:41,312::task::495::TaskManager.Task::(_debug) Task 0c76e5fb-4faf-4f4e-aef9-99a5dae4aebd: moving from state preparing -> state finished >Thread-15909534::DEBUG::2013-11-19 21:02:41,313::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15909534::DEBUG::2013-11-19 21:02:41,314::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15909534::DEBUG::2013-11-19 21:02:41,314::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15909534::DEBUG::2013-11-19 21:02:41,315::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15909534::DEBUG::2013-11-19 21:02:41,316::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15909534::DEBUG::2013-11-19 21:02:41,316::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15909534::DEBUG::2013-11-19 21:02:41,317::task::495::TaskManager.Task::(_debug) Task 0c76e5fb-4faf-4f4e-aef9-99a5dae4aebd: ref 0 aborting False >Thread-15909534::INFO::2013-11-19 21:02:41,318::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15909965::DEBUG::2013-11-19 21:02:41,636::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0835639 s, 12.3 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:02:43,646::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15910068::DEBUG::2013-11-19 21:02:44,196::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910068::INFO::2013-11-19 21:02:44,199::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910068::DEBUG::2013-11-19 21:02:44,200::task::495::TaskManager.Task::(_debug) Task a79bfb32-5d40-49b9-baee-afd6d347a6e7: moving from state init -> state preparing >Thread-15910068::DEBUG::2013-11-19 21:02:44,205::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910068::DEBUG::2013-11-19 21:02:44,209::task::495::TaskManager.Task::(_debug) Task a79bfb32-5d40-49b9-baee-afd6d347a6e7: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910068::DEBUG::2013-11-19 21:02:44,212::task::495::TaskManager.Task::(_debug) Task a79bfb32-5d40-49b9-baee-afd6d347a6e7: moving from state preparing -> state finished >Thread-15910068::DEBUG::2013-11-19 21:02:44,214::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910068::DEBUG::2013-11-19 21:02:44,215::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910068::DEBUG::2013-11-19 21:02:44,217::task::495::TaskManager.Task::(_debug) Task a79bfb32-5d40-49b9-baee-afd6d347a6e7: ref 0 aborting False >Thread-15910068::INFO::2013-11-19 21:02:44,218::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910069::DEBUG::2013-11-19 21:02:44,226::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910069::INFO::2013-11-19 21:02:44,228::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910069::DEBUG::2013-11-19 21:02:44,229::task::495::TaskManager.Task::(_debug) Task e89a35e4-e4f6-4f55-b541-aaa2a965b13f: moving from state init -> state preparing >Thread-15910069::DEBUG::2013-11-19 21:02:44,231::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`28fd7dcc-dc85-47a0-800c-dc5a0a2f5bf9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910069::DEBUG::2013-11-19 21:02:44,232::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910069::DEBUG::2013-11-19 21:02:44,235::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910069::DEBUG::2013-11-19 21:02:44,236::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`28fd7dcc-dc85-47a0-800c-dc5a0a2f5bf9`::Granted request >Thread-15910069::DEBUG::2013-11-19 21:02:44,239::task::495::TaskManager.Task::(_debug) Task e89a35e4-e4f6-4f55-b541-aaa2a965b13f: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910069::DEBUG::2013-11-19 21:02:44,240::task::495::TaskManager.Task::(_debug) Task e89a35e4-e4f6-4f55-b541-aaa2a965b13f: ref 1 aborting False >Thread-15910069::WARNING::2013-11-19 21:02:44,243::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910069::DEBUG::2013-11-19 21:02:44,246::task::495::TaskManager.Task::(_debug) Task e89a35e4-e4f6-4f55-b541-aaa2a965b13f: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910069::DEBUG::2013-11-19 21:02:44,247::task::495::TaskManager.Task::(_debug) Task e89a35e4-e4f6-4f55-b541-aaa2a965b13f: moving from state preparing -> state finished >Thread-15910069::DEBUG::2013-11-19 21:02:44,248::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910069::DEBUG::2013-11-19 21:02:44,249::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910069::DEBUG::2013-11-19 21:02:44,250::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910069::DEBUG::2013-11-19 21:02:44,252::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910069::DEBUG::2013-11-19 21:02:44,253::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910069::DEBUG::2013-11-19 21:02:44,254::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910069::DEBUG::2013-11-19 21:02:44,255::task::495::TaskManager.Task::(_debug) Task e89a35e4-e4f6-4f55-b541-aaa2a965b13f: ref 0 aborting False >Thread-15910069::INFO::2013-11-19 21:02:44,256::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15909965::DEBUG::2013-11-19 21:02:44,265::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0730093 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15910070::INFO::2013-11-19 21:02:44,753::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910070::DEBUG::2013-11-19 21:02:44,754::task::495::TaskManager.Task::(_debug) Task cb557e13-30e4-4861-aae1-5153e75622d2: moving from state init -> state preparing >Thread-15910070::DEBUG::2013-11-19 21:02:44,754::task::495::TaskManager.Task::(_debug) Task cb557e13-30e4-4861-aae1-5153e75622d2: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000346899032593', 'lastCheck': 1384894957.149045, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00153994560242', 'lastCheck': 1384894963.896831, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894958.709393, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000497102737427', 'lastCheck': 1384894956.5929899, 'valid': True, 'code': 0}} >Thread-15910070::DEBUG::2013-11-19 21:02:44,755::task::495::TaskManager.Task::(_debug) Task cb557e13-30e4-4861-aae1-5153e75622d2: moving from state preparing -> state finished >Thread-15910070::DEBUG::2013-11-19 21:02:44,756::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910070::DEBUG::2013-11-19 21:02:44,756::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910070::DEBUG::2013-11-19 21:02:44,757::task::495::TaskManager.Task::(_debug) Task cb557e13-30e4-4861-aae1-5153e75622d2: ref 0 aborting False >Thread-15910070::INFO::2013-11-19 21:02:44,758::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000346899032593', 'lastCheck': 1384894957.149045, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00153994560242', 'lastCheck': 1384894963.896831, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894958.709393, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000497102737427', 'lastCheck': 1384894956.5929899, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:02:46,280::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:02:46,822::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0963337 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:02:48,838::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:02:49,064::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:02:49,343::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0716114 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:02:51,359::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:02:51,895::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0752084 s, 13.6 MB/s\n'; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:02:53,514::vm::525::vm.Vm::(_rtcUpdate) vmId=`2afdc298-b10b-4bd1-8df2-2a401efd4f9c`::new rtc offset 2 >Dummy-15909965::DEBUG::2013-11-19 21:02:53,910::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15910076::DEBUG::2013-11-19 21:02:54,368::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910076::INFO::2013-11-19 21:02:54,370::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910076::DEBUG::2013-11-19 21:02:54,370::task::495::TaskManager.Task::(_debug) Task d6befbd3-ad71-44ce-9ee9-d9900973f58d: moving from state init -> state preparing >Thread-15910076::DEBUG::2013-11-19 21:02:54,371::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910076::DEBUG::2013-11-19 21:02:54,372::task::495::TaskManager.Task::(_debug) Task d6befbd3-ad71-44ce-9ee9-d9900973f58d: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910076::DEBUG::2013-11-19 21:02:54,373::task::495::TaskManager.Task::(_debug) Task d6befbd3-ad71-44ce-9ee9-d9900973f58d: moving from state preparing -> state finished >Thread-15910076::DEBUG::2013-11-19 21:02:54,374::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910076::DEBUG::2013-11-19 21:02:54,374::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910076::DEBUG::2013-11-19 21:02:54,375::task::495::TaskManager.Task::(_debug) Task d6befbd3-ad71-44ce-9ee9-d9900973f58d: ref 0 aborting False >Thread-15910076::INFO::2013-11-19 21:02:54,376::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910077::DEBUG::2013-11-19 21:02:54,381::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910077::INFO::2013-11-19 21:02:54,382::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910077::DEBUG::2013-11-19 21:02:54,383::task::495::TaskManager.Task::(_debug) Task c07175c6-f7cd-41b0-bc52-dead2a720c09: moving from state init -> state preparing >Thread-15910077::DEBUG::2013-11-19 21:02:54,385::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`27298ef2-63c2-4ff4-b370-f08e92e416fc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910077::DEBUG::2013-11-19 21:02:54,386::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910077::DEBUG::2013-11-19 21:02:54,387::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910077::DEBUG::2013-11-19 21:02:54,388::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`27298ef2-63c2-4ff4-b370-f08e92e416fc`::Granted request >Thread-15910077::DEBUG::2013-11-19 21:02:54,390::task::495::TaskManager.Task::(_debug) Task c07175c6-f7cd-41b0-bc52-dead2a720c09: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910077::DEBUG::2013-11-19 21:02:54,391::task::495::TaskManager.Task::(_debug) Task c07175c6-f7cd-41b0-bc52-dead2a720c09: ref 1 aborting False >Thread-15910077::WARNING::2013-11-19 21:02:54,393::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910077::DEBUG::2013-11-19 21:02:54,395::task::495::TaskManager.Task::(_debug) Task c07175c6-f7cd-41b0-bc52-dead2a720c09: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910077::DEBUG::2013-11-19 21:02:54,396::task::495::TaskManager.Task::(_debug) Task c07175c6-f7cd-41b0-bc52-dead2a720c09: moving from state preparing -> state finished >Thread-15910077::DEBUG::2013-11-19 21:02:54,397::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910077::DEBUG::2013-11-19 21:02:54,397::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910077::DEBUG::2013-11-19 21:02:54,398::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910077::DEBUG::2013-11-19 21:02:54,399::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910077::DEBUG::2013-11-19 21:02:54,400::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910077::DEBUG::2013-11-19 21:02:54,401::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910077::DEBUG::2013-11-19 21:02:54,402::task::495::TaskManager.Task::(_debug) Task c07175c6-f7cd-41b0-bc52-dead2a720c09: ref 0 aborting False >Thread-15910077::INFO::2013-11-19 21:02:54,403::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15909965::DEBUG::2013-11-19 21:02:54,502::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.105244 s, 9.7 MB/s\n'; <rc> = 0 >Thread-15910078::INFO::2013-11-19 21:02:55,091::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910078::DEBUG::2013-11-19 21:02:55,091::task::495::TaskManager.Task::(_debug) Task b1e746ae-fb6c-4069-b90b-eba5213a3bef: moving from state init -> state preparing >Thread-15910078::DEBUG::2013-11-19 21:02:55,092::task::495::TaskManager.Task::(_debug) Task b1e746ae-fb6c-4069-b90b-eba5213a3bef: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000378131866455', 'lastCheck': 1384894967.4729011, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000960111618042', 'lastCheck': 1384894974.165947, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894969.068589, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000412940979004', 'lastCheck': 1384894966.890193, 'valid': True, 'code': 0}} >Thread-15910078::DEBUG::2013-11-19 21:02:55,092::task::495::TaskManager.Task::(_debug) Task b1e746ae-fb6c-4069-b90b-eba5213a3bef: moving from state preparing -> state finished >Thread-15910078::DEBUG::2013-11-19 21:02:55,093::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910078::DEBUG::2013-11-19 21:02:55,093::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910078::DEBUG::2013-11-19 21:02:55,093::task::495::TaskManager.Task::(_debug) Task b1e746ae-fb6c-4069-b90b-eba5213a3bef: ref 0 aborting False >Thread-15910078::INFO::2013-11-19 21:02:55,094::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000378131866455', 'lastCheck': 1384894967.4729011, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000960111618042', 'lastCheck': 1384894974.165947, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894969.068589, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000412940979004', 'lastCheck': 1384894966.890193, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:02:56,515::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:02:57,035::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0956138 s, 10.7 MB/s\n'; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:02:58,657::libvirtvm::1401::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`2afdc298-b10b-4bd1-8df2-2a401efd4f9c`::event Saved detail 0 opaque None >Thread-15868525::ERROR::2013-11-19 21:02:58,977::guestIF::246::vm.Vm::(run) vmId=`2afdc298-b10b-4bd1-8df2-2a401efd4f9c`::Run exception: None >Traceback (most recent call last): > File "/usr/share/vdsm/guestIF.py", line 238, in run > File "/usr/share/vdsm/guestIF.py", line 214, in _readLine > File "/usr/share/vdsm/guestIF.py", line 200, in _readBuffer >error: [Errno 104] Connection reset by peer >Dummy-15909965::DEBUG::2013-11-19 21:02:59,051::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:02:59,288::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15868524::DEBUG::2013-11-19 21:02:59,536::libvirtvm::288::vm.Vm::(handleStatsException) vmId=`2afdc298-b10b-4bd1-8df2-2a401efd4f9c`::VM not found, moving to Down >Traceback (most recent call last): > File "/usr/share/vdsm/utils.py", line 373, in collect > File "/usr/share/vdsm/utils.py", line 250, in __call__ > File "/usr/share/vdsm/libvirtvm.py", line 161, in _sampleNet > File "/usr/share/vdsm/libvirtvm.py", line 445, in f > File "/usr/share/vdsm/libvirtconnection.py", line 63, in wrapper > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1734, in interfaceStats >libvirtError: Domain not found: no domain with matching uuid '2afdc298-b10b-4bd1-8df2-2a401efd4f9c' >libvirtEventLoop::DEBUG::2013-11-19 21:02:59,536::libvirtvm::1401::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`2afdc298-b10b-4bd1-8df2-2a401efd4f9c`::event Stopped detail 0 opaque None >libvirtEventLoop::INFO::2013-11-19 21:02:59,539::vm::445::vm.Vm::(_onQemuDeath) vmId=`2afdc298-b10b-4bd1-8df2-2a401efd4f9c`::underlying process disconnected >libvirtEventLoop::INFO::2013-11-19 21:02:59,540::libvirtvm::1304::vm.Vm::(releaseVm) vmId=`2afdc298-b10b-4bd1-8df2-2a401efd4f9c`::Release VM resources >Thread-15868524::DEBUG::2013-11-19 21:02:59,549::vm::742::vm.Vm::(setDownStatus) vmId=`2afdc298-b10b-4bd1-8df2-2a401efd4f9c`::Changed state to Down: Domain not found: no domain with matching uuid '2afdc298-b10b-4bd1-8df2-2a401efd4f9c' >Thread-15868524::DEBUG::2013-11-19 21:02:59,550::utils::314::vm.Vm::(stop) vmId=`2afdc298-b10b-4bd1-8df2-2a401efd4f9c`::Stop statistics collection >libvirtEventLoop::DEBUG::2013-11-19 21:02:59,550::utils::314::vm.Vm::(stop) vmId=`2afdc298-b10b-4bd1-8df2-2a401efd4f9c`::Stop statistics collection >libvirtEventLoop::WARNING::2013-11-19 21:02:59,554::libvirtvm::1315::vm.Vm::(releaseVm) vmId=`2afdc298-b10b-4bd1-8df2-2a401efd4f9c`::Traceback (most recent call last): > File "/usr/share/vdsm/libvirtvm.py", line 1312, in releaseVm > File "/usr/share/vdsm/libvirtvm.py", line 445, in f > File "/usr/share/vdsm/libvirtconnection.py", line 63, in wrapper > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 672, in destroy >libvirtError: Domain not found: no domain with matching uuid '2afdc298-b10b-4bd1-8df2-2a401efd4f9c' > >libvirtEventLoop::DEBUG::2013-11-19 21:02:59,556::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /sbin/service ksmtuned retune' (cwd None) >Thread-15868524::DEBUG::2013-11-19 21:02:59,744::utils::345::vm.Vm::(run) vmId=`2afdc298-b10b-4bd1-8df2-2a401efd4f9c`::Stats thread finished >Dummy-15909965::DEBUG::2013-11-19 21:02:59,791::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.283326 s, 3.6 MB/s\n'; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:02:59,996::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:02:59,999::vm::718::vm.Vm::(_cleanup) vmId=`2afdc298-b10b-4bd1-8df2-2a401efd4f9c`::Drive None cleanup >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,000::vm::718::vm.Vm::(_cleanup) vmId=`2afdc298-b10b-4bd1-8df2-2a401efd4f9c`::Drive /rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/e0628852-41b1-4535-8dda-ebc96cb00952/cd4f9324-c91d-405b-983f-62afa4daa8ad cleanup >libvirtEventLoop::INFO::2013-11-19 21:03:00,002::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: teardownVolume, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=e0628852-41b1-4535-8dda-ebc96cb00952 volUUID=cd4f9324-c91d-405b-983f-62afa4daa8ad) >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,002::task::495::TaskManager.Task::(_debug) Task f4526a4c-a2e3-4906-b3d5-ef95f9fcec69: moving from state init -> state preparing >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,004::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`3cd77753-67e2-446a-ac8e-83b58638c447`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1727' at 'public_teardownVolume' >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,005::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,006::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,006::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`3cd77753-67e2-446a-ac8e-83b58638c447`::Granted request >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,007::task::495::TaskManager.Task::(_debug) Task f4526a4c-a2e3-4906-b3d5-ef95f9fcec69: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,008::task::495::TaskManager.Task::(_debug) Task f4526a4c-a2e3-4906-b3d5-ef95f9fcec69: ref 1 aborting False >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,009::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.e0628852-41b1-4535-8dda-ebc96cb00952' >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,010::resourceManager::530::ResourceManager::(releaseResource) Released resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.e0628852-41b1-4535-8dda-ebc96cb00952' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,010::resourceManager::535::ResourceManager::(releaseResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.e0628852-41b1-4535-8dda-ebc96cb00952' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,011::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.cd4f9324-c91d-405b-983f-62afa4daa8ad' >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,012::resourceManager::530::ResourceManager::(releaseResource) Released resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.cd4f9324-c91d-405b-983f-62afa4daa8ad' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,013::resourceManager::535::ResourceManager::(releaseResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.cd4f9324-c91d-405b-983f-62afa4daa8ad' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,013::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.cd4f9324-c91d-405b-983f-62afa4daa8ad', Clearing records. >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,014::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.e0628852-41b1-4535-8dda-ebc96cb00952', Clearing records. >libvirtEventLoop::INFO::2013-11-19 21:03:00,015::blockVolume::386::Storage.Volume::(teardown) Tearing down volume 6b892067-ac7e-4039-a7be-5b7387525b16/cd4f9324-c91d-405b-983f-62afa4daa8ad justme False >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,015::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.cd4f9324-c91d-405b-983f-62afa4daa8ad' >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,016::resourceManager::530::ResourceManager::(releaseResource) Released resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.cd4f9324-c91d-405b-983f-62afa4daa8ad' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,016::resourceManager::535::ResourceManager::(releaseResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.cd4f9324-c91d-405b-983f-62afa4daa8ad' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,018::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --autobackup n --available n 6b892067-ac7e-4039-a7be-5b7387525b16/cd4f9324-c91d-405b-983f-62afa4daa8ad' (cwd None) >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,751::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,753::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,754::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,754::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.cd4f9324-c91d-405b-983f-62afa4daa8ad', Clearing records. >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,755::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:03:00,757::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15910083::DEBUG::2013-11-19 21:03:01,320::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('2afdc298-b10b-4bd1-8df2-2a401efd4f9c',) {} >Thread-15910083::DEBUG::2013-11-19 21:03:01,322::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering down', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19979', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': u'5901', 'timeOffset': 2L, 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:11', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}, u'vnet1': {'macAddr': u'00:1a:4a:a8:64:12', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '100', 'name': u'vnet1'}}, 'vmId': '2afdc298-b10b-4bd1-8df2-2a401efd4f9c', 'monitorResponse': '0', 'cpuUser': '19.41', 'disks': {u'hda': {'readLatency': '0', 'apparentsize': '32346472448', 'writeLatency': '360629', 'imageID': 'e0628852-41b1-4535-8dda-ebc96cb00952', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '32346472448', 'writeRate': '17860.71'}}, 'boot': 'c', 'statsAge': '2.04', 'cpuIdle': '80.59', 'elapsedTime': '128783', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >libvirtEventLoop::DEBUG::2013-11-19 21:03:01,519::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:03:01,775::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:03:01,776::task::495::TaskManager.Task::(_debug) Task f4526a4c-a2e3-4906-b3d5-ef95f9fcec69: finished: None >libvirtEventLoop::DEBUG::2013-11-19 21:03:01,776::task::495::TaskManager.Task::(_debug) Task f4526a4c-a2e3-4906-b3d5-ef95f9fcec69: moving from state preparing -> state finished >libvirtEventLoop::DEBUG::2013-11-19 21:03:01,776::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >libvirtEventLoop::DEBUG::2013-11-19 21:03:01,777::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >libvirtEventLoop::DEBUG::2013-11-19 21:03:01,777::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >libvirtEventLoop::DEBUG::2013-11-19 21:03:01,778::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:03:01,778::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:03:01,778::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >libvirtEventLoop::DEBUG::2013-11-19 21:03:01,779::task::495::TaskManager.Task::(_debug) Task f4526a4c-a2e3-4906-b3d5-ef95f9fcec69: ref 0 aborting False >libvirtEventLoop::INFO::2013-11-19 21:03:01,779::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: teardownVolume, Return response: {'status': {'message': 'OK', 'code': 0}} >libvirtEventLoop::DEBUG::2013-11-19 21:03:01,781::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/libexec/vdsm/hooks/after_vm_destroy/50_vhostmd' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:01,955::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:02,593::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0726782 s, 14.1 MB/s\n'; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:03:02,690::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >libvirtEventLoop::INFO::2013-11-19 21:03:02,691::hooks::51::root::(_runHooksDir) >libvirtEventLoop::DEBUG::2013-11-19 21:03:02,698::vm::742::vm.Vm::(setDownStatus) vmId=`2afdc298-b10b-4bd1-8df2-2a401efd4f9c`::Changed state to Down: User shut down >libvirtEventLoop::DEBUG::2013-11-19 21:03:02,698::utils::314::vm.Vm::(stop) vmId=`2afdc298-b10b-4bd1-8df2-2a401efd4f9c`::Stop statistics collection >Thread-15910086::DEBUG::2013-11-19 21:03:03,380::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('2afdc298-b10b-4bd1-8df2-2a401efd4f9c',) {} >Thread-15910086::DEBUG::2013-11-19 21:03:03,380::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Down', 'timeOffset': 2L, 'vmId': '2afdc298-b10b-4bd1-8df2-2a401efd4f9c', 'exitMessage': 'User shut down', 'exitCode': 0}]} >Thread-15910087::DEBUG::2013-11-19 21:03:03,465::clientIF::54::vds::(wrapper) [192.168.100.135]::call destroy with ('2afdc298-b10b-4bd1-8df2-2a401efd4f9c',) {} >Thread-15910087::INFO::2013-11-19 21:03:03,466::clientIF::450::vds::(destroy) vmContainerLock acquired by vm 2afdc298-b10b-4bd1-8df2-2a401efd4f9c >Thread-15910087::DEBUG::2013-11-19 21:03:03,467::libvirtvm::1344::vm.Vm::(destroy) vmId=`2afdc298-b10b-4bd1-8df2-2a401efd4f9c`::destroy Called >Thread-15910087::DEBUG::2013-11-19 21:03:03,467::libvirtvm::1339::vm.Vm::(deleteVm) vmId=`2afdc298-b10b-4bd1-8df2-2a401efd4f9c`::Total desktops after destroy of 2afdc298-b10b-4bd1-8df2-2a401efd4f9c is 2 >Thread-15910087::DEBUG::2013-11-19 21:03:03,468::clientIF::59::vds::(wrapper) return destroy with {'status': {'message': 'Machine destroyed', 'code': 0}} >Thread-15910088::DEBUG::2013-11-19 21:03:04,541::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910088::INFO::2013-11-19 21:03:04,543::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910088::DEBUG::2013-11-19 21:03:04,544::task::495::TaskManager.Task::(_debug) Task 8df135f6-4ef4-4235-9123-f1de00141cf6: moving from state init -> state preparing >Thread-15910088::DEBUG::2013-11-19 21:03:04,544::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910088::DEBUG::2013-11-19 21:03:04,545::task::495::TaskManager.Task::(_debug) Task 8df135f6-4ef4-4235-9123-f1de00141cf6: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910088::DEBUG::2013-11-19 21:03:04,546::task::495::TaskManager.Task::(_debug) Task 8df135f6-4ef4-4235-9123-f1de00141cf6: moving from state preparing -> state finished >Thread-15910088::DEBUG::2013-11-19 21:03:04,546::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910088::DEBUG::2013-11-19 21:03:04,547::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910088::DEBUG::2013-11-19 21:03:04,547::task::495::TaskManager.Task::(_debug) Task 8df135f6-4ef4-4235-9123-f1de00141cf6: ref 0 aborting False >Thread-15910088::INFO::2013-11-19 21:03:04,548::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910089::DEBUG::2013-11-19 21:03:04,552::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910089::INFO::2013-11-19 21:03:04,553::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910089::DEBUG::2013-11-19 21:03:04,554::task::495::TaskManager.Task::(_debug) Task e93591a6-de64-47fd-bcbf-1d976e1dd4f7: moving from state init -> state preparing >Thread-15910089::DEBUG::2013-11-19 21:03:04,555::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`cb8ceef9-13fc-4370-b76a-354716efc87b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910089::DEBUG::2013-11-19 21:03:04,556::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910089::DEBUG::2013-11-19 21:03:04,557::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910089::DEBUG::2013-11-19 21:03:04,557::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`cb8ceef9-13fc-4370-b76a-354716efc87b`::Granted request >Thread-15910089::DEBUG::2013-11-19 21:03:04,558::task::495::TaskManager.Task::(_debug) Task e93591a6-de64-47fd-bcbf-1d976e1dd4f7: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910089::DEBUG::2013-11-19 21:03:04,559::task::495::TaskManager.Task::(_debug) Task e93591a6-de64-47fd-bcbf-1d976e1dd4f7: ref 1 aborting False >Thread-15910089::WARNING::2013-11-19 21:03:04,561::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910089::DEBUG::2013-11-19 21:03:04,562::task::495::TaskManager.Task::(_debug) Task e93591a6-de64-47fd-bcbf-1d976e1dd4f7: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910089::DEBUG::2013-11-19 21:03:04,563::task::495::TaskManager.Task::(_debug) Task e93591a6-de64-47fd-bcbf-1d976e1dd4f7: moving from state preparing -> state finished >Thread-15910089::DEBUG::2013-11-19 21:03:04,563::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910089::DEBUG::2013-11-19 21:03:04,564::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910089::DEBUG::2013-11-19 21:03:04,565::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910089::DEBUG::2013-11-19 21:03:04,565::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910089::DEBUG::2013-11-19 21:03:04,566::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910089::DEBUG::2013-11-19 21:03:04,567::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910089::DEBUG::2013-11-19 21:03:04,567::task::495::TaskManager.Task::(_debug) Task e93591a6-de64-47fd-bcbf-1d976e1dd4f7: ref 0 aborting False >Thread-15910089::INFO::2013-11-19 21:03:04,568::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15909965::DEBUG::2013-11-19 21:03:04,603::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:05,169::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.079084 s, 12.9 MB/s\n'; <rc> = 0 >Thread-15910090::INFO::2013-11-19 21:03:05,580::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910090::DEBUG::2013-11-19 21:03:05,581::task::495::TaskManager.Task::(_debug) Task 833269ef-fc81-4d55-879f-3160655adacb: moving from state init -> state preparing >Thread-15910090::DEBUG::2013-11-19 21:03:05,582::task::495::TaskManager.Task::(_debug) Task 833269ef-fc81-4d55-879f-3160655adacb: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000433206558228', 'lastCheck': 1384894977.6984739, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148487091064', 'lastCheck': 1384894984.289892, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894979.2972829, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000394821166992', 'lastCheck': 1384894977.1592281, 'valid': True, 'code': 0}} >Thread-15910090::DEBUG::2013-11-19 21:03:05,583::task::495::TaskManager.Task::(_debug) Task 833269ef-fc81-4d55-879f-3160655adacb: moving from state preparing -> state finished >Thread-15910090::DEBUG::2013-11-19 21:03:05,584::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910090::DEBUG::2013-11-19 21:03:05,584::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910090::DEBUG::2013-11-19 21:03:05,585::task::495::TaskManager.Task::(_debug) Task 833269ef-fc81-4d55-879f-3160655adacb: ref 0 aborting False >Thread-15910090::INFO::2013-11-19 21:03:05,585::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000433206558228', 'lastCheck': 1384894977.6984739, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148487091064', 'lastCheck': 1384894984.289892, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894979.2972829, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000394821166992', 'lastCheck': 1384894977.1592281, 'valid': True, 'code': 0}} >Thread-15909535::DEBUG::2013-11-19 21:03:06,735::guestIF::97::vm.Vm::(_handleMessage) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::u'None' >Dummy-15909965::DEBUG::2013-11-19 21:03:07,186::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:07,677::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0776669 s, 13.2 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:03:09,372::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:03:09,693::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:10,266::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0756869 s, 13.5 MB/s\n'; <rc> = 0 >Thread-15909535::DEBUG::2013-11-19 21:03:11,423::guestIF::112::vm.Vm::(_handleMessage) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Guest system shuts down. >libvirtEventLoop::DEBUG::2013-11-19 21:03:11,450::vm::525::vm.Vm::(_rtcUpdate) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::new rtc offset -1 >libvirtEventLoop::DEBUG::2013-11-19 21:03:11,451::vm::525::vm.Vm::(_rtcUpdate) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::new rtc offset -1 >libvirtEventLoop::DEBUG::2013-11-19 21:03:11,452::vm::525::vm.Vm::(_rtcUpdate) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::new rtc offset -1 >libvirtEventLoop::DEBUG::2013-11-19 21:03:11,453::vm::525::vm.Vm::(_rtcUpdate) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::new rtc offset -1 >libvirtEventLoop::DEBUG::2013-11-19 21:03:11,454::vm::525::vm.Vm::(_rtcUpdate) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::new rtc offset -1 >libvirtEventLoop::DEBUG::2013-11-19 21:03:11,455::vm::525::vm.Vm::(_rtcUpdate) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::new rtc offset -1 >libvirtEventLoop::DEBUG::2013-11-19 21:03:11,456::vm::525::vm.Vm::(_rtcUpdate) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::new rtc offset -1 >Dummy-15909965::DEBUG::2013-11-19 21:03:12,280::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:12,849::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0810334 s, 12.6 MB/s\n'; <rc> = 0 >Thread-15910096::DEBUG::2013-11-19 21:03:14,681::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910096::INFO::2013-11-19 21:03:14,683::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910096::DEBUG::2013-11-19 21:03:14,684::task::495::TaskManager.Task::(_debug) Task 67e868e0-f4f0-46b2-b2bb-48dab9c2d500: moving from state init -> state preparing >Thread-15910096::DEBUG::2013-11-19 21:03:14,685::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910096::DEBUG::2013-11-19 21:03:14,685::task::495::TaskManager.Task::(_debug) Task 67e868e0-f4f0-46b2-b2bb-48dab9c2d500: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910096::DEBUG::2013-11-19 21:03:14,686::task::495::TaskManager.Task::(_debug) Task 67e868e0-f4f0-46b2-b2bb-48dab9c2d500: moving from state preparing -> state finished >Thread-15910096::DEBUG::2013-11-19 21:03:14,687::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910096::DEBUG::2013-11-19 21:03:14,687::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910096::DEBUG::2013-11-19 21:03:14,688::task::495::TaskManager.Task::(_debug) Task 67e868e0-f4f0-46b2-b2bb-48dab9c2d500: ref 0 aborting False >Thread-15910096::INFO::2013-11-19 21:03:14,689::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910097::DEBUG::2013-11-19 21:03:14,694::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910097::INFO::2013-11-19 21:03:14,696::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910097::DEBUG::2013-11-19 21:03:14,696::task::495::TaskManager.Task::(_debug) Task 6d12caa5-a34f-42c5-8eef-b7c61b11673a: moving from state init -> state preparing >Thread-15910097::DEBUG::2013-11-19 21:03:14,699::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`19c02e71-b108-4e93-b37b-cdf2bdb395e6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910097::DEBUG::2013-11-19 21:03:14,700::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910097::DEBUG::2013-11-19 21:03:14,701::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910097::DEBUG::2013-11-19 21:03:14,702::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`19c02e71-b108-4e93-b37b-cdf2bdb395e6`::Granted request >Thread-15910097::DEBUG::2013-11-19 21:03:14,704::task::495::TaskManager.Task::(_debug) Task 6d12caa5-a34f-42c5-8eef-b7c61b11673a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910097::DEBUG::2013-11-19 21:03:14,705::task::495::TaskManager.Task::(_debug) Task 6d12caa5-a34f-42c5-8eef-b7c61b11673a: ref 1 aborting False >Thread-15910097::WARNING::2013-11-19 21:03:14,709::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910097::DEBUG::2013-11-19 21:03:14,710::task::495::TaskManager.Task::(_debug) Task 6d12caa5-a34f-42c5-8eef-b7c61b11673a: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910097::DEBUG::2013-11-19 21:03:14,712::task::495::TaskManager.Task::(_debug) Task 6d12caa5-a34f-42c5-8eef-b7c61b11673a: moving from state preparing -> state finished >Thread-15910097::DEBUG::2013-11-19 21:03:14,713::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910097::DEBUG::2013-11-19 21:03:14,714::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910097::DEBUG::2013-11-19 21:03:14,715::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910097::DEBUG::2013-11-19 21:03:14,716::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910097::DEBUG::2013-11-19 21:03:14,717::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910097::DEBUG::2013-11-19 21:03:14,718::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910097::DEBUG::2013-11-19 21:03:14,719::task::495::TaskManager.Task::(_debug) Task 6d12caa5-a34f-42c5-8eef-b7c61b11673a: ref 0 aborting False >Thread-15910097::INFO::2013-11-19 21:03:14,719::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15909965::DEBUG::2013-11-19 21:03:14,866::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:15,376::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0756449 s, 13.5 MB/s\n'; <rc> = 0 >Thread-15910098::INFO::2013-11-19 21:03:15,971::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910098::DEBUG::2013-11-19 21:03:15,972::task::495::TaskManager.Task::(_debug) Task ee89203b-9054-4bc2-9e10-7ffe31c6ca06: moving from state init -> state preparing >Thread-15910098::DEBUG::2013-11-19 21:03:15,972::task::495::TaskManager.Task::(_debug) Task ee89203b-9054-4bc2-9e10-7ffe31c6ca06: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000358819961548', 'lastCheck': 1384894987.8386061, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00100898742676', 'lastCheck': 1384894994.5179939, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894989.3733709, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000553846359253', 'lastCheck': 1384894987.368474, 'valid': True, 'code': 0}} >Thread-15910098::DEBUG::2013-11-19 21:03:15,973::task::495::TaskManager.Task::(_debug) Task ee89203b-9054-4bc2-9e10-7ffe31c6ca06: moving from state preparing -> state finished >Thread-15910098::DEBUG::2013-11-19 21:03:15,974::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910098::DEBUG::2013-11-19 21:03:15,975::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910098::DEBUG::2013-11-19 21:03:15,975::task::495::TaskManager.Task::(_debug) Task ee89203b-9054-4bc2-9e10-7ffe31c6ca06: ref 0 aborting False >Thread-15910098::INFO::2013-11-19 21:03:15,976::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000358819961548', 'lastCheck': 1384894987.8386061, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00100898742676', 'lastCheck': 1384894994.5179939, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894989.3733709, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000553846359253', 'lastCheck': 1384894987.368474, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:03:17,392::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:17,946::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.101882 s, 10.1 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:03:19,608::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:03:19,962::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:20,522::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0804972 s, 12.7 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:03:22,537::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:23,094::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0718544 s, 14.3 MB/s\n'; <rc> = 0 >Thread-4922911::INFO::2013-11-19 21:03:23,391::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-4922911::DEBUG::2013-11-19 21:03:23,395::task::495::TaskManager.Task::(_debug) Task 40c410dd-e80b-4c52-a285-bc062942733c: moving from state init -> state preparing >Thread-4922911::DEBUG::2013-11-19 21:03:23,398::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`b35ed45a-2221-4efa-b636-5df7f68ec2ab`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-4922911::DEBUG::2013-11-19 21:03:23,399::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-4922911::DEBUG::2013-11-19 21:03:23,403::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-4922911::DEBUG::2013-11-19 21:03:23,408::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`b35ed45a-2221-4efa-b636-5df7f68ec2ab`::Granted request >Thread-4922911::DEBUG::2013-11-19 21:03:23,411::task::495::TaskManager.Task::(_debug) Task 40c410dd-e80b-4c52-a285-bc062942733c: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-4922911::DEBUG::2013-11-19 21:03:23,417::task::495::TaskManager.Task::(_debug) Task 40c410dd-e80b-4c52-a285-bc062942733c: ref 1 aborting False >Thread-4922911::DEBUG::2013-11-19 21:03:23,424::task::495::TaskManager.Task::(_debug) Task 40c410dd-e80b-4c52-a285-bc062942733c: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-4922911::DEBUG::2013-11-19 21:03:23,430::task::495::TaskManager.Task::(_debug) Task 40c410dd-e80b-4c52-a285-bc062942733c: moving from state preparing -> state finished >Thread-4922911::DEBUG::2013-11-19 21:03:23,434::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-4922911::DEBUG::2013-11-19 21:03:23,434::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4922911::DEBUG::2013-11-19 21:03:23,437::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-4922911::DEBUG::2013-11-19 21:03:23,437::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-4922911::DEBUG::2013-11-19 21:03:23,438::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-4922911::DEBUG::2013-11-19 21:03:23,439::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-4922911::DEBUG::2013-11-19 21:03:23,439::task::495::TaskManager.Task::(_debug) Task 40c410dd-e80b-4c52-a285-bc062942733c: ref 0 aborting False >Thread-4922911::INFO::2013-11-19 21:03:23,440::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-4922911::INFO::2013-11-19 21:03:23,441::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-4922911::DEBUG::2013-11-19 21:03:23,444::task::495::TaskManager.Task::(_debug) Task 0e3e2ae3-83fe-4117-bdc5-1c3c1d8aed45: moving from state init -> state preparing >Thread-4922911::DEBUG::2013-11-19 21:03:23,446::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`3617c09b-5727-4f65-b01b-11dc9cae8d02`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-4922911::DEBUG::2013-11-19 21:03:23,451::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-4922911::DEBUG::2013-11-19 21:03:23,456::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-4922911::DEBUG::2013-11-19 21:03:23,463::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`3617c09b-5727-4f65-b01b-11dc9cae8d02`::Granted request >Thread-4922911::DEBUG::2013-11-19 21:03:23,464::task::495::TaskManager.Task::(_debug) Task 0e3e2ae3-83fe-4117-bdc5-1c3c1d8aed45: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-4922911::DEBUG::2013-11-19 21:03:23,468::task::495::TaskManager.Task::(_debug) Task 0e3e2ae3-83fe-4117-bdc5-1c3c1d8aed45: ref 1 aborting False >Thread-4922911::DEBUG::2013-11-19 21:03:23,471::task::495::TaskManager.Task::(_debug) Task 0e3e2ae3-83fe-4117-bdc5-1c3c1d8aed45: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-4922911::DEBUG::2013-11-19 21:03:23,471::task::495::TaskManager.Task::(_debug) Task 0e3e2ae3-83fe-4117-bdc5-1c3c1d8aed45: moving from state preparing -> state finished >Thread-4922911::DEBUG::2013-11-19 21:03:23,472::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-4922911::DEBUG::2013-11-19 21:03:23,476::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4922911::DEBUG::2013-11-19 21:03:23,476::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-4922911::DEBUG::2013-11-19 21:03:23,477::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-4922911::DEBUG::2013-11-19 21:03:23,478::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-4922911::DEBUG::2013-11-19 21:03:23,479::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-4922911::DEBUG::2013-11-19 21:03:23,480::task::495::TaskManager.Task::(_debug) Task 0e3e2ae3-83fe-4117-bdc5-1c3c1d8aed45: ref 0 aborting False >Thread-4922911::INFO::2013-11-19 21:03:23,485::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-4922911::INFO::2013-11-19 21:03:23,487::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-4922911::DEBUG::2013-11-19 21:03:23,488::task::495::TaskManager.Task::(_debug) Task 28d6a99e-f359-4b5b-b997-8f926d8489d0: moving from state init -> state preparing >Thread-4922911::DEBUG::2013-11-19 21:03:23,494::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`2a0bcb50-b4b3-4f80-9cc9-efb6df195dbc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-4922911::DEBUG::2013-11-19 21:03:23,495::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-4922911::DEBUG::2013-11-19 21:03:23,503::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-4922911::DEBUG::2013-11-19 21:03:23,503::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`2a0bcb50-b4b3-4f80-9cc9-efb6df195dbc`::Granted request >Thread-4922911::DEBUG::2013-11-19 21:03:23,508::task::495::TaskManager.Task::(_debug) Task 28d6a99e-f359-4b5b-b997-8f926d8489d0: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-4922911::DEBUG::2013-11-19 21:03:23,508::task::495::TaskManager.Task::(_debug) Task 28d6a99e-f359-4b5b-b997-8f926d8489d0: ref 1 aborting False >Thread-4922911::DEBUG::2013-11-19 21:03:23,509::task::495::TaskManager.Task::(_debug) Task 28d6a99e-f359-4b5b-b997-8f926d8489d0: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-4922911::DEBUG::2013-11-19 21:03:23,510::task::495::TaskManager.Task::(_debug) Task 28d6a99e-f359-4b5b-b997-8f926d8489d0: moving from state preparing -> state finished >Thread-4922911::DEBUG::2013-11-19 21:03:23,511::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-4922911::DEBUG::2013-11-19 21:03:23,516::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4922911::DEBUG::2013-11-19 21:03:23,521::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-4922911::DEBUG::2013-11-19 21:03:23,522::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-4922911::DEBUG::2013-11-19 21:03:23,522::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-4922911::DEBUG::2013-11-19 21:03:23,523::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-4922911::DEBUG::2013-11-19 21:03:23,523::task::495::TaskManager.Task::(_debug) Task 28d6a99e-f359-4b5b-b997-8f926d8489d0: ref 0 aborting False >Thread-4922911::INFO::2013-11-19 21:03:23,528::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-4922911::INFO::2013-11-19 21:03:23,531::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-4922911::DEBUG::2013-11-19 21:03:23,534::task::495::TaskManager.Task::(_debug) Task 4f59f614-8116-40cf-8730-200d66774124: moving from state init -> state preparing >Thread-4922911::DEBUG::2013-11-19 21:03:23,541::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`463cf201-208a-48f2-9690-c12a5c001bb4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-4922911::DEBUG::2013-11-19 21:03:23,542::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-4922911::DEBUG::2013-11-19 21:03:23,543::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-4922911::DEBUG::2013-11-19 21:03:23,551::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`463cf201-208a-48f2-9690-c12a5c001bb4`::Granted request >Thread-4922911::DEBUG::2013-11-19 21:03:23,556::task::495::TaskManager.Task::(_debug) Task 4f59f614-8116-40cf-8730-200d66774124: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-4922911::DEBUG::2013-11-19 21:03:23,560::task::495::TaskManager.Task::(_debug) Task 4f59f614-8116-40cf-8730-200d66774124: ref 1 aborting False >Thread-4922911::DEBUG::2013-11-19 21:03:23,561::task::495::TaskManager.Task::(_debug) Task 4f59f614-8116-40cf-8730-200d66774124: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-4922911::DEBUG::2013-11-19 21:03:23,568::task::495::TaskManager.Task::(_debug) Task 4f59f614-8116-40cf-8730-200d66774124: moving from state preparing -> state finished >Thread-4922911::DEBUG::2013-11-19 21:03:23,568::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-4922911::DEBUG::2013-11-19 21:03:23,569::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4922911::DEBUG::2013-11-19 21:03:23,569::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-4922911::DEBUG::2013-11-19 21:03:23,572::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-4922911::DEBUG::2013-11-19 21:03:23,572::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-4922911::DEBUG::2013-11-19 21:03:23,578::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-4922911::DEBUG::2013-11-19 21:03:23,579::task::495::TaskManager.Task::(_debug) Task 4f59f614-8116-40cf-8730-200d66774124: ref 0 aborting False >Thread-4922911::INFO::2013-11-19 21:03:23,579::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >libvirtEventLoop::DEBUG::2013-11-19 21:03:23,729::libvirtvm::1401::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::event Saved detail 0 opaque None >libvirtEventLoop::DEBUG::2013-11-19 21:03:24,837::libvirtvm::1401::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::event Stopped detail 0 opaque None >libvirtEventLoop::INFO::2013-11-19 21:03:24,839::vm::445::vm.Vm::(_onQemuDeath) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::underlying process disconnected >Thread-15910104::DEBUG::2013-11-19 21:03:24,840::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >libvirtEventLoop::INFO::2013-11-19 21:03:24,841::libvirtvm::1304::vm.Vm::(releaseVm) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Release VM resources >Thread-15910104::INFO::2013-11-19 21:03:24,841::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910104::DEBUG::2013-11-19 21:03:24,843::task::495::TaskManager.Task::(_debug) Task 56f08a01-e922-42d7-ba70-a0e7bcdeecb2: moving from state init -> state preparing >Thread-15910104::DEBUG::2013-11-19 21:03:24,845::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910104::DEBUG::2013-11-19 21:03:24,846::task::495::TaskManager.Task::(_debug) Task 56f08a01-e922-42d7-ba70-a0e7bcdeecb2: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910104::DEBUG::2013-11-19 21:03:24,847::task::495::TaskManager.Task::(_debug) Task 56f08a01-e922-42d7-ba70-a0e7bcdeecb2: moving from state preparing -> state finished >Thread-15910104::DEBUG::2013-11-19 21:03:24,849::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910104::DEBUG::2013-11-19 21:03:24,850::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910104::DEBUG::2013-11-19 21:03:24,850::task::495::TaskManager.Task::(_debug) Task 56f08a01-e922-42d7-ba70-a0e7bcdeecb2: ref 0 aborting False >Thread-15910104::INFO::2013-11-19 21:03:24,851::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >libvirtEventLoop::DEBUG::2013-11-19 21:03:24,852::utils::314::vm.Vm::(stop) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Stop statistics collection >Thread-15909534::DEBUG::2013-11-19 21:03:24,853::utils::345::vm.Vm::(run) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Stats thread finished >libvirtEventLoop::WARNING::2013-11-19 21:03:24,856::libvirtvm::1315::vm.Vm::(releaseVm) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Traceback (most recent call last): > File "/usr/share/vdsm/libvirtvm.py", line 1312, in releaseVm > File "/usr/share/vdsm/libvirtvm.py", line 445, in f > File "/usr/share/vdsm/libvirtconnection.py", line 63, in wrapper > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 672, in destroy >libvirtError: Domain not found: no domain with matching uuid '4760db3e-d3a8-4045-9153-12c36ec16ba2' > >Thread-15910105::DEBUG::2013-11-19 21:03:24,857::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >libvirtEventLoop::DEBUG::2013-11-19 21:03:24,858::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /sbin/service ksmtuned retune' (cwd None) >Thread-15910105::INFO::2013-11-19 21:03:24,859::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910105::DEBUG::2013-11-19 21:03:24,860::task::495::TaskManager.Task::(_debug) Task 39066e94-d8f7-4914-84d2-db7ac98e707b: moving from state init -> state preparing >Thread-15910105::DEBUG::2013-11-19 21:03:25,067::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`26cd29de-63fc-48a8-8b13-a5407383fcce`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910105::DEBUG::2013-11-19 21:03:25,068::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910105::DEBUG::2013-11-19 21:03:25,070::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910105::DEBUG::2013-11-19 21:03:25,070::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`26cd29de-63fc-48a8-8b13-a5407383fcce`::Granted request >Thread-15910105::DEBUG::2013-11-19 21:03:25,072::task::495::TaskManager.Task::(_debug) Task 39066e94-d8f7-4914-84d2-db7ac98e707b: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910105::DEBUG::2013-11-19 21:03:25,072::task::495::TaskManager.Task::(_debug) Task 39066e94-d8f7-4914-84d2-db7ac98e707b: ref 1 aborting False >Thread-15910105::WARNING::2013-11-19 21:03:25,076::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910105::DEBUG::2013-11-19 21:03:25,077::task::495::TaskManager.Task::(_debug) Task 39066e94-d8f7-4914-84d2-db7ac98e707b: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910105::DEBUG::2013-11-19 21:03:25,078::task::495::TaskManager.Task::(_debug) Task 39066e94-d8f7-4914-84d2-db7ac98e707b: moving from state preparing -> state finished >Thread-15910105::DEBUG::2013-11-19 21:03:25,079::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910105::DEBUG::2013-11-19 21:03:25,079::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910105::DEBUG::2013-11-19 21:03:25,080::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910105::DEBUG::2013-11-19 21:03:25,081::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910105::DEBUG::2013-11-19 21:03:25,081::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910105::DEBUG::2013-11-19 21:03:25,082::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910105::DEBUG::2013-11-19 21:03:25,083::task::495::TaskManager.Task::(_debug) Task 39066e94-d8f7-4914-84d2-db7ac98e707b: ref 0 aborting False >Thread-15910105::INFO::2013-11-19 21:03:25,083::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15909965::DEBUG::2013-11-19 21:03:25,110::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,354::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,357::vm::718::vm.Vm::(_cleanup) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Drive None cleanup >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,358::vm::718::vm.Vm::(_cleanup) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Drive /rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/6f97dfa5-ee35-4f37-8495-1e6b3136b92c/5eaa8dde-10d8-45b5-9e10-5417c297c4be cleanup >libvirtEventLoop::INFO::2013-11-19 21:03:25,361::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: teardownVolume, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,362::task::495::TaskManager.Task::(_debug) Task 6fbeb9b9-3fb8-4453-8a4a-1ed471197e23: moving from state init -> state preparing >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,363::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`5556e88c-0c68-40db-94e7-0bf9a22e58a4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1727' at 'public_teardownVolume' >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,364::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,366::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,366::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`5556e88c-0c68-40db-94e7-0bf9a22e58a4`::Granted request >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,368::task::495::TaskManager.Task::(_debug) Task 6fbeb9b9-3fb8-4453-8a4a-1ed471197e23: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,369::task::495::TaskManager.Task::(_debug) Task 6fbeb9b9-3fb8-4453-8a4a-1ed471197e23: ref 1 aborting False >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,370::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.6f97dfa5-ee35-4f37-8495-1e6b3136b92c' >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,371::resourceManager::530::ResourceManager::(releaseResource) Released resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.6f97dfa5-ee35-4f37-8495-1e6b3136b92c' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,372::resourceManager::535::ResourceManager::(releaseResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.6f97dfa5-ee35-4f37-8495-1e6b3136b92c' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,377::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.5eaa8dde-10d8-45b5-9e10-5417c297c4be' >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,378::resourceManager::530::ResourceManager::(releaseResource) Released resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.5eaa8dde-10d8-45b5-9e10-5417c297c4be' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,379::resourceManager::535::ResourceManager::(releaseResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.5eaa8dde-10d8-45b5-9e10-5417c297c4be' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,379::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.5eaa8dde-10d8-45b5-9e10-5417c297c4be', Clearing records. >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,380::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.6f97dfa5-ee35-4f37-8495-1e6b3136b92c', Clearing records. >libvirtEventLoop::INFO::2013-11-19 21:03:25,381::blockVolume::386::Storage.Volume::(teardown) Tearing down volume 6b892067-ac7e-4039-a7be-5b7387525b16/5eaa8dde-10d8-45b5-9e10-5417c297c4be justme False >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,381::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.5eaa8dde-10d8-45b5-9e10-5417c297c4be' >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,382::resourceManager::530::ResourceManager::(releaseResource) Released resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.5eaa8dde-10d8-45b5-9e10-5417c297c4be' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,383::resourceManager::535::ResourceManager::(releaseResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.5eaa8dde-10d8-45b5-9e10-5417c297c4be' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:03:25,384::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --autobackup n --available n 6b892067-ac7e-4039-a7be-5b7387525b16/5eaa8dde-10d8-45b5-9e10-5417c297c4be' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:25,762::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0883389 s, 11.6 MB/s\n'; <rc> = 0 >Thread-15910106::INFO::2013-11-19 21:03:26,223::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910106::DEBUG::2013-11-19 21:03:26,224::task::495::TaskManager.Task::(_debug) Task 79fa6f6f-4503-4ad5-b0ae-fe25057ee66f: moving from state init -> state preparing >Thread-15910106::DEBUG::2013-11-19 21:03:26,235::task::495::TaskManager.Task::(_debug) Task 79fa6f6f-4503-4ad5-b0ae-fe25057ee66f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000423192977905', 'lastCheck': 1384894998.062989, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00182700157166', 'lastCheck': 1384895004.77634, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894999.6098909, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00100803375244', 'lastCheck': 1384894997.6138971, 'valid': True, 'code': 0}} >Thread-15910106::DEBUG::2013-11-19 21:03:26,247::task::495::TaskManager.Task::(_debug) Task 79fa6f6f-4503-4ad5-b0ae-fe25057ee66f: moving from state preparing -> state finished >libvirtEventLoop::DEBUG::2013-11-19 21:03:26,248::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15910106::DEBUG::2013-11-19 21:03:26,252::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >libvirtEventLoop::DEBUG::2013-11-19 21:03:26,253::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15910106::DEBUG::2013-11-19 21:03:26,254::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >libvirtEventLoop::DEBUG::2013-11-19 21:03:26,254::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15910106::DEBUG::2013-11-19 21:03:26,255::task::495::TaskManager.Task::(_debug) Task 79fa6f6f-4503-4ad5-b0ae-fe25057ee66f: ref 0 aborting False >libvirtEventLoop::DEBUG::2013-11-19 21:03:26,256::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.5eaa8dde-10d8-45b5-9e10-5417c297c4be', Clearing records. >Thread-15910106::INFO::2013-11-19 21:03:26,257::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000423192977905', 'lastCheck': 1384894998.062989, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00182700157166', 'lastCheck': 1384895004.77634, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384894999.6098909, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00100803375244', 'lastCheck': 1384894997.6138971, 'valid': True, 'code': 0}} >libvirtEventLoop::DEBUG::2013-11-19 21:03:26,259::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:03:26,261::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >libvirtEventLoop::DEBUG::2013-11-19 21:03:27,135::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:03:27,341::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:03:27,342::task::495::TaskManager.Task::(_debug) Task 6fbeb9b9-3fb8-4453-8a4a-1ed471197e23: finished: None >libvirtEventLoop::DEBUG::2013-11-19 21:03:27,342::task::495::TaskManager.Task::(_debug) Task 6fbeb9b9-3fb8-4453-8a4a-1ed471197e23: moving from state preparing -> state finished >libvirtEventLoop::DEBUG::2013-11-19 21:03:27,343::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >libvirtEventLoop::DEBUG::2013-11-19 21:03:27,343::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >libvirtEventLoop::DEBUG::2013-11-19 21:03:27,343::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >libvirtEventLoop::DEBUG::2013-11-19 21:03:27,344::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:03:27,344::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:03:27,344::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >libvirtEventLoop::DEBUG::2013-11-19 21:03:27,345::task::495::TaskManager.Task::(_debug) Task 6fbeb9b9-3fb8-4453-8a4a-1ed471197e23: ref 0 aborting False >libvirtEventLoop::INFO::2013-11-19 21:03:27,345::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: teardownVolume, Return response: {'status': {'message': 'OK', 'code': 0}} >libvirtEventLoop::DEBUG::2013-11-19 21:03:27,346::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/libexec/vdsm/hooks/after_vm_destroy/50_vhostmd' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:27,773::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:28,231::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0460724 s, 22.2 MB/s\n'; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:03:28,243::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >libvirtEventLoop::INFO::2013-11-19 21:03:28,244::hooks::51::root::(_runHooksDir) >libvirtEventLoop::DEBUG::2013-11-19 21:03:28,248::vm::742::vm.Vm::(setDownStatus) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Changed state to Down: User shut down >libvirtEventLoop::DEBUG::2013-11-19 21:03:28,248::utils::314::vm.Vm::(stop) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Stop statistics collection >Thread-15910110::DEBUG::2013-11-19 21:03:28,633::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4760db3e-d3a8-4045-9153-12c36ec16ba2',) {} >Thread-15910110::DEBUG::2013-11-19 21:03:28,633::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Down', 'timeOffset': -1L, 'vmId': '4760db3e-d3a8-4045-9153-12c36ec16ba2', 'exitMessage': 'User shut down', 'exitCode': 0}]} >Thread-15910111::DEBUG::2013-11-19 21:03:28,641::clientIF::54::vds::(wrapper) [192.168.100.135]::call destroy with ('4760db3e-d3a8-4045-9153-12c36ec16ba2',) {} >Thread-15910111::INFO::2013-11-19 21:03:28,641::clientIF::450::vds::(destroy) vmContainerLock acquired by vm 4760db3e-d3a8-4045-9153-12c36ec16ba2 >Thread-15910111::DEBUG::2013-11-19 21:03:28,642::libvirtvm::1344::vm.Vm::(destroy) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::destroy Called >Thread-15910111::DEBUG::2013-11-19 21:03:28,643::libvirtvm::1339::vm.Vm::(deleteVm) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Total desktops after destroy of 4760db3e-d3a8-4045-9153-12c36ec16ba2 is 1 >Thread-15910111::DEBUG::2013-11-19 21:03:28,643::clientIF::59::vds::(wrapper) return destroy with {'status': {'message': 'Machine destroyed', 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:03:29,780::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:03:30,241::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:30,763::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0787295 s, 13.0 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:03:32,777::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:33,328::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0752112 s, 13.6 MB/s\n'; <rc> = 0 >Thread-15910115::DEBUG::2013-11-19 21:03:35,186::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910115::INFO::2013-11-19 21:03:35,188::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910115::DEBUG::2013-11-19 21:03:35,188::task::495::TaskManager.Task::(_debug) Task 5c1ac5db-09ad-4fa4-8943-93899127788c: moving from state init -> state preparing >Thread-15910115::DEBUG::2013-11-19 21:03:35,189::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910115::DEBUG::2013-11-19 21:03:35,190::task::495::TaskManager.Task::(_debug) Task 5c1ac5db-09ad-4fa4-8943-93899127788c: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910115::DEBUG::2013-11-19 21:03:35,190::task::495::TaskManager.Task::(_debug) Task 5c1ac5db-09ad-4fa4-8943-93899127788c: moving from state preparing -> state finished >Thread-15910115::DEBUG::2013-11-19 21:03:35,191::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910115::DEBUG::2013-11-19 21:03:35,191::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910115::DEBUG::2013-11-19 21:03:35,192::task::495::TaskManager.Task::(_debug) Task 5c1ac5db-09ad-4fa4-8943-93899127788c: ref 0 aborting False >Thread-15910115::INFO::2013-11-19 21:03:35,192::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910116::DEBUG::2013-11-19 21:03:35,197::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910116::INFO::2013-11-19 21:03:35,197::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910116::DEBUG::2013-11-19 21:03:35,198::task::495::TaskManager.Task::(_debug) Task 9c043bbd-0313-4b7a-943e-f78722ee2b01: moving from state init -> state preparing >Thread-15910116::DEBUG::2013-11-19 21:03:35,199::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`cc4aba83-e4c6-413c-8e4f-1b404fc65895`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910116::DEBUG::2013-11-19 21:03:35,200::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910116::DEBUG::2013-11-19 21:03:35,201::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910116::DEBUG::2013-11-19 21:03:35,201::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`cc4aba83-e4c6-413c-8e4f-1b404fc65895`::Granted request >Thread-15910116::DEBUG::2013-11-19 21:03:35,202::task::495::TaskManager.Task::(_debug) Task 9c043bbd-0313-4b7a-943e-f78722ee2b01: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910116::DEBUG::2013-11-19 21:03:35,203::task::495::TaskManager.Task::(_debug) Task 9c043bbd-0313-4b7a-943e-f78722ee2b01: ref 1 aborting False >Thread-15910116::WARNING::2013-11-19 21:03:35,205::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910116::DEBUG::2013-11-19 21:03:35,205::task::495::TaskManager.Task::(_debug) Task 9c043bbd-0313-4b7a-943e-f78722ee2b01: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910116::DEBUG::2013-11-19 21:03:35,206::task::495::TaskManager.Task::(_debug) Task 9c043bbd-0313-4b7a-943e-f78722ee2b01: moving from state preparing -> state finished >Thread-15910116::DEBUG::2013-11-19 21:03:35,207::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910116::DEBUG::2013-11-19 21:03:35,207::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910116::DEBUG::2013-11-19 21:03:35,208::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910116::DEBUG::2013-11-19 21:03:35,209::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910116::DEBUG::2013-11-19 21:03:35,209::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910116::DEBUG::2013-11-19 21:03:35,210::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910116::DEBUG::2013-11-19 21:03:35,210::task::495::TaskManager.Task::(_debug) Task 9c043bbd-0313-4b7a-943e-f78722ee2b01: ref 0 aborting False >Thread-15910116::INFO::2013-11-19 21:03:35,211::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15909965::DEBUG::2013-11-19 21:03:35,344::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:35,826::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0469543 s, 21.8 MB/s\n'; <rc> = 0 >Thread-15910117::INFO::2013-11-19 21:03:36,740::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910117::DEBUG::2013-11-19 21:03:36,741::task::495::TaskManager.Task::(_debug) Task 26a5dfc5-f8b8-4403-998a-b5345daa3636: moving from state init -> state preparing >Thread-15910117::DEBUG::2013-11-19 21:03:36,742::task::495::TaskManager.Task::(_debug) Task 26a5dfc5-f8b8-4403-998a-b5345daa3636: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000458002090454', 'lastCheck': 1384895008.5292549, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00245499610901', 'lastCheck': 1384895015.010802, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895009.780951, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000570058822632', 'lastCheck': 1384895007.776001, 'valid': True, 'code': 0}} >Thread-15910117::DEBUG::2013-11-19 21:03:36,743::task::495::TaskManager.Task::(_debug) Task 26a5dfc5-f8b8-4403-998a-b5345daa3636: moving from state preparing -> state finished >Thread-15910117::DEBUG::2013-11-19 21:03:36,743::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910117::DEBUG::2013-11-19 21:03:36,744::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910117::DEBUG::2013-11-19 21:03:36,744::task::495::TaskManager.Task::(_debug) Task 26a5dfc5-f8b8-4403-998a-b5345daa3636: ref 0 aborting False >Thread-15910117::INFO::2013-11-19 21:03:36,745::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000458002090454', 'lastCheck': 1384895008.5292549, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00245499610901', 'lastCheck': 1384895015.010802, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895009.780951, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000570058822632', 'lastCheck': 1384895007.776001, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:03:37,837::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:38,377::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0740086 s, 13.8 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:03:40,064::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:03:40,391::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:40,913::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0741586 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:03:42,928::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:43,457::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0752537 s, 13.6 MB/s\n'; <rc> = 0 >Thread-15910123::DEBUG::2013-11-19 21:03:45,327::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910123::INFO::2013-11-19 21:03:45,328::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910123::DEBUG::2013-11-19 21:03:45,329::task::495::TaskManager.Task::(_debug) Task b961c98b-347a-4f2b-9c47-82cd8ada15bb: moving from state init -> state preparing >Thread-15910123::DEBUG::2013-11-19 21:03:45,330::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910123::DEBUG::2013-11-19 21:03:45,331::task::495::TaskManager.Task::(_debug) Task b961c98b-347a-4f2b-9c47-82cd8ada15bb: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910123::DEBUG::2013-11-19 21:03:45,331::task::495::TaskManager.Task::(_debug) Task b961c98b-347a-4f2b-9c47-82cd8ada15bb: moving from state preparing -> state finished >Thread-15910123::DEBUG::2013-11-19 21:03:45,332::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910123::DEBUG::2013-11-19 21:03:45,332::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910123::DEBUG::2013-11-19 21:03:45,333::task::495::TaskManager.Task::(_debug) Task b961c98b-347a-4f2b-9c47-82cd8ada15bb: ref 0 aborting False >Thread-15910123::INFO::2013-11-19 21:03:45,334::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910124::DEBUG::2013-11-19 21:03:45,338::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910124::INFO::2013-11-19 21:03:45,339::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910124::DEBUG::2013-11-19 21:03:45,340::task::495::TaskManager.Task::(_debug) Task 962c0d91-044b-47d3-a381-f147cca1e4ca: moving from state init -> state preparing >Thread-15910124::DEBUG::2013-11-19 21:03:45,341::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3d8588c6-4150-4e8a-a7fc-71efa42a2a9a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910124::DEBUG::2013-11-19 21:03:45,342::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910124::DEBUG::2013-11-19 21:03:45,343::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910124::DEBUG::2013-11-19 21:03:45,343::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3d8588c6-4150-4e8a-a7fc-71efa42a2a9a`::Granted request >Thread-15910124::DEBUG::2013-11-19 21:03:45,344::task::495::TaskManager.Task::(_debug) Task 962c0d91-044b-47d3-a381-f147cca1e4ca: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910124::DEBUG::2013-11-19 21:03:45,345::task::495::TaskManager.Task::(_debug) Task 962c0d91-044b-47d3-a381-f147cca1e4ca: ref 1 aborting False >Thread-15910124::WARNING::2013-11-19 21:03:45,347::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910124::DEBUG::2013-11-19 21:03:45,348::task::495::TaskManager.Task::(_debug) Task 962c0d91-044b-47d3-a381-f147cca1e4ca: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910124::DEBUG::2013-11-19 21:03:45,349::task::495::TaskManager.Task::(_debug) Task 962c0d91-044b-47d3-a381-f147cca1e4ca: moving from state preparing -> state finished >Thread-15910124::DEBUG::2013-11-19 21:03:45,349::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910124::DEBUG::2013-11-19 21:03:45,350::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910124::DEBUG::2013-11-19 21:03:45,350::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910124::DEBUG::2013-11-19 21:03:45,351::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910124::DEBUG::2013-11-19 21:03:45,352::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910124::DEBUG::2013-11-19 21:03:45,352::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910124::DEBUG::2013-11-19 21:03:45,353::task::495::TaskManager.Task::(_debug) Task 962c0d91-044b-47d3-a381-f147cca1e4ca: ref 0 aborting False >Thread-15910124::INFO::2013-11-19 21:03:45,353::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15909965::DEBUG::2013-11-19 21:03:45,473::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:45,972::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0453893 s, 22.6 MB/s\n'; <rc> = 0 >Thread-15910125::INFO::2013-11-19 21:03:46,890::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910125::DEBUG::2013-11-19 21:03:46,890::task::495::TaskManager.Task::(_debug) Task f23c6d0a-d074-4037-b164-ebc5226585f2: moving from state init -> state preparing >Thread-15910125::DEBUG::2013-11-19 21:03:46,891::task::495::TaskManager.Task::(_debug) Task f23c6d0a-d074-4037-b164-ebc5226585f2: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000286817550659', 'lastCheck': 1384895018.574862, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144410133362', 'lastCheck': 1384895025.1583879, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895020.0649731, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000739097595215', 'lastCheck': 1384895018.0603249, 'valid': True, 'code': 0}} >Thread-15910125::DEBUG::2013-11-19 21:03:46,892::task::495::TaskManager.Task::(_debug) Task f23c6d0a-d074-4037-b164-ebc5226585f2: moving from state preparing -> state finished >Thread-15910125::DEBUG::2013-11-19 21:03:46,893::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910125::DEBUG::2013-11-19 21:03:46,893::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910125::DEBUG::2013-11-19 21:03:46,894::task::495::TaskManager.Task::(_debug) Task f23c6d0a-d074-4037-b164-ebc5226585f2: ref 0 aborting False >Thread-15910125::INFO::2013-11-19 21:03:46,894::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000286817550659', 'lastCheck': 1384895018.574862, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144410133362', 'lastCheck': 1384895025.1583879, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895020.0649731, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000739097595215', 'lastCheck': 1384895018.0603249, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:03:47,983::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:48,535::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0724856 s, 14.1 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:03:50,223::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:03:50,550::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:51,126::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.08746 s, 11.7 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:03:53,140::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:53,666::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0705313 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15910131::DEBUG::2013-11-19 21:03:55,459::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910131::INFO::2013-11-19 21:03:55,461::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910131::DEBUG::2013-11-19 21:03:55,461::task::495::TaskManager.Task::(_debug) Task d47a6308-b704-4ccf-aec4-8006623cff11: moving from state init -> state preparing >Thread-15910131::DEBUG::2013-11-19 21:03:55,462::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910131::DEBUG::2013-11-19 21:03:55,463::task::495::TaskManager.Task::(_debug) Task d47a6308-b704-4ccf-aec4-8006623cff11: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910131::DEBUG::2013-11-19 21:03:55,463::task::495::TaskManager.Task::(_debug) Task d47a6308-b704-4ccf-aec4-8006623cff11: moving from state preparing -> state finished >Thread-15910131::DEBUG::2013-11-19 21:03:55,464::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910131::DEBUG::2013-11-19 21:03:55,465::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910131::DEBUG::2013-11-19 21:03:55,465::task::495::TaskManager.Task::(_debug) Task d47a6308-b704-4ccf-aec4-8006623cff11: ref 0 aborting False >Thread-15910131::INFO::2013-11-19 21:03:55,466::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910132::DEBUG::2013-11-19 21:03:55,470::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910132::INFO::2013-11-19 21:03:55,471::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910132::DEBUG::2013-11-19 21:03:55,472::task::495::TaskManager.Task::(_debug) Task 93d5fbf8-3af6-4030-9c27-f2cace062c7a: moving from state init -> state preparing >Thread-15910132::DEBUG::2013-11-19 21:03:55,473::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`733f130d-f94c-4de0-bfbc-176bf53edef4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910132::DEBUG::2013-11-19 21:03:55,473::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910132::DEBUG::2013-11-19 21:03:55,474::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910132::DEBUG::2013-11-19 21:03:55,475::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`733f130d-f94c-4de0-bfbc-176bf53edef4`::Granted request >Thread-15910132::DEBUG::2013-11-19 21:03:55,476::task::495::TaskManager.Task::(_debug) Task 93d5fbf8-3af6-4030-9c27-f2cace062c7a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910132::DEBUG::2013-11-19 21:03:55,476::task::495::TaskManager.Task::(_debug) Task 93d5fbf8-3af6-4030-9c27-f2cace062c7a: ref 1 aborting False >Thread-15910132::WARNING::2013-11-19 21:03:55,479::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910132::DEBUG::2013-11-19 21:03:55,479::task::495::TaskManager.Task::(_debug) Task 93d5fbf8-3af6-4030-9c27-f2cace062c7a: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910132::DEBUG::2013-11-19 21:03:55,480::task::495::TaskManager.Task::(_debug) Task 93d5fbf8-3af6-4030-9c27-f2cace062c7a: moving from state preparing -> state finished >Thread-15910132::DEBUG::2013-11-19 21:03:55,481::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910132::DEBUG::2013-11-19 21:03:55,481::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910132::DEBUG::2013-11-19 21:03:55,482::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910132::DEBUG::2013-11-19 21:03:55,483::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910132::DEBUG::2013-11-19 21:03:55,483::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910132::DEBUG::2013-11-19 21:03:55,484::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910132::DEBUG::2013-11-19 21:03:55,485::task::495::TaskManager.Task::(_debug) Task 93d5fbf8-3af6-4030-9c27-f2cace062c7a: ref 0 aborting False >Thread-15910132::INFO::2013-11-19 21:03:55,485::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15909965::DEBUG::2013-11-19 21:03:55,681::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:56,159::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0499763 s, 20.5 MB/s\n'; <rc> = 0 >Thread-15910133::INFO::2013-11-19 21:03:57,105::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910133::DEBUG::2013-11-19 21:03:57,106::task::495::TaskManager.Task::(_debug) Task 6c59b50d-f575-4ed4-a595-9fa80f3b38d6: moving from state init -> state preparing >Thread-15910133::DEBUG::2013-11-19 21:03:57,107::task::495::TaskManager.Task::(_debug) Task 6c59b50d-f575-4ed4-a595-9fa80f3b38d6: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000808954238892', 'lastCheck': 1384895028.7077301, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00134301185608', 'lastCheck': 1384895035.3667779, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895030.2249441, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00803112983704', 'lastCheck': 1384895028.241358, 'valid': True, 'code': 0}} >Thread-15910133::DEBUG::2013-11-19 21:03:57,108::task::495::TaskManager.Task::(_debug) Task 6c59b50d-f575-4ed4-a595-9fa80f3b38d6: moving from state preparing -> state finished >Thread-15910133::DEBUG::2013-11-19 21:03:57,109::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910133::DEBUG::2013-11-19 21:03:57,109::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910133::DEBUG::2013-11-19 21:03:57,110::task::495::TaskManager.Task::(_debug) Task 6c59b50d-f575-4ed4-a595-9fa80f3b38d6: ref 0 aborting False >Thread-15910133::INFO::2013-11-19 21:03:57,110::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000808954238892', 'lastCheck': 1384895028.7077301, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00134301185608', 'lastCheck': 1384895035.3667779, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895030.2249441, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00803112983704', 'lastCheck': 1384895028.241358, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:03:58,170::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:03:58,734::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0792493 s, 12.9 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:04:00,396::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:04:00,749::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:04:01,281::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0737496 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:04:03,295::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:04:03,814::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0577656 s, 17.7 MB/s\n'; <rc> = 0 >Thread-15910139::DEBUG::2013-11-19 21:04:05,592::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910139::INFO::2013-11-19 21:04:05,594::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910139::DEBUG::2013-11-19 21:04:05,594::task::495::TaskManager.Task::(_debug) Task 9268ad65-ac27-4976-899b-410714ff06ae: moving from state init -> state preparing >Thread-15910139::DEBUG::2013-11-19 21:04:05,603::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910139::DEBUG::2013-11-19 21:04:05,603::task::495::TaskManager.Task::(_debug) Task 9268ad65-ac27-4976-899b-410714ff06ae: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910139::DEBUG::2013-11-19 21:04:05,604::task::495::TaskManager.Task::(_debug) Task 9268ad65-ac27-4976-899b-410714ff06ae: moving from state preparing -> state finished >Thread-15910139::DEBUG::2013-11-19 21:04:05,604::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910139::DEBUG::2013-11-19 21:04:05,605::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910139::DEBUG::2013-11-19 21:04:05,606::task::495::TaskManager.Task::(_debug) Task 9268ad65-ac27-4976-899b-410714ff06ae: ref 0 aborting False >Thread-15910139::INFO::2013-11-19 21:04:05,606::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910140::DEBUG::2013-11-19 21:04:05,611::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910140::INFO::2013-11-19 21:04:05,611::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910140::DEBUG::2013-11-19 21:04:05,612::task::495::TaskManager.Task::(_debug) Task f7d7b021-6c79-41a6-aee2-71fb1fa65bb7: moving from state init -> state preparing >Thread-15910140::DEBUG::2013-11-19 21:04:05,613::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1a1b60fb-263b-4793-ac86-80e496ba8b55`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910140::DEBUG::2013-11-19 21:04:05,614::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910140::DEBUG::2013-11-19 21:04:05,615::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910140::DEBUG::2013-11-19 21:04:05,615::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1a1b60fb-263b-4793-ac86-80e496ba8b55`::Granted request >Thread-15910140::DEBUG::2013-11-19 21:04:05,616::task::495::TaskManager.Task::(_debug) Task f7d7b021-6c79-41a6-aee2-71fb1fa65bb7: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910140::DEBUG::2013-11-19 21:04:05,617::task::495::TaskManager.Task::(_debug) Task f7d7b021-6c79-41a6-aee2-71fb1fa65bb7: ref 1 aborting False >Thread-15910140::WARNING::2013-11-19 21:04:05,619::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910140::DEBUG::2013-11-19 21:04:05,620::task::495::TaskManager.Task::(_debug) Task f7d7b021-6c79-41a6-aee2-71fb1fa65bb7: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910140::DEBUG::2013-11-19 21:04:05,621::task::495::TaskManager.Task::(_debug) Task f7d7b021-6c79-41a6-aee2-71fb1fa65bb7: moving from state preparing -> state finished >Thread-15910140::DEBUG::2013-11-19 21:04:05,621::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910140::DEBUG::2013-11-19 21:04:05,622::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910140::DEBUG::2013-11-19 21:04:05,623::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910140::DEBUG::2013-11-19 21:04:05,623::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910140::DEBUG::2013-11-19 21:04:05,624::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910140::DEBUG::2013-11-19 21:04:05,624::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910140::DEBUG::2013-11-19 21:04:05,625::task::495::TaskManager.Task::(_debug) Task f7d7b021-6c79-41a6-aee2-71fb1fa65bb7: ref 0 aborting False >Thread-15910140::INFO::2013-11-19 21:04:05,626::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15909965::DEBUG::2013-11-19 21:04:05,829::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:04:06,412::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0930995 s, 11.0 MB/s\n'; <rc> = 0 >Thread-4922912::DEBUG::2013-11-19 21:04:06,503::guestIF::97::vm.Vm::(_handleMessage) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::u'None' >Thread-15910141::INFO::2013-11-19 21:04:07,267::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910141::DEBUG::2013-11-19 21:04:07,268::task::495::TaskManager.Task::(_debug) Task 05bbb3cf-5776-49bb-ac6c-8bb18713e4a4: moving from state init -> state preparing >Thread-15910141::DEBUG::2013-11-19 21:04:07,269::task::495::TaskManager.Task::(_debug) Task 05bbb3cf-5776-49bb-ac6c-8bb18713e4a4: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00046706199646', 'lastCheck': 1384895038.910903, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137209892273', 'lastCheck': 1384895045.549921, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895040.396832, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000672817230225', 'lastCheck': 1384895038.3919809, 'valid': True, 'code': 0}} >Thread-15910141::DEBUG::2013-11-19 21:04:07,269::task::495::TaskManager.Task::(_debug) Task 05bbb3cf-5776-49bb-ac6c-8bb18713e4a4: moving from state preparing -> state finished >Thread-15910141::DEBUG::2013-11-19 21:04:07,270::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910141::DEBUG::2013-11-19 21:04:07,271::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910141::DEBUG::2013-11-19 21:04:07,271::task::495::TaskManager.Task::(_debug) Task 05bbb3cf-5776-49bb-ac6c-8bb18713e4a4: ref 0 aborting False >Thread-15910141::INFO::2013-11-19 21:04:07,272::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00046706199646', 'lastCheck': 1384895038.910903, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137209892273', 'lastCheck': 1384895045.549921, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895040.396832, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000672817230225', 'lastCheck': 1384895038.3919809, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:04:08,427::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:04:08,954::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0708984 s, 14.4 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:04:10,669::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:04:10,970::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:04:11,450::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0632862 s, 16.2 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:04:13,472::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:04:14,010::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0745248 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15910147::DEBUG::2013-11-19 21:04:15,741::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910147::INFO::2013-11-19 21:04:15,742::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910147::DEBUG::2013-11-19 21:04:15,743::task::495::TaskManager.Task::(_debug) Task 430a7f15-06cb-4dd4-971e-59c6afabbb34: moving from state init -> state preparing >Thread-15910147::DEBUG::2013-11-19 21:04:15,744::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910147::DEBUG::2013-11-19 21:04:15,744::task::495::TaskManager.Task::(_debug) Task 430a7f15-06cb-4dd4-971e-59c6afabbb34: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910147::DEBUG::2013-11-19 21:04:15,745::task::495::TaskManager.Task::(_debug) Task 430a7f15-06cb-4dd4-971e-59c6afabbb34: moving from state preparing -> state finished >Thread-15910147::DEBUG::2013-11-19 21:04:15,745::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910147::DEBUG::2013-11-19 21:04:15,746::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910147::DEBUG::2013-11-19 21:04:15,747::task::495::TaskManager.Task::(_debug) Task 430a7f15-06cb-4dd4-971e-59c6afabbb34: ref 0 aborting False >Thread-15910147::INFO::2013-11-19 21:04:15,747::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910148::DEBUG::2013-11-19 21:04:15,751::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910148::INFO::2013-11-19 21:04:15,752::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910148::DEBUG::2013-11-19 21:04:15,753::task::495::TaskManager.Task::(_debug) Task 1d9f9f2d-c587-4cf9-8fd0-a8051b8ede8a: moving from state init -> state preparing >Thread-15910148::DEBUG::2013-11-19 21:04:15,754::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c587b635-00c3-46cd-a032-47b1e4bb5702`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910148::DEBUG::2013-11-19 21:04:15,755::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910148::DEBUG::2013-11-19 21:04:15,755::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910148::DEBUG::2013-11-19 21:04:15,756::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c587b635-00c3-46cd-a032-47b1e4bb5702`::Granted request >Thread-15910148::DEBUG::2013-11-19 21:04:15,757::task::495::TaskManager.Task::(_debug) Task 1d9f9f2d-c587-4cf9-8fd0-a8051b8ede8a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910148::DEBUG::2013-11-19 21:04:15,758::task::495::TaskManager.Task::(_debug) Task 1d9f9f2d-c587-4cf9-8fd0-a8051b8ede8a: ref 1 aborting False >Thread-15910148::WARNING::2013-11-19 21:04:15,760::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910148::DEBUG::2013-11-19 21:04:15,761::task::495::TaskManager.Task::(_debug) Task 1d9f9f2d-c587-4cf9-8fd0-a8051b8ede8a: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910148::DEBUG::2013-11-19 21:04:15,762::task::495::TaskManager.Task::(_debug) Task 1d9f9f2d-c587-4cf9-8fd0-a8051b8ede8a: moving from state preparing -> state finished >Thread-15910148::DEBUG::2013-11-19 21:04:15,762::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910148::DEBUG::2013-11-19 21:04:15,763::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910148::DEBUG::2013-11-19 21:04:15,763::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910148::DEBUG::2013-11-19 21:04:15,764::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910148::DEBUG::2013-11-19 21:04:15,765::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910148::DEBUG::2013-11-19 21:04:15,765::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910148::DEBUG::2013-11-19 21:04:15,766::task::495::TaskManager.Task::(_debug) Task 1d9f9f2d-c587-4cf9-8fd0-a8051b8ede8a: ref 0 aborting False >Thread-15910148::INFO::2013-11-19 21:04:15,766::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15909965::DEBUG::2013-11-19 21:04:16,025::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:04:16,555::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0733971 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15910149::INFO::2013-11-19 21:04:17,734::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910149::DEBUG::2013-11-19 21:04:17,735::task::495::TaskManager.Task::(_debug) Task 5a8672fd-e8d3-487f-a4d4-86df34edd0cd: moving from state init -> state preparing >Thread-15910149::DEBUG::2013-11-19 21:04:17,736::task::495::TaskManager.Task::(_debug) Task 5a8672fd-e8d3-487f-a4d4-86df34edd0cd: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000461101531982', 'lastCheck': 1384895049.085433, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00112199783325', 'lastCheck': 1384895055.69243, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895050.6699841, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000598192214966', 'lastCheck': 1384895048.6668911, 'valid': True, 'code': 0}} >Thread-15910149::DEBUG::2013-11-19 21:04:17,737::task::495::TaskManager.Task::(_debug) Task 5a8672fd-e8d3-487f-a4d4-86df34edd0cd: moving from state preparing -> state finished >Thread-15910149::DEBUG::2013-11-19 21:04:17,738::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910149::DEBUG::2013-11-19 21:04:17,739::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910149::DEBUG::2013-11-19 21:04:17,739::task::495::TaskManager.Task::(_debug) Task 5a8672fd-e8d3-487f-a4d4-86df34edd0cd: ref 0 aborting False >Thread-15910149::INFO::2013-11-19 21:04:17,740::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000461101531982', 'lastCheck': 1384895049.085433, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00112199783325', 'lastCheck': 1384895055.69243, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895050.6699841, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000598192214966', 'lastCheck': 1384895048.6668911, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:04:18,569::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:04:19,132::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0743848 s, 13.8 MB/s\n'; <rc> = 0 >Thread-42838::DEBUG::2013-11-19 21:04:19,256::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 21:04:19,900::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:04:20,817::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:04:21,147::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15910152::DEBUG::2013-11-19 21:04:21,438::clientIF::54::vds::(wrapper) [192.168.100.135]::call shutdown with ('6574d88c-00ac-46f4-99ce-a2726932bfda', '30', 'The Red Hat Enterprise Virtualization Manager is shutting down this Virtual Machine') {} >Thread-15910152::DEBUG::2013-11-19 21:04:21,441::vm::673::vm.Vm::(shutdown) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::guestAgent shutdown called >Thread-15910152::DEBUG::2013-11-19 21:04:21,442::guestIF::179::vm.Vm::(desktopShutdown) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::desktopShutdown called >Thread-15910152::DEBUG::2013-11-19 21:04:21,444::clientIF::59::vds::(wrapper) return shutdown with {'status': {'message': 'Machine shut down', 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:04:21,710::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.071086 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15910155::DEBUG::2013-11-19 21:04:21,945::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910155::DEBUG::2013-11-19 21:04:21,946::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '92', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'UserLoggedOn', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663671', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '306111', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '197517', 'readRate': '0.00', 'truesize': '64424509440', 'writeRate': '2352.91'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '371802', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '181637', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '936.10'}, u'vdd': {'readLatency': '8479202', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '404.80', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.49', 'cpuUser': '162.62', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Dummy-15909965::DEBUG::2013-11-19 21:04:23,725::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15910157::DEBUG::2013-11-19 21:04:24,019::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910157::DEBUG::2013-11-19 21:04:24,021::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '92', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'UserLoggedOn', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663673', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '306111', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '197517', 'readRate': '0.00', 'truesize': '64424509440', 'writeRate': '2352.91'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '371802', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '181637', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '936.10'}, u'vdd': {'readLatency': '8479202', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '404.80', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.56', 'cpuUser': '162.62', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Dummy-15909965::DEBUG::2013-11-19 21:04:24,264::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0739422 s, 13.8 MB/s\n'; <rc> = 0 >Thread-4922911::INFO::2013-11-19 21:04:24,460::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-4922911::DEBUG::2013-11-19 21:04:24,461::task::495::TaskManager.Task::(_debug) Task 7e9fb6db-dce4-4dce-b896-424fda534495: moving from state init -> state preparing >Thread-4922911::DEBUG::2013-11-19 21:04:24,462::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fac99520-b517-4649-8aec-3a83cb430c41`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-4922911::DEBUG::2013-11-19 21:04:24,463::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-4922911::DEBUG::2013-11-19 21:04:24,464::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-4922911::DEBUG::2013-11-19 21:04:24,464::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fac99520-b517-4649-8aec-3a83cb430c41`::Granted request >Thread-4922911::DEBUG::2013-11-19 21:04:24,465::task::495::TaskManager.Task::(_debug) Task 7e9fb6db-dce4-4dce-b896-424fda534495: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-4922911::DEBUG::2013-11-19 21:04:24,466::task::495::TaskManager.Task::(_debug) Task 7e9fb6db-dce4-4dce-b896-424fda534495: ref 1 aborting False >Thread-4922911::DEBUG::2013-11-19 21:04:24,467::task::495::TaskManager.Task::(_debug) Task 7e9fb6db-dce4-4dce-b896-424fda534495: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-4922911::DEBUG::2013-11-19 21:04:24,467::task::495::TaskManager.Task::(_debug) Task 7e9fb6db-dce4-4dce-b896-424fda534495: moving from state preparing -> state finished >Thread-4922911::DEBUG::2013-11-19 21:04:24,468::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-4922911::DEBUG::2013-11-19 21:04:24,468::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4922911::DEBUG::2013-11-19 21:04:24,469::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-4922911::DEBUG::2013-11-19 21:04:24,470::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-4922911::DEBUG::2013-11-19 21:04:24,470::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-4922911::DEBUG::2013-11-19 21:04:24,470::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-4922911::DEBUG::2013-11-19 21:04:24,471::task::495::TaskManager.Task::(_debug) Task 7e9fb6db-dce4-4dce-b896-424fda534495: ref 0 aborting False >Thread-4922911::INFO::2013-11-19 21:04:24,472::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-4922911::INFO::2013-11-19 21:04:24,472::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-4922911::DEBUG::2013-11-19 21:04:24,473::task::495::TaskManager.Task::(_debug) Task 49269880-962c-43c8-b8f0-8e0a4833dc74: moving from state init -> state preparing >Thread-4922911::DEBUG::2013-11-19 21:04:24,474::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`e2cc679f-b0dd-4944-9595-798b3a3993d4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-4922911::DEBUG::2013-11-19 21:04:24,475::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-4922911::DEBUG::2013-11-19 21:04:24,475::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-4922911::DEBUG::2013-11-19 21:04:24,476::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`e2cc679f-b0dd-4944-9595-798b3a3993d4`::Granted request >Thread-4922911::DEBUG::2013-11-19 21:04:24,477::task::495::TaskManager.Task::(_debug) Task 49269880-962c-43c8-b8f0-8e0a4833dc74: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-4922911::DEBUG::2013-11-19 21:04:24,477::task::495::TaskManager.Task::(_debug) Task 49269880-962c-43c8-b8f0-8e0a4833dc74: ref 1 aborting False >Thread-4922911::DEBUG::2013-11-19 21:04:24,478::task::495::TaskManager.Task::(_debug) Task 49269880-962c-43c8-b8f0-8e0a4833dc74: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-4922911::DEBUG::2013-11-19 21:04:24,478::task::495::TaskManager.Task::(_debug) Task 49269880-962c-43c8-b8f0-8e0a4833dc74: moving from state preparing -> state finished >Thread-4922911::DEBUG::2013-11-19 21:04:24,479::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-4922911::DEBUG::2013-11-19 21:04:24,479::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4922911::DEBUG::2013-11-19 21:04:24,480::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-4922911::DEBUG::2013-11-19 21:04:24,481::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-4922911::DEBUG::2013-11-19 21:04:24,481::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-4922911::DEBUG::2013-11-19 21:04:24,482::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-4922911::DEBUG::2013-11-19 21:04:24,482::task::495::TaskManager.Task::(_debug) Task 49269880-962c-43c8-b8f0-8e0a4833dc74: ref 0 aborting False >Thread-4922911::INFO::2013-11-19 21:04:24,483::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-4922911::INFO::2013-11-19 21:04:24,484::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-4922911::DEBUG::2013-11-19 21:04:24,484::task::495::TaskManager.Task::(_debug) Task 4f37c8fd-7353-4c83-b509-63cb9c2250de: moving from state init -> state preparing >Thread-4922911::DEBUG::2013-11-19 21:04:24,485::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`b7ad225b-954f-40fe-9cbc-791d92fd418c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-4922911::DEBUG::2013-11-19 21:04:24,486::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-4922911::DEBUG::2013-11-19 21:04:24,486::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-4922911::DEBUG::2013-11-19 21:04:24,487::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`b7ad225b-954f-40fe-9cbc-791d92fd418c`::Granted request >Thread-4922911::DEBUG::2013-11-19 21:04:24,488::task::495::TaskManager.Task::(_debug) Task 4f37c8fd-7353-4c83-b509-63cb9c2250de: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-4922911::DEBUG::2013-11-19 21:04:24,488::task::495::TaskManager.Task::(_debug) Task 4f37c8fd-7353-4c83-b509-63cb9c2250de: ref 1 aborting False >Thread-4922911::DEBUG::2013-11-19 21:04:24,489::task::495::TaskManager.Task::(_debug) Task 4f37c8fd-7353-4c83-b509-63cb9c2250de: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-4922911::DEBUG::2013-11-19 21:04:24,489::task::495::TaskManager.Task::(_debug) Task 4f37c8fd-7353-4c83-b509-63cb9c2250de: moving from state preparing -> state finished >Thread-4922911::DEBUG::2013-11-19 21:04:24,490::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-4922911::DEBUG::2013-11-19 21:04:24,490::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4922911::DEBUG::2013-11-19 21:04:24,491::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-4922911::DEBUG::2013-11-19 21:04:24,492::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-4922911::DEBUG::2013-11-19 21:04:24,492::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-4922911::DEBUG::2013-11-19 21:04:24,493::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-4922911::DEBUG::2013-11-19 21:04:24,493::task::495::TaskManager.Task::(_debug) Task 4f37c8fd-7353-4c83-b509-63cb9c2250de: ref 0 aborting False >Thread-4922911::INFO::2013-11-19 21:04:24,494::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-4922911::INFO::2013-11-19 21:04:24,494::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-4922911::DEBUG::2013-11-19 21:04:24,495::task::495::TaskManager.Task::(_debug) Task fc6ca8ba-2241-4bf5-bde8-06accb662ca0: moving from state init -> state preparing >Thread-4922911::DEBUG::2013-11-19 21:04:24,496::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`bcea7f96-f18a-4957-a2be-a029d1c43b6e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-4922911::DEBUG::2013-11-19 21:04:24,497::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-4922911::DEBUG::2013-11-19 21:04:24,497::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-4922911::DEBUG::2013-11-19 21:04:24,498::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`bcea7f96-f18a-4957-a2be-a029d1c43b6e`::Granted request >Thread-4922911::DEBUG::2013-11-19 21:04:24,499::task::495::TaskManager.Task::(_debug) Task fc6ca8ba-2241-4bf5-bde8-06accb662ca0: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-4922911::DEBUG::2013-11-19 21:04:24,499::task::495::TaskManager.Task::(_debug) Task fc6ca8ba-2241-4bf5-bde8-06accb662ca0: ref 1 aborting False >Thread-4922911::DEBUG::2013-11-19 21:04:24,500::task::495::TaskManager.Task::(_debug) Task fc6ca8ba-2241-4bf5-bde8-06accb662ca0: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-4922911::DEBUG::2013-11-19 21:04:24,500::task::495::TaskManager.Task::(_debug) Task fc6ca8ba-2241-4bf5-bde8-06accb662ca0: moving from state preparing -> state finished >Thread-4922911::DEBUG::2013-11-19 21:04:24,501::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-4922911::DEBUG::2013-11-19 21:04:24,501::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4922911::DEBUG::2013-11-19 21:04:24,502::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-4922911::DEBUG::2013-11-19 21:04:24,503::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-4922911::DEBUG::2013-11-19 21:04:24,503::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-4922911::DEBUG::2013-11-19 21:04:24,504::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-4922911::DEBUG::2013-11-19 21:04:24,504::task::495::TaskManager.Task::(_debug) Task fc6ca8ba-2241-4bf5-bde8-06accb662ca0: ref 0 aborting False >Thread-4922911::INFO::2013-11-19 21:04:24,505::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15910158::DEBUG::2013-11-19 21:04:25,891::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910158::INFO::2013-11-19 21:04:25,892::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910158::DEBUG::2013-11-19 21:04:25,892::task::495::TaskManager.Task::(_debug) Task 455407ee-994f-4170-88e4-25a4d5c75416: moving from state init -> state preparing >Thread-15910158::DEBUG::2013-11-19 21:04:25,893::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910158::DEBUG::2013-11-19 21:04:25,894::task::495::TaskManager.Task::(_debug) Task 455407ee-994f-4170-88e4-25a4d5c75416: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910158::DEBUG::2013-11-19 21:04:25,894::task::495::TaskManager.Task::(_debug) Task 455407ee-994f-4170-88e4-25a4d5c75416: moving from state preparing -> state finished >Thread-15910158::DEBUG::2013-11-19 21:04:25,895::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910158::DEBUG::2013-11-19 21:04:25,895::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910158::DEBUG::2013-11-19 21:04:25,896::task::495::TaskManager.Task::(_debug) Task 455407ee-994f-4170-88e4-25a4d5c75416: ref 0 aborting False >Thread-15910158::INFO::2013-11-19 21:04:25,896::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910159::DEBUG::2013-11-19 21:04:25,901::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910159::INFO::2013-11-19 21:04:25,902::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910159::DEBUG::2013-11-19 21:04:25,903::task::495::TaskManager.Task::(_debug) Task 85c6b465-aca0-464e-ad83-29b81421b5cf: moving from state init -> state preparing >Thread-15910159::DEBUG::2013-11-19 21:04:25,905::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3acfe5f9-4a5f-46ee-bb2c-627f4be4f36e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910159::DEBUG::2013-11-19 21:04:25,906::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910159::DEBUG::2013-11-19 21:04:25,906::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910159::DEBUG::2013-11-19 21:04:25,907::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3acfe5f9-4a5f-46ee-bb2c-627f4be4f36e`::Granted request >Thread-15910159::DEBUG::2013-11-19 21:04:25,908::task::495::TaskManager.Task::(_debug) Task 85c6b465-aca0-464e-ad83-29b81421b5cf: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910159::DEBUG::2013-11-19 21:04:25,908::task::495::TaskManager.Task::(_debug) Task 85c6b465-aca0-464e-ad83-29b81421b5cf: ref 1 aborting False >Thread-15910159::WARNING::2013-11-19 21:04:25,910::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910159::DEBUG::2013-11-19 21:04:25,911::task::495::TaskManager.Task::(_debug) Task 85c6b465-aca0-464e-ad83-29b81421b5cf: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910159::DEBUG::2013-11-19 21:04:25,912::task::495::TaskManager.Task::(_debug) Task 85c6b465-aca0-464e-ad83-29b81421b5cf: moving from state preparing -> state finished >Thread-15910159::DEBUG::2013-11-19 21:04:25,913::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910159::DEBUG::2013-11-19 21:04:25,913::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910159::DEBUG::2013-11-19 21:04:25,914::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910159::DEBUG::2013-11-19 21:04:25,915::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910159::DEBUG::2013-11-19 21:04:25,915::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910159::DEBUG::2013-11-19 21:04:25,916::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910159::DEBUG::2013-11-19 21:04:25,916::task::495::TaskManager.Task::(_debug) Task 85c6b465-aca0-464e-ad83-29b81421b5cf: ref 0 aborting False >Thread-15910159::INFO::2013-11-19 21:04:25,917::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910161::DEBUG::2013-11-19 21:04:26,093::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910161::DEBUG::2013-11-19 21:04:26,094::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '92', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'UserLoggedOn', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663675', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.64', 'cpuUser': '207.50', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Dummy-15909965::DEBUG::2013-11-19 21:04:26,276::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:04:26,879::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0759159 s, 13.5 MB/s\n'; <rc> = 0 >Thread-15910162::INFO::2013-11-19 21:04:28,192::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910162::DEBUG::2013-11-19 21:04:28,193::task::495::TaskManager.Task::(_debug) Task a0f975c5-68ea-4894-aea7-32a21ee3ee5f: moving from state init -> state preparing >Thread-15910162::DEBUG::2013-11-19 21:04:28,194::task::495::TaskManager.Task::(_debug) Task a0f975c5-68ea-4894-aea7-32a21ee3ee5f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000324010848999', 'lastCheck': 1384895059.9078059, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00121212005615', 'lastCheck': 1384895065.95099, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895060.8186281, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00952196121216', 'lastCheck': 1384895058.8261299, 'valid': True, 'code': 0}} >Thread-15910162::DEBUG::2013-11-19 21:04:28,194::task::495::TaskManager.Task::(_debug) Task a0f975c5-68ea-4894-aea7-32a21ee3ee5f: moving from state preparing -> state finished >Thread-15910162::DEBUG::2013-11-19 21:04:28,195::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910162::DEBUG::2013-11-19 21:04:28,195::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910162::DEBUG::2013-11-19 21:04:28,196::task::495::TaskManager.Task::(_debug) Task a0f975c5-68ea-4894-aea7-32a21ee3ee5f: ref 0 aborting False >Thread-15910162::INFO::2013-11-19 21:04:28,197::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000324010848999', 'lastCheck': 1384895059.9078059, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00121212005615', 'lastCheck': 1384895065.95099, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895060.8186281, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00952196121216', 'lastCheck': 1384895058.8261299, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:04:28,892::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:04:29,424::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0737402 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15910165::DEBUG::2013-11-19 21:04:30,338::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910165::DEBUG::2013-11-19 21:04:30,339::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '92', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'UserLoggedOn', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663679', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.77', 'cpuUser': '207.50', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Thread-1024767::ERROR::2013-11-19 21:04:31,116::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:04:31,439::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:04:31,985::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0736101 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15910167::DEBUG::2013-11-19 21:04:32,416::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910167::DEBUG::2013-11-19 21:04:32,417::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '92', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'UserLoggedOn', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663681', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.85', 'cpuUser': '207.50', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Dummy-15909965::DEBUG::2013-11-19 21:04:34,001::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15910169::DEBUG::2013-11-19 21:04:34,502::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910169::DEBUG::2013-11-19 21:04:34,505::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '92', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'UserLoggedOn', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663683', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.93', 'cpuUser': '207.50', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Dummy-15909965::DEBUG::2013-11-19 21:04:34,555::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0734953 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15910170::DEBUG::2013-11-19 21:04:36,115::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910170::INFO::2013-11-19 21:04:36,116::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910170::DEBUG::2013-11-19 21:04:36,117::task::495::TaskManager.Task::(_debug) Task 2c72cbfc-3617-48d5-802f-46e587d912aa: moving from state init -> state preparing >Thread-15910170::DEBUG::2013-11-19 21:04:36,117::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910170::DEBUG::2013-11-19 21:04:36,118::task::495::TaskManager.Task::(_debug) Task 2c72cbfc-3617-48d5-802f-46e587d912aa: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910170::DEBUG::2013-11-19 21:04:36,119::task::495::TaskManager.Task::(_debug) Task 2c72cbfc-3617-48d5-802f-46e587d912aa: moving from state preparing -> state finished >Thread-15910170::DEBUG::2013-11-19 21:04:36,119::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910170::DEBUG::2013-11-19 21:04:36,120::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910170::DEBUG::2013-11-19 21:04:36,120::task::495::TaskManager.Task::(_debug) Task 2c72cbfc-3617-48d5-802f-46e587d912aa: ref 0 aborting False >Thread-15910170::INFO::2013-11-19 21:04:36,121::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910171::DEBUG::2013-11-19 21:04:36,126::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910171::INFO::2013-11-19 21:04:36,127::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910171::DEBUG::2013-11-19 21:04:36,127::task::495::TaskManager.Task::(_debug) Task 55f38594-d585-4238-936c-2f07856b4af6: moving from state init -> state preparing >Thread-15910171::DEBUG::2013-11-19 21:04:36,128::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`8e9f9448-1763-4880-ab4d-1455d79856e7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910171::DEBUG::2013-11-19 21:04:36,129::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910171::DEBUG::2013-11-19 21:04:36,130::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910171::DEBUG::2013-11-19 21:04:36,130::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`8e9f9448-1763-4880-ab4d-1455d79856e7`::Granted request >Thread-15910171::DEBUG::2013-11-19 21:04:36,131::task::495::TaskManager.Task::(_debug) Task 55f38594-d585-4238-936c-2f07856b4af6: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910171::DEBUG::2013-11-19 21:04:36,132::task::495::TaskManager.Task::(_debug) Task 55f38594-d585-4238-936c-2f07856b4af6: ref 1 aborting False >Thread-15910171::WARNING::2013-11-19 21:04:36,134::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910171::DEBUG::2013-11-19 21:04:36,135::task::495::TaskManager.Task::(_debug) Task 55f38594-d585-4238-936c-2f07856b4af6: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910171::DEBUG::2013-11-19 21:04:36,136::task::495::TaskManager.Task::(_debug) Task 55f38594-d585-4238-936c-2f07856b4af6: moving from state preparing -> state finished >Thread-15910171::DEBUG::2013-11-19 21:04:36,136::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910171::DEBUG::2013-11-19 21:04:36,137::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910171::DEBUG::2013-11-19 21:04:36,138::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910171::DEBUG::2013-11-19 21:04:36,138::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910171::DEBUG::2013-11-19 21:04:36,139::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910171::DEBUG::2013-11-19 21:04:36,139::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910171::DEBUG::2013-11-19 21:04:36,140::task::495::TaskManager.Task::(_debug) Task 55f38594-d585-4238-936c-2f07856b4af6: ref 0 aborting False >Thread-15910171::INFO::2013-11-19 21:04:36,141::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15909965::DEBUG::2013-11-19 21:04:36,573::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15910173::DEBUG::2013-11-19 21:04:36,803::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910173::DEBUG::2013-11-19 21:04:36,807::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '92', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'UserLoggedOn', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663686', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.23', 'cpuUser': '207.50', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Dummy-15909965::DEBUG::2013-11-19 21:04:37,115::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0742137 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15910174::INFO::2013-11-19 21:04:38,864::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910174::DEBUG::2013-11-19 21:04:38,865::task::495::TaskManager.Task::(_debug) Task 5aec09de-ea39-4c40-ab59-92e954b3a0d8: moving from state init -> state preparing >Thread-15910174::DEBUG::2013-11-19 21:04:38,866::task::495::TaskManager.Task::(_debug) Task 5aec09de-ea39-4c40-ab59-92e954b3a0d8: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000329971313477', 'lastCheck': 1384895070.1167641, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00170803070068', 'lastCheck': 1384895076.2175169, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895071.1171169, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000751972198486', 'lastCheck': 1384895069.11534, 'valid': True, 'code': 0}} >Thread-15910174::DEBUG::2013-11-19 21:04:38,867::task::495::TaskManager.Task::(_debug) Task 5aec09de-ea39-4c40-ab59-92e954b3a0d8: moving from state preparing -> state finished >Thread-15910174::DEBUG::2013-11-19 21:04:38,867::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910174::DEBUG::2013-11-19 21:04:38,868::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910174::DEBUG::2013-11-19 21:04:38,868::task::495::TaskManager.Task::(_debug) Task 5aec09de-ea39-4c40-ab59-92e954b3a0d8: ref 0 aborting False >Thread-15910174::INFO::2013-11-19 21:04:38,869::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000329971313477', 'lastCheck': 1384895070.1167641, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00170803070068', 'lastCheck': 1384895076.2175169, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895071.1171169, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000751972198486', 'lastCheck': 1384895069.11534, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:04:39,130::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:04:39,686::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0958235 s, 10.7 MB/s\n'; <rc> = 0 >Thread-15910177::DEBUG::2013-11-19 21:04:41,065::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910177::DEBUG::2013-11-19 21:04:41,067::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '92', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'UserLoggedOn', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663690', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.48', 'cpuUser': '139.33', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Thread-1024767::ERROR::2013-11-19 21:04:41,337::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:04:41,700::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:04:42,205::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0509604 s, 20.1 MB/s\n'; <rc> = 0 >Thread-15910179::DEBUG::2013-11-19 21:04:43,198::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910179::DEBUG::2013-11-19 21:04:43,200::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '92', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'UserLoggedOn', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663692', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.62', 'cpuUser': '139.33', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Dummy-15909965::DEBUG::2013-11-19 21:04:44,216::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:04:44,776::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0778722 s, 13.1 MB/s\n'; <rc> = 0 >Thread-15910181::DEBUG::2013-11-19 21:04:45,287::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910181::DEBUG::2013-11-19 21:04:45,289::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '92', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'UserLoggedOn', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663694', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.70', 'cpuUser': '139.33', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Thread-15910182::DEBUG::2013-11-19 21:04:46,221::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910182::INFO::2013-11-19 21:04:46,223::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910182::DEBUG::2013-11-19 21:04:46,224::task::495::TaskManager.Task::(_debug) Task 9a709b40-7d55-4988-ba50-1a6792c777ad: moving from state init -> state preparing >Thread-15910182::DEBUG::2013-11-19 21:04:46,224::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910182::DEBUG::2013-11-19 21:04:46,225::task::495::TaskManager.Task::(_debug) Task 9a709b40-7d55-4988-ba50-1a6792c777ad: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910182::DEBUG::2013-11-19 21:04:46,226::task::495::TaskManager.Task::(_debug) Task 9a709b40-7d55-4988-ba50-1a6792c777ad: moving from state preparing -> state finished >Thread-15910182::DEBUG::2013-11-19 21:04:46,226::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910182::DEBUG::2013-11-19 21:04:46,227::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910182::DEBUG::2013-11-19 21:04:46,227::task::495::TaskManager.Task::(_debug) Task 9a709b40-7d55-4988-ba50-1a6792c777ad: ref 0 aborting False >Thread-15910182::INFO::2013-11-19 21:04:46,228::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910183::DEBUG::2013-11-19 21:04:46,233::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910183::INFO::2013-11-19 21:04:46,233::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910183::DEBUG::2013-11-19 21:04:46,234::task::495::TaskManager.Task::(_debug) Task e977c7b9-e2a5-44ac-9ffb-12d647528273: moving from state init -> state preparing >Thread-15910183::DEBUG::2013-11-19 21:04:46,235::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ccd4e4fc-5391-4aff-89f8-744bd9377f01`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910183::DEBUG::2013-11-19 21:04:46,236::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910183::DEBUG::2013-11-19 21:04:46,237::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910183::DEBUG::2013-11-19 21:04:46,237::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ccd4e4fc-5391-4aff-89f8-744bd9377f01`::Granted request >Thread-15910183::DEBUG::2013-11-19 21:04:46,242::task::495::TaskManager.Task::(_debug) Task e977c7b9-e2a5-44ac-9ffb-12d647528273: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910183::DEBUG::2013-11-19 21:04:46,243::task::495::TaskManager.Task::(_debug) Task e977c7b9-e2a5-44ac-9ffb-12d647528273: ref 1 aborting False >Thread-15910183::WARNING::2013-11-19 21:04:46,245::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910183::DEBUG::2013-11-19 21:04:46,246::task::495::TaskManager.Task::(_debug) Task e977c7b9-e2a5-44ac-9ffb-12d647528273: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910183::DEBUG::2013-11-19 21:04:46,247::task::495::TaskManager.Task::(_debug) Task e977c7b9-e2a5-44ac-9ffb-12d647528273: moving from state preparing -> state finished >Thread-15910183::DEBUG::2013-11-19 21:04:46,247::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910183::DEBUG::2013-11-19 21:04:46,248::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910183::DEBUG::2013-11-19 21:04:46,249::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910183::DEBUG::2013-11-19 21:04:46,249::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910183::DEBUG::2013-11-19 21:04:46,250::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910183::DEBUG::2013-11-19 21:04:46,250::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910183::DEBUG::2013-11-19 21:04:46,251::task::495::TaskManager.Task::(_debug) Task e977c7b9-e2a5-44ac-9ffb-12d647528273: ref 0 aborting False >Thread-15910183::INFO::2013-11-19 21:04:46,252::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15909965::DEBUG::2013-11-19 21:04:46,791::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:04:47,338::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0775395 s, 13.2 MB/s\n'; <rc> = 0 >Thread-15910185::DEBUG::2013-11-19 21:04:47,385::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910185::DEBUG::2013-11-19 21:04:47,387::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '92', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'UserLoggedOn', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663696', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.80', 'cpuUser': '139.33', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Dummy-15909965::DEBUG::2013-11-19 21:04:49,353::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15910186::INFO::2013-11-19 21:04:49,544::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910186::DEBUG::2013-11-19 21:04:49,545::task::495::TaskManager.Task::(_debug) Task 5fa2b204-50a4-40ed-9c96-ed7a65909080: moving from state init -> state preparing >Thread-15910186::DEBUG::2013-11-19 21:04:49,546::task::495::TaskManager.Task::(_debug) Task 5fa2b204-50a4-40ed-9c96-ed7a65909080: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000360012054443', 'lastCheck': 1384895080.338366, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137996673584', 'lastCheck': 1384895086.4436231, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895081.338659, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000998020172119', 'lastCheck': 1384895089.5396891, 'valid': True, 'code': 0}} >Thread-15910186::DEBUG::2013-11-19 21:04:49,547::task::495::TaskManager.Task::(_debug) Task 5fa2b204-50a4-40ed-9c96-ed7a65909080: moving from state preparing -> state finished >Thread-15910186::DEBUG::2013-11-19 21:04:49,548::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910186::DEBUG::2013-11-19 21:04:49,549::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910186::DEBUG::2013-11-19 21:04:49,549::task::495::TaskManager.Task::(_debug) Task 5fa2b204-50a4-40ed-9c96-ed7a65909080: ref 0 aborting False >Thread-15910186::INFO::2013-11-19 21:04:49,550::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000360012054443', 'lastCheck': 1384895080.338366, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137996673584', 'lastCheck': 1384895086.4436231, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895081.338659, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000998020172119', 'lastCheck': 1384895089.5396891, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:04:49,820::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0579583 s, 17.7 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:04:51,544::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910189::DEBUG::2013-11-19 21:04:51,665::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910189::DEBUG::2013-11-19 21:04:51,666::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '92', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'UserLoggedOn', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663701', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.08', 'cpuUser': '139.33', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Dummy-15909965::DEBUG::2013-11-19 21:04:51,830::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:04:52,353::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0768156 s, 13.3 MB/s\n'; <rc> = 0 >Thread-15910191::DEBUG::2013-11-19 21:04:53,790::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910191::DEBUG::2013-11-19 21:04:53,791::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '92', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'UserLoggedOn', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663703', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.20', 'cpuUser': '139.33', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Dummy-15909965::DEBUG::2013-11-19 21:04:54,369::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:04:54,870::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0546511 s, 18.7 MB/s\n'; <rc> = 0 >Thread-15910193::DEBUG::2013-11-19 21:04:55,851::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910193::DEBUG::2013-11-19 21:04:55,852::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '92', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'UserLoggedOn', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663705', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.23', 'cpuUser': '172.48', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Thread-15910194::DEBUG::2013-11-19 21:04:56,412::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910194::INFO::2013-11-19 21:04:56,414::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910194::DEBUG::2013-11-19 21:04:56,415::task::495::TaskManager.Task::(_debug) Task b6a2055c-7c9e-4aef-8a86-0ff071e32cd8: moving from state init -> state preparing >Thread-15910194::DEBUG::2013-11-19 21:04:56,416::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910194::DEBUG::2013-11-19 21:04:56,416::task::495::TaskManager.Task::(_debug) Task b6a2055c-7c9e-4aef-8a86-0ff071e32cd8: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910194::DEBUG::2013-11-19 21:04:56,417::task::495::TaskManager.Task::(_debug) Task b6a2055c-7c9e-4aef-8a86-0ff071e32cd8: moving from state preparing -> state finished >Thread-15910194::DEBUG::2013-11-19 21:04:56,418::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910194::DEBUG::2013-11-19 21:04:56,418::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910194::DEBUG::2013-11-19 21:04:56,419::task::495::TaskManager.Task::(_debug) Task b6a2055c-7c9e-4aef-8a86-0ff071e32cd8: ref 0 aborting False >Thread-15910194::INFO::2013-11-19 21:04:56,420::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910195::DEBUG::2013-11-19 21:04:56,424::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910195::INFO::2013-11-19 21:04:56,425::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910195::DEBUG::2013-11-19 21:04:56,426::task::495::TaskManager.Task::(_debug) Task 39f8fc9c-aa50-4258-bc4a-861b38ae973a: moving from state init -> state preparing >Thread-15910195::DEBUG::2013-11-19 21:04:56,427::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`88abad86-ca7a-403c-bc59-f165a52ba377`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910195::DEBUG::2013-11-19 21:04:56,428::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910195::DEBUG::2013-11-19 21:04:56,429::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910195::DEBUG::2013-11-19 21:04:56,429::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`88abad86-ca7a-403c-bc59-f165a52ba377`::Granted request >Thread-15910195::DEBUG::2013-11-19 21:04:56,434::task::495::TaskManager.Task::(_debug) Task 39f8fc9c-aa50-4258-bc4a-861b38ae973a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910195::DEBUG::2013-11-19 21:04:56,435::task::495::TaskManager.Task::(_debug) Task 39f8fc9c-aa50-4258-bc4a-861b38ae973a: ref 1 aborting False >Thread-15910195::WARNING::2013-11-19 21:04:56,438::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910195::DEBUG::2013-11-19 21:04:56,439::task::495::TaskManager.Task::(_debug) Task 39f8fc9c-aa50-4258-bc4a-861b38ae973a: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910195::DEBUG::2013-11-19 21:04:56,440::task::495::TaskManager.Task::(_debug) Task 39f8fc9c-aa50-4258-bc4a-861b38ae973a: moving from state preparing -> state finished >Thread-15910195::DEBUG::2013-11-19 21:04:56,441::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910195::DEBUG::2013-11-19 21:04:56,441::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910195::DEBUG::2013-11-19 21:04:56,442::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910195::DEBUG::2013-11-19 21:04:56,443::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910195::DEBUG::2013-11-19 21:04:56,443::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910195::DEBUG::2013-11-19 21:04:56,444::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910195::DEBUG::2013-11-19 21:04:56,445::task::495::TaskManager.Task::(_debug) Task 39f8fc9c-aa50-4258-bc4a-861b38ae973a: ref 0 aborting False >Thread-15910195::INFO::2013-11-19 21:04:56,446::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-4922912::DEBUG::2013-11-19 21:04:56,691::guestIF::97::vm.Vm::(_handleMessage) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::u'None' >Dummy-15909965::DEBUG::2013-11-19 21:04:56,881::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:04:57,513::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0886279 s, 11.6 MB/s\n'; <rc> = 0 >Thread-15910197::DEBUG::2013-11-19 21:04:57,942::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910197::DEBUG::2013-11-19 21:04:57,944::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '92', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'LoggedOff', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663707', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.28', 'cpuUser': '172.48', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Dummy-15909965::DEBUG::2013-11-19 21:04:59,523::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:05:00,039::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0747381 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15910198::INFO::2013-11-19 21:05:00,074::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910198::DEBUG::2013-11-19 21:05:00,075::task::495::TaskManager.Task::(_debug) Task 6720d63b-bbde-4736-94f3-aa7ae5afac72: moving from state init -> state preparing >Thread-15910198::DEBUG::2013-11-19 21:05:00,076::task::495::TaskManager.Task::(_debug) Task 6720d63b-bbde-4736-94f3-aa7ae5afac72: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000296115875244', 'lastCheck': 1384895090.5389211, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00112795829773', 'lastCheck': 1384895096.6210091, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895091.545125, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00135493278503', 'lastCheck': 1384895099.738771, 'valid': True, 'code': 0}} >Thread-15910198::DEBUG::2013-11-19 21:05:00,077::task::495::TaskManager.Task::(_debug) Task 6720d63b-bbde-4736-94f3-aa7ae5afac72: moving from state preparing -> state finished >Thread-15910198::DEBUG::2013-11-19 21:05:00,078::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910198::DEBUG::2013-11-19 21:05:00,078::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910198::DEBUG::2013-11-19 21:05:00,079::task::495::TaskManager.Task::(_debug) Task 6720d63b-bbde-4736-94f3-aa7ae5afac72: ref 0 aborting False >Thread-15910198::INFO::2013-11-19 21:05:00,080::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000296115875244', 'lastCheck': 1384895090.5389211, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00112795829773', 'lastCheck': 1384895096.6210091, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895091.545125, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00135493278503', 'lastCheck': 1384895099.738771, 'valid': True, 'code': 0}} >Thread-15910153::DEBUG::2013-11-19 21:05:01,445::vm::692::vm.Vm::(_timedShutdown) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::_timedShutdown Called >Thread-1024767::ERROR::2013-11-19 21:05:01,742::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:05:02,055::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15910201::DEBUG::2013-11-19 21:05:02,327::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910201::DEBUG::2013-11-19 21:05:02,342::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '91', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'LoggedOff', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663711', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.59', 'cpuUser': '172.48', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Dummy-15909965::DEBUG::2013-11-19 21:05:02,648::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0822316 s, 12.5 MB/s\n'; <rc> = 0 >Thread-4922912::DEBUG::2013-11-19 21:05:03,671::guestIF::97::vm.Vm::(_handleMessage) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::u'None' >Thread-15910203::DEBUG::2013-11-19 21:05:04,491::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910203::DEBUG::2013-11-19 21:05:04,492::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '91', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'LoggedOff', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663713', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.75', 'cpuUser': '172.48', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Dummy-15909965::DEBUG::2013-11-19 21:05:04,662::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:05:05,261::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0812764 s, 12.6 MB/s\n'; <rc> = 0 >Thread-15910204::DEBUG::2013-11-19 21:05:06,614::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910204::INFO::2013-11-19 21:05:06,617::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910204::DEBUG::2013-11-19 21:05:06,623::task::495::TaskManager.Task::(_debug) Task a6196c2a-5b31-41b0-8296-51a3985f525c: moving from state init -> state preparing >Thread-15910204::DEBUG::2013-11-19 21:05:06,627::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910204::DEBUG::2013-11-19 21:05:06,629::task::495::TaskManager.Task::(_debug) Task a6196c2a-5b31-41b0-8296-51a3985f525c: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910204::DEBUG::2013-11-19 21:05:06,631::task::495::TaskManager.Task::(_debug) Task a6196c2a-5b31-41b0-8296-51a3985f525c: moving from state preparing -> state finished >Thread-15910204::DEBUG::2013-11-19 21:05:06,636::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910204::DEBUG::2013-11-19 21:05:06,639::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910204::DEBUG::2013-11-19 21:05:06,642::task::495::TaskManager.Task::(_debug) Task a6196c2a-5b31-41b0-8296-51a3985f525c: ref 0 aborting False >Thread-15910204::INFO::2013-11-19 21:05:06,646::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910207::DEBUG::2013-11-19 21:05:06,697::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910207::DEBUG::2013-11-19 21:05:06,698::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '91', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'LoggedOff', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663716', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.79', 'cpuUser': '172.48', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Thread-15910206::DEBUG::2013-11-19 21:05:06,703::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910206::INFO::2013-11-19 21:05:06,704::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910206::DEBUG::2013-11-19 21:05:06,705::task::495::TaskManager.Task::(_debug) Task bc69bf5c-e9ce-4f38-9913-41eefa127ee9: moving from state init -> state preparing >Thread-15910206::DEBUG::2013-11-19 21:05:06,711::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`32804ff6-4298-471b-aeda-8d77e2f02f98`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910206::DEBUG::2013-11-19 21:05:06,716::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910206::DEBUG::2013-11-19 21:05:06,717::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910206::DEBUG::2013-11-19 21:05:06,718::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`32804ff6-4298-471b-aeda-8d77e2f02f98`::Granted request >Thread-15910206::DEBUG::2013-11-19 21:05:06,719::task::495::TaskManager.Task::(_debug) Task bc69bf5c-e9ce-4f38-9913-41eefa127ee9: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910206::DEBUG::2013-11-19 21:05:06,722::task::495::TaskManager.Task::(_debug) Task bc69bf5c-e9ce-4f38-9913-41eefa127ee9: ref 1 aborting False >Thread-15910206::WARNING::2013-11-19 21:05:06,729::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910206::DEBUG::2013-11-19 21:05:06,748::task::495::TaskManager.Task::(_debug) Task bc69bf5c-e9ce-4f38-9913-41eefa127ee9: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910206::DEBUG::2013-11-19 21:05:06,768::task::495::TaskManager.Task::(_debug) Task bc69bf5c-e9ce-4f38-9913-41eefa127ee9: moving from state preparing -> state finished >Thread-15910206::DEBUG::2013-11-19 21:05:06,768::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910206::DEBUG::2013-11-19 21:05:06,774::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910206::DEBUG::2013-11-19 21:05:06,775::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910206::DEBUG::2013-11-19 21:05:06,784::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910206::DEBUG::2013-11-19 21:05:06,785::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910206::DEBUG::2013-11-19 21:05:06,785::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910206::DEBUG::2013-11-19 21:05:06,786::task::495::TaskManager.Task::(_debug) Task bc69bf5c-e9ce-4f38-9913-41eefa127ee9: ref 0 aborting False >Thread-15910206::INFO::2013-11-19 21:05:06,795::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15909965::DEBUG::2013-11-19 21:05:07,276::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:05:07,855::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0780894 s, 13.1 MB/s\n'; <rc> = 0 >Thread-15910209::DEBUG::2013-11-19 21:05:08,794::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910209::DEBUG::2013-11-19 21:05:08,797::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '91', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'LoggedOff', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663718', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.88', 'cpuUser': '172.48', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Dummy-15909965::DEBUG::2013-11-19 21:05:09,871::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:05:10,247::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0436375 s, 23.5 MB/s\n'; <rc> = 0 >Thread-15910210::INFO::2013-11-19 21:05:10,948::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910210::DEBUG::2013-11-19 21:05:10,949::task::495::TaskManager.Task::(_debug) Task 2332005f-5c81-41a1-9fa8-a768b95a0e00: moving from state init -> state preparing >Thread-15910210::DEBUG::2013-11-19 21:05:10,951::task::495::TaskManager.Task::(_debug) Task 2332005f-5c81-41a1-9fa8-a768b95a0e00: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000334978103638', 'lastCheck': 1384895100.739496, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148987770081', 'lastCheck': 1384895106.9134469, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895101.7436941, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000579833984375', 'lastCheck': 1384895110.038691, 'valid': True, 'code': 0}} >Thread-15910210::DEBUG::2013-11-19 21:05:10,967::task::495::TaskManager.Task::(_debug) Task 2332005f-5c81-41a1-9fa8-a768b95a0e00: moving from state preparing -> state finished >Thread-15910210::DEBUG::2013-11-19 21:05:10,968::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910210::DEBUG::2013-11-19 21:05:10,969::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910210::DEBUG::2013-11-19 21:05:10,970::task::495::TaskManager.Task::(_debug) Task 2332005f-5c81-41a1-9fa8-a768b95a0e00: ref 0 aborting False >Thread-15910210::INFO::2013-11-19 21:05:10,975::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000334978103638', 'lastCheck': 1384895100.739496, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148987770081', 'lastCheck': 1384895106.9134469, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895101.7436941, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000579833984375', 'lastCheck': 1384895110.038691, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:05:12,039::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:05:12,257::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:05:12,818::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.109251 s, 9.4 MB/s\n'; <rc> = 0 >Thread-15910213::DEBUG::2013-11-19 21:05:13,132::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910213::DEBUG::2013-11-19 21:05:13,134::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '91', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'LoggedOff', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663722', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.09', 'cpuUser': '175.49', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Thread-4922912::DEBUG::2013-11-19 21:05:14,508::guestIF::112::vm.Vm::(_handleMessage) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Guest system shuts down. >libvirtEventLoop::DEBUG::2013-11-19 21:05:14,595::vm::525::vm.Vm::(_rtcUpdate) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::new rtc offset -1 >libvirtEventLoop::DEBUG::2013-11-19 21:05:14,598::vm::525::vm.Vm::(_rtcUpdate) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::new rtc offset -1 >libvirtEventLoop::DEBUG::2013-11-19 21:05:14,603::vm::525::vm.Vm::(_rtcUpdate) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::new rtc offset -1 >libvirtEventLoop::DEBUG::2013-11-19 21:05:14,605::vm::525::vm.Vm::(_rtcUpdate) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::new rtc offset -1 >libvirtEventLoop::DEBUG::2013-11-19 21:05:14,606::vm::525::vm.Vm::(_rtcUpdate) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::new rtc offset -1 >libvirtEventLoop::DEBUG::2013-11-19 21:05:14,608::vm::525::vm.Vm::(_rtcUpdate) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::new rtc offset -1 >libvirtEventLoop::DEBUG::2013-11-19 21:05:14,609::vm::525::vm.Vm::(_rtcUpdate) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::new rtc offset -1 >Dummy-15909965::DEBUG::2013-11-19 21:05:14,846::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15910215::DEBUG::2013-11-19 21:05:15,262::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910215::DEBUG::2013-11-19 21:05:15,264::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '91', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'LoggedOff', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663724', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.19', 'cpuUser': '175.49', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Dummy-15909965::DEBUG::2013-11-19 21:05:15,512::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.148877 s, 6.9 MB/s\n'; <rc> = 0 >Thread-15910216::DEBUG::2013-11-19 21:05:16,925::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910216::INFO::2013-11-19 21:05:16,926::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910216::DEBUG::2013-11-19 21:05:16,927::task::495::TaskManager.Task::(_debug) Task 5c65d653-481c-4bf8-a3e6-6ece3a326f75: moving from state init -> state preparing >Thread-15910216::DEBUG::2013-11-19 21:05:16,928::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910216::DEBUG::2013-11-19 21:05:16,929::task::495::TaskManager.Task::(_debug) Task 5c65d653-481c-4bf8-a3e6-6ece3a326f75: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910216::DEBUG::2013-11-19 21:05:16,936::task::495::TaskManager.Task::(_debug) Task 5c65d653-481c-4bf8-a3e6-6ece3a326f75: moving from state preparing -> state finished >Thread-15910216::DEBUG::2013-11-19 21:05:16,940::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910216::DEBUG::2013-11-19 21:05:16,940::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910216::DEBUG::2013-11-19 21:05:16,941::task::495::TaskManager.Task::(_debug) Task 5c65d653-481c-4bf8-a3e6-6ece3a326f75: ref 0 aborting False >Thread-15910216::INFO::2013-11-19 21:05:16,941::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910217::DEBUG::2013-11-19 21:05:16,955::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910217::INFO::2013-11-19 21:05:16,956::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910217::DEBUG::2013-11-19 21:05:16,957::task::495::TaskManager.Task::(_debug) Task 80481dac-54e5-4b92-ab0e-a4f91b9dad41: moving from state init -> state preparing >Thread-15910217::DEBUG::2013-11-19 21:05:16,960::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c2d9d2d7-53c4-449e-b4b3-a17435af21b0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910217::DEBUG::2013-11-19 21:05:16,966::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910217::DEBUG::2013-11-19 21:05:16,967::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910217::DEBUG::2013-11-19 21:05:16,971::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c2d9d2d7-53c4-449e-b4b3-a17435af21b0`::Granted request >Thread-15910217::DEBUG::2013-11-19 21:05:16,980::task::495::TaskManager.Task::(_debug) Task 80481dac-54e5-4b92-ab0e-a4f91b9dad41: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910217::DEBUG::2013-11-19 21:05:16,981::task::495::TaskManager.Task::(_debug) Task 80481dac-54e5-4b92-ab0e-a4f91b9dad41: ref 1 aborting False >Thread-15910217::WARNING::2013-11-19 21:05:16,991::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910217::DEBUG::2013-11-19 21:05:17,001::task::495::TaskManager.Task::(_debug) Task 80481dac-54e5-4b92-ab0e-a4f91b9dad41: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910217::DEBUG::2013-11-19 21:05:17,013::task::495::TaskManager.Task::(_debug) Task 80481dac-54e5-4b92-ab0e-a4f91b9dad41: moving from state preparing -> state finished >Thread-15910217::DEBUG::2013-11-19 21:05:17,014::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910217::DEBUG::2013-11-19 21:05:17,023::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910217::DEBUG::2013-11-19 21:05:17,024::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910217::DEBUG::2013-11-19 21:05:17,025::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910217::DEBUG::2013-11-19 21:05:17,026::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910217::DEBUG::2013-11-19 21:05:17,027::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910217::DEBUG::2013-11-19 21:05:17,028::task::495::TaskManager.Task::(_debug) Task 80481dac-54e5-4b92-ab0e-a4f91b9dad41: ref 0 aborting False >Thread-15910217::INFO::2013-11-19 21:05:17,028::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910219::DEBUG::2013-11-19 21:05:17,445::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910219::DEBUG::2013-11-19 21:05:17,449::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '91', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'LoggedOff', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663726', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.37', 'cpuUser': '175.49', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Dummy-15909965::DEBUG::2013-11-19 21:05:17,526::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:05:18,129::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0786111 s, 13.0 MB/s\n'; <rc> = 0 >Thread-15910221::DEBUG::2013-11-19 21:05:19,574::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910221::DEBUG::2013-11-19 21:05:19,576::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '91', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'LoggedOff', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663729', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.49', 'cpuUser': '175.49', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Dummy-15909965::DEBUG::2013-11-19 21:05:20,141::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:05:20,787::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0945618 s, 10.8 MB/s\n'; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:05:21,211::libvirtvm::1401::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::event Saved detail 0 opaque None >Thread-15910222::INFO::2013-11-19 21:05:21,715::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910222::DEBUG::2013-11-19 21:05:21,716::task::495::TaskManager.Task::(_debug) Task 53f89f69-0c8c-4125-b49a-44c1f892b333: moving from state init -> state preparing >Thread-15910222::DEBUG::2013-11-19 21:05:21,717::task::495::TaskManager.Task::(_debug) Task 53f89f69-0c8c-4125-b49a-44c1f892b333: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000465869903564', 'lastCheck': 1384895121.08833, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144600868225', 'lastCheck': 1384895117.086225, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895112.040729, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000355958938599', 'lastCheck': 1384895120.082762, 'valid': True, 'code': 0}} >Thread-15910222::DEBUG::2013-11-19 21:05:21,743::task::495::TaskManager.Task::(_debug) Task 53f89f69-0c8c-4125-b49a-44c1f892b333: moving from state preparing -> state finished >Thread-15910222::DEBUG::2013-11-19 21:05:21,744::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910222::DEBUG::2013-11-19 21:05:21,744::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910222::DEBUG::2013-11-19 21:05:21,745::task::495::TaskManager.Task::(_debug) Task 53f89f69-0c8c-4125-b49a-44c1f892b333: ref 0 aborting False >Thread-15910222::INFO::2013-11-19 21:05:21,746::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000465869903564', 'lastCheck': 1384895121.08833, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144600868225', 'lastCheck': 1384895117.086225, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895112.040729, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000355958938599', 'lastCheck': 1384895120.082762, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:05:22,085::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15909965::DEBUG::2013-11-19 21:05:22,803::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:05:23,346::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0828931 s, 12.4 MB/s\n'; <rc> = 0 >Thread-15910225::DEBUG::2013-11-19 21:05:24,017::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910225::DEBUG::2013-11-19 21:05:24,021::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '91', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'LoggedOff', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663733', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '0.93', 'cpuUser': '175.49', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Thread-4922911::INFO::2013-11-19 21:05:25,089::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-4922911::DEBUG::2013-11-19 21:05:25,095::task::495::TaskManager.Task::(_debug) Task 93c1a13e-4053-490f-a818-be632d9d5483: moving from state init -> state preparing >Thread-4922911::DEBUG::2013-11-19 21:05:25,101::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`407955dd-5dd9-40f6-922e-2ec4897c5e11`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-4922911::DEBUG::2013-11-19 21:05:25,101::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-4922911::DEBUG::2013-11-19 21:05:25,102::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-4922911::DEBUG::2013-11-19 21:05:25,102::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`407955dd-5dd9-40f6-922e-2ec4897c5e11`::Granted request >Thread-4922911::DEBUG::2013-11-19 21:05:25,108::task::495::TaskManager.Task::(_debug) Task 93c1a13e-4053-490f-a818-be632d9d5483: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-4922911::DEBUG::2013-11-19 21:05:25,112::task::495::TaskManager.Task::(_debug) Task 93c1a13e-4053-490f-a818-be632d9d5483: ref 1 aborting False >Thread-4922911::DEBUG::2013-11-19 21:05:25,118::task::495::TaskManager.Task::(_debug) Task 93c1a13e-4053-490f-a818-be632d9d5483: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-4922911::DEBUG::2013-11-19 21:05:25,118::task::495::TaskManager.Task::(_debug) Task 93c1a13e-4053-490f-a818-be632d9d5483: moving from state preparing -> state finished >Thread-4922911::DEBUG::2013-11-19 21:05:25,118::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-4922911::DEBUG::2013-11-19 21:05:25,119::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4922911::DEBUG::2013-11-19 21:05:25,121::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-4922911::DEBUG::2013-11-19 21:05:25,123::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-4922911::DEBUG::2013-11-19 21:05:25,124::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-4922911::DEBUG::2013-11-19 21:05:25,129::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-4922911::DEBUG::2013-11-19 21:05:25,129::task::495::TaskManager.Task::(_debug) Task 93c1a13e-4053-490f-a818-be632d9d5483: ref 0 aborting False >Thread-4922911::INFO::2013-11-19 21:05:25,131::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-4922911::INFO::2013-11-19 21:05:25,132::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-4922911::DEBUG::2013-11-19 21:05:25,137::task::495::TaskManager.Task::(_debug) Task 73ea90f1-dbd6-4b94-95fe-f93a97b80759: moving from state init -> state preparing >Thread-4922911::DEBUG::2013-11-19 21:05:25,138::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`a3b2c6f6-6fa2-4ee9-9505-252912cb6da1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-4922911::DEBUG::2013-11-19 21:05:25,143::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-4922911::DEBUG::2013-11-19 21:05:25,143::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-4922911::DEBUG::2013-11-19 21:05:25,146::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`a3b2c6f6-6fa2-4ee9-9505-252912cb6da1`::Granted request >Thread-4922911::DEBUG::2013-11-19 21:05:25,152::task::495::TaskManager.Task::(_debug) Task 73ea90f1-dbd6-4b94-95fe-f93a97b80759: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-4922911::DEBUG::2013-11-19 21:05:25,155::task::495::TaskManager.Task::(_debug) Task 73ea90f1-dbd6-4b94-95fe-f93a97b80759: ref 1 aborting False >Thread-4922911::DEBUG::2013-11-19 21:05:25,155::task::495::TaskManager.Task::(_debug) Task 73ea90f1-dbd6-4b94-95fe-f93a97b80759: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-4922911::DEBUG::2013-11-19 21:05:25,158::task::495::TaskManager.Task::(_debug) Task 73ea90f1-dbd6-4b94-95fe-f93a97b80759: moving from state preparing -> state finished >Thread-4922911::DEBUG::2013-11-19 21:05:25,163::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-4922911::DEBUG::2013-11-19 21:05:25,163::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4922911::DEBUG::2013-11-19 21:05:25,167::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-4922911::DEBUG::2013-11-19 21:05:25,173::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-4922911::DEBUG::2013-11-19 21:05:25,176::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-4922911::DEBUG::2013-11-19 21:05:25,176::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-4922911::DEBUG::2013-11-19 21:05:25,181::task::495::TaskManager.Task::(_debug) Task 73ea90f1-dbd6-4b94-95fe-f93a97b80759: ref 0 aborting False >Thread-4922911::INFO::2013-11-19 21:05:25,181::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-4922911::INFO::2013-11-19 21:05:25,182::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-4922911::DEBUG::2013-11-19 21:05:25,187::task::495::TaskManager.Task::(_debug) Task 31d6c011-6cbf-443e-970a-811caf9b7bb7: moving from state init -> state preparing >Thread-4922911::DEBUG::2013-11-19 21:05:25,189::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`a8eabbd5-ed9a-4e68-af4c-5f60dd031919`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-4922911::DEBUG::2013-11-19 21:05:25,195::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-4922911::DEBUG::2013-11-19 21:05:25,199::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-4922911::DEBUG::2013-11-19 21:05:25,200::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`a8eabbd5-ed9a-4e68-af4c-5f60dd031919`::Granted request >Thread-4922911::DEBUG::2013-11-19 21:05:25,207::task::495::TaskManager.Task::(_debug) Task 31d6c011-6cbf-443e-970a-811caf9b7bb7: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-4922911::DEBUG::2013-11-19 21:05:25,210::task::495::TaskManager.Task::(_debug) Task 31d6c011-6cbf-443e-970a-811caf9b7bb7: ref 1 aborting False >Thread-4922911::DEBUG::2013-11-19 21:05:25,210::task::495::TaskManager.Task::(_debug) Task 31d6c011-6cbf-443e-970a-811caf9b7bb7: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-4922911::DEBUG::2013-11-19 21:05:25,211::task::495::TaskManager.Task::(_debug) Task 31d6c011-6cbf-443e-970a-811caf9b7bb7: moving from state preparing -> state finished >Thread-4922911::DEBUG::2013-11-19 21:05:25,211::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-4922911::DEBUG::2013-11-19 21:05:25,211::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4922911::DEBUG::2013-11-19 21:05:25,212::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-4922911::DEBUG::2013-11-19 21:05:25,217::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-4922911::DEBUG::2013-11-19 21:05:25,221::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-4922911::DEBUG::2013-11-19 21:05:25,221::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-4922911::DEBUG::2013-11-19 21:05:25,222::task::495::TaskManager.Task::(_debug) Task 31d6c011-6cbf-443e-970a-811caf9b7bb7: ref 0 aborting False >Thread-4922911::INFO::2013-11-19 21:05:25,222::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-4922911::INFO::2013-11-19 21:05:25,222::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-4922911::DEBUG::2013-11-19 21:05:25,223::task::495::TaskManager.Task::(_debug) Task e8c8710c-a140-4ed4-89a0-14718fbbeafb: moving from state init -> state preparing >Thread-4922911::DEBUG::2013-11-19 21:05:25,225::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`b42d073e-4255-4883-822e-9502c4345fbb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-4922911::DEBUG::2013-11-19 21:05:25,231::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-4922911::DEBUG::2013-11-19 21:05:25,236::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-4922911::DEBUG::2013-11-19 21:05:25,242::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`b42d073e-4255-4883-822e-9502c4345fbb`::Granted request >Thread-4922911::DEBUG::2013-11-19 21:05:25,242::task::495::TaskManager.Task::(_debug) Task e8c8710c-a140-4ed4-89a0-14718fbbeafb: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-4922911::DEBUG::2013-11-19 21:05:25,243::task::495::TaskManager.Task::(_debug) Task e8c8710c-a140-4ed4-89a0-14718fbbeafb: ref 1 aborting False >Thread-4922911::DEBUG::2013-11-19 21:05:25,247::task::495::TaskManager.Task::(_debug) Task e8c8710c-a140-4ed4-89a0-14718fbbeafb: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-4922911::DEBUG::2013-11-19 21:05:25,248::task::495::TaskManager.Task::(_debug) Task e8c8710c-a140-4ed4-89a0-14718fbbeafb: moving from state preparing -> state finished >Thread-4922911::DEBUG::2013-11-19 21:05:25,248::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-4922911::DEBUG::2013-11-19 21:05:25,249::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-4922911::DEBUG::2013-11-19 21:05:25,250::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-4922911::DEBUG::2013-11-19 21:05:25,253::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-4922911::DEBUG::2013-11-19 21:05:25,258::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-4922911::DEBUG::2013-11-19 21:05:25,258::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-4922911::DEBUG::2013-11-19 21:05:25,259::task::495::TaskManager.Task::(_debug) Task e8c8710c-a140-4ed4-89a0-14718fbbeafb: ref 0 aborting False >Thread-4922911::INFO::2013-11-19 21:05:25,260::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15909965::DEBUG::2013-11-19 21:05:25,360::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:05:25,876::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0743445 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15910227::DEBUG::2013-11-19 21:05:26,106::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910227::DEBUG::2013-11-19 21:05:26,107::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '91', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'LoggedOff', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663735', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '1.02', 'cpuUser': '144.52', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Thread-15910228::DEBUG::2013-11-19 21:05:27,160::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910228::INFO::2013-11-19 21:05:27,162::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910228::DEBUG::2013-11-19 21:05:27,163::task::495::TaskManager.Task::(_debug) Task 491c6fda-03b3-427a-9589-1dd587c8b2b7: moving from state init -> state preparing >Thread-15910228::DEBUG::2013-11-19 21:05:27,165::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910228::DEBUG::2013-11-19 21:05:27,165::task::495::TaskManager.Task::(_debug) Task 491c6fda-03b3-427a-9589-1dd587c8b2b7: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910228::DEBUG::2013-11-19 21:05:27,166::task::495::TaskManager.Task::(_debug) Task 491c6fda-03b3-427a-9589-1dd587c8b2b7: moving from state preparing -> state finished >Thread-15910228::DEBUG::2013-11-19 21:05:27,167::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910228::DEBUG::2013-11-19 21:05:27,167::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910228::DEBUG::2013-11-19 21:05:27,168::task::495::TaskManager.Task::(_debug) Task 491c6fda-03b3-427a-9589-1dd587c8b2b7: ref 0 aborting False >Thread-15910228::INFO::2013-11-19 21:05:27,168::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910229::DEBUG::2013-11-19 21:05:27,175::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910229::INFO::2013-11-19 21:05:27,175::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910229::DEBUG::2013-11-19 21:05:27,175::task::495::TaskManager.Task::(_debug) Task 77e0d661-181c-4680-b626-825fa748d29d: moving from state init -> state preparing >Thread-15910229::DEBUG::2013-11-19 21:05:27,181::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`4aa04163-746a-4e7d-9308-53c63073de63`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910229::DEBUG::2013-11-19 21:05:27,181::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910229::DEBUG::2013-11-19 21:05:27,182::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910229::DEBUG::2013-11-19 21:05:27,182::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`4aa04163-746a-4e7d-9308-53c63073de63`::Granted request >Thread-15910229::DEBUG::2013-11-19 21:05:27,183::task::495::TaskManager.Task::(_debug) Task 77e0d661-181c-4680-b626-825fa748d29d: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910229::DEBUG::2013-11-19 21:05:27,183::task::495::TaskManager.Task::(_debug) Task 77e0d661-181c-4680-b626-825fa748d29d: ref 1 aborting False >Thread-15910229::WARNING::2013-11-19 21:05:27,190::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910229::DEBUG::2013-11-19 21:05:27,198::task::495::TaskManager.Task::(_debug) Task 77e0d661-181c-4680-b626-825fa748d29d: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910229::DEBUG::2013-11-19 21:05:27,211::task::495::TaskManager.Task::(_debug) Task 77e0d661-181c-4680-b626-825fa748d29d: moving from state preparing -> state finished >Thread-15910229::DEBUG::2013-11-19 21:05:27,212::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910229::DEBUG::2013-11-19 21:05:27,212::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910229::DEBUG::2013-11-19 21:05:27,217::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910229::DEBUG::2013-11-19 21:05:27,218::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910229::DEBUG::2013-11-19 21:05:27,218::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910229::DEBUG::2013-11-19 21:05:27,222::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910229::DEBUG::2013-11-19 21:05:27,223::task::495::TaskManager.Task::(_debug) Task 77e0d661-181c-4680-b626-825fa748d29d: ref 0 aborting False >Thread-15910229::INFO::2013-11-19 21:05:27,223::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15909965::DEBUG::2013-11-19 21:05:27,889::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15910231::DEBUG::2013-11-19 21:05:28,258::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910231::DEBUG::2013-11-19 21:05:28,263::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '91', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'LoggedOff', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663737', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '3.17', 'cpuUser': '144.52', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Dummy-15909965::DEBUG::2013-11-19 21:05:28,417::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0979412 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:05:30,433::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15910233::DEBUG::2013-11-19 21:05:30,724::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910233::DEBUG::2013-11-19 21:05:30,736::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '91', 'acpiEnable': 'true', 'pid': '57291', 'displayPort': u'5908', 'session': 'LoggedOff', 'netIfaces': [{u'inet6': [], u'hw': u'00-1a-4a-a8-64-08', u'inet': [u'192.168.100.70'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': -1L, 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '38686343168', u'fs': u'NTFS'}, {u'path': u'h:\\', u'total': '107237863424', u'used': '99742478336', u'fs': u'NTFS'}, {u'path': u'l:\\', u'total': '150187474944', u'used': '130882863104', u'fs': u'NTFS'}, {u'path': u'x:\\', u'total': '429360349184', u'used': '330617978880', u'fs': u'NTFS'}], 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'displayType': 'qxl', 'guestName': u'KLEE-AXDB01', 'boot': 'cd', 'elapsedTime': '18663740', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Microsoft Report Viewer Redistributable 2008 SP1', u'Zabbix Agent 2.0.8', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'Microsoft SQL Server Compact 3.5 SP2 ENU', u'RHEV-Spice-Agent64', u'Microsoft Visual Studio Tools for Applications 2.0 - ENU', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'Microsoft Office 2003 Web Components', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10', u'Microsoft SQL Server Browser', u'Microsoft Report Viewer Redistributable 2008 (KB971119)', u'Microsoft SQL Server 2008 R2 Policies', u'Microsoft SQL Server Compact 3.5 SP2 Query Tools ENU'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Powering down', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'displayIp': '0', 'lastLogout': 1366231390.568368, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '3652045', 'apparentsize': '64424509440', 'writeLatency': '312295', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '183455', 'readRate': '145382.38', 'truesize': '64424509440', 'writeRate': '189200.51'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '107374182400', 'writeRate': '0.00'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '150323855360', 'writeRate': '0.00'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'readRate': '0.00', 'truesize': '429496729600', 'writeRate': '0.00'}}, 'monitorResponse': '0', 'statsAge': '5.65', 'cpuUser': '144.52', 'cpuIdle': '0.00', 'lastLogin': 1384895046.5037899, 'guestIPs': u'192.168.100.70', 'displaySecurePort': u'5909'}]} >Dummy-15909965::DEBUG::2013-11-19 21:05:30,982::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0741267 s, 13.8 MB/s\n'; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,176::libvirtvm::1401::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::event Stopped detail 0 opaque None >Thread-4922911::ERROR::2013-11-19 21:05:31,177::utils::377::vm.Vm::(collect) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Stats function failed: <AdvancedStatsFunction _sampleDisk at 0x1b96148> >Traceback (most recent call last): > File "/usr/share/vdsm/utils.py", line 373, in collect > File "/usr/share/vdsm/utils.py", line 250, in __call__ > File "/usr/share/vdsm/libvirtvm.py", line 101, in _sampleDisk > File "/usr/share/vdsm/libvirtvm.py", line 445, in f > File "/usr/share/vdsm/libvirtconnection.py", line 63, in wrapper > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1687, in blockStats >libvirtError: Unable to read from monitor: Connection reset by peer >libvirtEventLoop::INFO::2013-11-19 21:05:31,178::vm::445::vm.Vm::(_onQemuDeath) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::underlying process disconnected >libvirtEventLoop::INFO::2013-11-19 21:05:31,187::libvirtvm::1304::vm.Vm::(releaseVm) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Release VM resources >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,205::utils::314::vm.Vm::(stop) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Stop statistics collection >Thread-4922911::DEBUG::2013-11-19 21:05:31,207::utils::345::vm.Vm::(run) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Stats thread finished >libvirtEventLoop::WARNING::2013-11-19 21:05:31,210::libvirtvm::1315::vm.Vm::(releaseVm) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Traceback (most recent call last): > File "/usr/share/vdsm/libvirtvm.py", line 1312, in releaseVm > File "/usr/share/vdsm/libvirtvm.py", line 445, in f > File "/usr/share/vdsm/libvirtconnection.py", line 63, in wrapper > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 672, in destroy >libvirtError: Domain not found: no domain with matching uuid '6574d88c-00ac-46f4-99ce-a2726932bfda' > >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,212::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /sbin/service ksmtuned retune' (cwd None) >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,661::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,663::vm::718::vm.Vm::(_cleanup) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Drive /rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/772472cb-2429-466d-96cd-00b3301ef78a/images/d368307e-425e-423a-85b7-266c1e4a99ec/fdc91117-476f-4b82-8fc4-d682fa283ecc cleanup >libvirtEventLoop::INFO::2013-11-19 21:05:31,665::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: teardownVolume, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,666::task::495::TaskManager.Task::(_debug) Task 03a96d28-dbe3-4697-aeab-759d31bc7387: moving from state init -> state preparing >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,667::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`b56d8e44-8e7a-413c-9f3a-00a32b453478`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1727' at 'public_teardownVolume' >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,667::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,668::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,669::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`b56d8e44-8e7a-413c-9f3a-00a32b453478`::Granted request >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,670::task::495::TaskManager.Task::(_debug) Task 03a96d28-dbe3-4697-aeab-759d31bc7387: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,670::task::495::TaskManager.Task::(_debug) Task 03a96d28-dbe3-4697-aeab-759d31bc7387: ref 1 aborting False >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,671::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.d368307e-425e-423a-85b7-266c1e4a99ec' >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,672::resourceManager::530::ResourceManager::(releaseResource) Released resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.d368307e-425e-423a-85b7-266c1e4a99ec' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,672::resourceManager::535::ResourceManager::(releaseResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.d368307e-425e-423a-85b7-266c1e4a99ec' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,673::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.fdc91117-476f-4b82-8fc4-d682fa283ecc' >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,673::resourceManager::530::ResourceManager::(releaseResource) Released resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.fdc91117-476f-4b82-8fc4-d682fa283ecc' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,674::resourceManager::535::ResourceManager::(releaseResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.fdc91117-476f-4b82-8fc4-d682fa283ecc' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,674::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.fdc91117-476f-4b82-8fc4-d682fa283ecc', Clearing records. >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,675::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.d368307e-425e-423a-85b7-266c1e4a99ec', Clearing records. >libvirtEventLoop::INFO::2013-11-19 21:05:31,676::blockVolume::386::Storage.Volume::(teardown) Tearing down volume 772472cb-2429-466d-96cd-00b3301ef78a/fdc91117-476f-4b82-8fc4-d682fa283ecc justme False >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,676::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.fdc91117-476f-4b82-8fc4-d682fa283ecc' >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,677::resourceManager::530::ResourceManager::(releaseResource) Released resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.fdc91117-476f-4b82-8fc4-d682fa283ecc' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,677::resourceManager::535::ResourceManager::(releaseResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.fdc91117-476f-4b82-8fc4-d682fa283ecc' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:05:31,678::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --autobackup n --available n 772472cb-2429-466d-96cd-00b3301ef78a/fdc91117-476f-4b82-8fc4-d682fa283ecc' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:05:32,100::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >libvirtEventLoop::DEBUG::2013-11-19 21:05:32,445::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:05:32,446::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:05:32,447::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:05:32,447::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.fdc91117-476f-4b82-8fc4-d682fa283ecc', Clearing records. >libvirtEventLoop::DEBUG::2013-11-19 21:05:32,448::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:05:32,449::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15910234::INFO::2013-11-19 21:05:32,855::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910234::DEBUG::2013-11-19 21:05:32,856::task::495::TaskManager.Task::(_debug) Task 5e9b5241-84a7-4099-920d-aa05f95cd98a: moving from state init -> state preparing >Thread-15910234::DEBUG::2013-11-19 21:05:32,857::task::495::TaskManager.Task::(_debug) Task 5e9b5241-84a7-4099-920d-aa05f95cd98a: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000362873077393', 'lastCheck': 1384895131.101028, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00131106376648', 'lastCheck': 1384895127.1006629, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895132.101685, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000398874282837', 'lastCheck': 1384895130.094244, 'valid': True, 'code': 0}} >Thread-15910234::DEBUG::2013-11-19 21:05:32,858::task::495::TaskManager.Task::(_debug) Task 5e9b5241-84a7-4099-920d-aa05f95cd98a: moving from state preparing -> state finished >Thread-15910234::DEBUG::2013-11-19 21:05:32,858::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910234::DEBUG::2013-11-19 21:05:32,859::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910234::DEBUG::2013-11-19 21:05:32,859::task::495::TaskManager.Task::(_debug) Task 5e9b5241-84a7-4099-920d-aa05f95cd98a: ref 0 aborting False >Thread-15910234::INFO::2013-11-19 21:05:32,860::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000362873077393', 'lastCheck': 1384895131.101028, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00131106376648', 'lastCheck': 1384895127.1006629, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895132.101685, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000398874282837', 'lastCheck': 1384895130.094244, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:05:32,999::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,228::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,248::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,249::task::495::TaskManager.Task::(_debug) Task 03a96d28-dbe3-4697-aeab-759d31bc7387: finished: None >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,250::task::495::TaskManager.Task::(_debug) Task 03a96d28-dbe3-4697-aeab-759d31bc7387: moving from state preparing -> state finished >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,250::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,251::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,252::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,253::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,253::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,254::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,254::task::495::TaskManager.Task::(_debug) Task 03a96d28-dbe3-4697-aeab-759d31bc7387: ref 0 aborting False >libvirtEventLoop::INFO::2013-11-19 21:05:33,255::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: teardownVolume, Return response: {'status': {'message': 'OK', 'code': 0}} >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,256::vm::718::vm.Vm::(_cleanup) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Drive None cleanup >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,256::vm::718::vm.Vm::(_cleanup) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Drive /rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/772472cb-2429-466d-96cd-00b3301ef78a/images/df8b790c-c762-4feb-9378-d64a321ce43c/2e903e43-1016-4339-a7f3-c5723a995725 cleanup >libvirtEventLoop::INFO::2013-11-19 21:05:33,257::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: teardownVolume, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,258::task::495::TaskManager.Task::(_debug) Task 0ac4afa1-2ce3-431f-ad53-3cdb138cc7fa: moving from state init -> state preparing >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,259::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`98bba4a6-eff2-4264-9f84-f4462c37bf00`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1727' at 'public_teardownVolume' >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,260::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,261::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,261::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`98bba4a6-eff2-4264-9f84-f4462c37bf00`::Granted request >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,262::task::495::TaskManager.Task::(_debug) Task 0ac4afa1-2ce3-431f-ad53-3cdb138cc7fa: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,263::task::495::TaskManager.Task::(_debug) Task 0ac4afa1-2ce3-431f-ad53-3cdb138cc7fa: ref 1 aborting False >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,263::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.df8b790c-c762-4feb-9378-d64a321ce43c' >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,264::resourceManager::530::ResourceManager::(releaseResource) Released resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.df8b790c-c762-4feb-9378-d64a321ce43c' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,265::resourceManager::535::ResourceManager::(releaseResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.df8b790c-c762-4feb-9378-d64a321ce43c' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,265::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.2e903e43-1016-4339-a7f3-c5723a995725' >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,266::resourceManager::530::ResourceManager::(releaseResource) Released resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.2e903e43-1016-4339-a7f3-c5723a995725' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,266::resourceManager::535::ResourceManager::(releaseResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.2e903e43-1016-4339-a7f3-c5723a995725' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,267::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.2e903e43-1016-4339-a7f3-c5723a995725', Clearing records. >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,267::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.df8b790c-c762-4feb-9378-d64a321ce43c', Clearing records. >libvirtEventLoop::INFO::2013-11-19 21:05:33,268::blockVolume::386::Storage.Volume::(teardown) Tearing down volume 772472cb-2429-466d-96cd-00b3301ef78a/2e903e43-1016-4339-a7f3-c5723a995725 justme False >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,269::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.2e903e43-1016-4339-a7f3-c5723a995725' >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,269::resourceManager::530::ResourceManager::(releaseResource) Released resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.2e903e43-1016-4339-a7f3-c5723a995725' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,270::resourceManager::535::ResourceManager::(releaseResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.2e903e43-1016-4339-a7f3-c5723a995725' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,271::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --autobackup n --available n 772472cb-2429-466d-96cd-00b3301ef78a/2e903e43-1016-4339-a7f3-c5723a995725' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:05:33,666::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.152164 s, 6.7 MB/s\n'; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,971::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,973::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,973::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,974::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.2e903e43-1016-4339-a7f3-c5723a995725', Clearing records. >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,975::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:05:33,976::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,624::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,651::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,652::task::495::TaskManager.Task::(_debug) Task 0ac4afa1-2ce3-431f-ad53-3cdb138cc7fa: finished: None >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,652::task::495::TaskManager.Task::(_debug) Task 0ac4afa1-2ce3-431f-ad53-3cdb138cc7fa: moving from state preparing -> state finished >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,653::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,653::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,654::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,655::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,655::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,656::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,657::task::495::TaskManager.Task::(_debug) Task 0ac4afa1-2ce3-431f-ad53-3cdb138cc7fa: ref 0 aborting False >libvirtEventLoop::INFO::2013-11-19 21:05:34,657::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: teardownVolume, Return response: {'status': {'message': 'OK', 'code': 0}} >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,658::vm::718::vm.Vm::(_cleanup) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Drive /rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/772472cb-2429-466d-96cd-00b3301ef78a/images/c4460480-32d3-469f-b0f4-c03bb6d74e41/e6984064-e09e-4c0c-8b51-a8a1711994cf cleanup >libvirtEventLoop::INFO::2013-11-19 21:05:34,659::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: teardownVolume, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,659::task::495::TaskManager.Task::(_debug) Task 7d493234-f77b-418f-9ed2-0a24a3b9c15d: moving from state init -> state preparing >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,660::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`c0f840eb-fb9a-4bf8-966b-eff58b6b3b37`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1727' at 'public_teardownVolume' >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,661::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,661::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,662::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`c0f840eb-fb9a-4bf8-966b-eff58b6b3b37`::Granted request >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,663::task::495::TaskManager.Task::(_debug) Task 7d493234-f77b-418f-9ed2-0a24a3b9c15d: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,663::task::495::TaskManager.Task::(_debug) Task 7d493234-f77b-418f-9ed2-0a24a3b9c15d: ref 1 aborting False >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,664::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.c4460480-32d3-469f-b0f4-c03bb6d74e41' >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,665::resourceManager::530::ResourceManager::(releaseResource) Released resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.c4460480-32d3-469f-b0f4-c03bb6d74e41' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,665::resourceManager::535::ResourceManager::(releaseResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.c4460480-32d3-469f-b0f4-c03bb6d74e41' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,666::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.e6984064-e09e-4c0c-8b51-a8a1711994cf' >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,666::resourceManager::530::ResourceManager::(releaseResource) Released resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.e6984064-e09e-4c0c-8b51-a8a1711994cf' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,667::resourceManager::535::ResourceManager::(releaseResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.e6984064-e09e-4c0c-8b51-a8a1711994cf' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,667::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.e6984064-e09e-4c0c-8b51-a8a1711994cf', Clearing records. >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,668::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.c4460480-32d3-469f-b0f4-c03bb6d74e41', Clearing records. >libvirtEventLoop::INFO::2013-11-19 21:05:34,668::blockVolume::386::Storage.Volume::(teardown) Tearing down volume 772472cb-2429-466d-96cd-00b3301ef78a/e6984064-e09e-4c0c-8b51-a8a1711994cf justme False >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,669::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.e6984064-e09e-4c0c-8b51-a8a1711994cf' >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,669::resourceManager::530::ResourceManager::(releaseResource) Released resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.e6984064-e09e-4c0c-8b51-a8a1711994cf' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,670::resourceManager::535::ResourceManager::(releaseResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.e6984064-e09e-4c0c-8b51-a8a1711994cf' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:05:34,671::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --autobackup n --available n 772472cb-2429-466d-96cd-00b3301ef78a/e6984064-e09e-4c0c-8b51-a8a1711994cf' (cwd None) >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,283::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,285::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,285::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,286::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.e6984064-e09e-4c0c-8b51-a8a1711994cf', Clearing records. >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,287::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,288::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:05:35,723::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,943::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,963::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,964::task::495::TaskManager.Task::(_debug) Task 7d493234-f77b-418f-9ed2-0a24a3b9c15d: finished: None >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,964::task::495::TaskManager.Task::(_debug) Task 7d493234-f77b-418f-9ed2-0a24a3b9c15d: moving from state preparing -> state finished >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,965::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,966::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,967::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,967::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,968::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,968::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,969::task::495::TaskManager.Task::(_debug) Task 7d493234-f77b-418f-9ed2-0a24a3b9c15d: ref 0 aborting False >libvirtEventLoop::INFO::2013-11-19 21:05:35,970::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: teardownVolume, Return response: {'status': {'message': 'OK', 'code': 0}} >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,970::vm::718::vm.Vm::(_cleanup) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Drive /rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/772472cb-2429-466d-96cd-00b3301ef78a/images/13efb45b-f4d2-4c94-bbdb-91ce9b9b7321/088640f7-316c-4e2b-bfdc-0f64cf2913c8 cleanup >libvirtEventLoop::INFO::2013-11-19 21:05:35,972::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: teardownVolume, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,972::task::495::TaskManager.Task::(_debug) Task f9dcf002-cb98-43cd-a054-32a0ca4f488b: moving from state init -> state preparing >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,973::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`20abb79d-7689-4795-9e64-7343d3683033`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1727' at 'public_teardownVolume' >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,974::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,974::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,975::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`20abb79d-7689-4795-9e64-7343d3683033`::Granted request >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,976::task::495::TaskManager.Task::(_debug) Task f9dcf002-cb98-43cd-a054-32a0ca4f488b: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,976::task::495::TaskManager.Task::(_debug) Task f9dcf002-cb98-43cd-a054-32a0ca4f488b: ref 1 aborting False >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,977::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.13efb45b-f4d2-4c94-bbdb-91ce9b9b7321' >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,978::resourceManager::530::ResourceManager::(releaseResource) Released resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.13efb45b-f4d2-4c94-bbdb-91ce9b9b7321' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,978::resourceManager::535::ResourceManager::(releaseResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.13efb45b-f4d2-4c94-bbdb-91ce9b9b7321' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,979::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.088640f7-316c-4e2b-bfdc-0f64cf2913c8' >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,979::resourceManager::530::ResourceManager::(releaseResource) Released resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.088640f7-316c-4e2b-bfdc-0f64cf2913c8' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,980::resourceManager::535::ResourceManager::(releaseResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.088640f7-316c-4e2b-bfdc-0f64cf2913c8' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,980::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.088640f7-316c-4e2b-bfdc-0f64cf2913c8', Clearing records. >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,981::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', Clearing records. >libvirtEventLoop::INFO::2013-11-19 21:05:35,982::blockVolume::386::Storage.Volume::(teardown) Tearing down volume 772472cb-2429-466d-96cd-00b3301ef78a/088640f7-316c-4e2b-bfdc-0f64cf2913c8 justme False >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,982::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.088640f7-316c-4e2b-bfdc-0f64cf2913c8' >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,983::resourceManager::530::ResourceManager::(releaseResource) Released resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.088640f7-316c-4e2b-bfdc-0f64cf2913c8' (0 active users) >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,983::resourceManager::535::ResourceManager::(releaseResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.088640f7-316c-4e2b-bfdc-0f64cf2913c8' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-19 21:05:35,984::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --autobackup n --available n 772472cb-2429-466d-96cd-00b3301ef78a/088640f7-316c-4e2b-bfdc-0f64cf2913c8' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:05:36,445::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.155637 s, 6.6 MB/s\n'; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:05:36,757::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:05:36,758::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:05:36,759::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:05:36,760::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.088640f7-316c-4e2b-bfdc-0f64cf2913c8', Clearing records. >libvirtEventLoop::DEBUG::2013-11-19 21:05:36,761::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:05:36,762::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >libvirtEventLoop::DEBUG::2013-11-19 21:05:37,367::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-19 21:05:37,393::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >libvirtEventLoop::DEBUG::2013-11-19 21:05:37,394::task::495::TaskManager.Task::(_debug) Task f9dcf002-cb98-43cd-a054-32a0ca4f488b: finished: None >libvirtEventLoop::DEBUG::2013-11-19 21:05:37,395::task::495::TaskManager.Task::(_debug) Task f9dcf002-cb98-43cd-a054-32a0ca4f488b: moving from state preparing -> state finished >Thread-15910238::DEBUG::2013-11-19 21:05:37,396::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >libvirtEventLoop::DEBUG::2013-11-19 21:05:37,397::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >libvirtEventLoop::DEBUG::2013-11-19 21:05:37,402::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910238::INFO::2013-11-19 21:05:37,403::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >libvirtEventLoop::DEBUG::2013-11-19 21:05:37,404::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15910238::DEBUG::2013-11-19 21:05:37,410::task::495::TaskManager.Task::(_debug) Task 4e53534e-ac5a-4a8c-9b1f-b65da1fc8a0a: moving from state init -> state preparing >Thread-15910238::DEBUG::2013-11-19 21:05:37,411::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >libvirtEventLoop::DEBUG::2013-11-19 21:05:37,412::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15910238::DEBUG::2013-11-19 21:05:37,412::task::495::TaskManager.Task::(_debug) Task 4e53534e-ac5a-4a8c-9b1f-b65da1fc8a0a: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >libvirtEventLoop::DEBUG::2013-11-19 21:05:37,413::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15910238::DEBUG::2013-11-19 21:05:37,414::task::495::TaskManager.Task::(_debug) Task 4e53534e-ac5a-4a8c-9b1f-b65da1fc8a0a: moving from state preparing -> state finished >libvirtEventLoop::DEBUG::2013-11-19 21:05:37,414::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15910238::DEBUG::2013-11-19 21:05:37,415::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >libvirtEventLoop::DEBUG::2013-11-19 21:05:37,416::task::495::TaskManager.Task::(_debug) Task f9dcf002-cb98-43cd-a054-32a0ca4f488b: ref 0 aborting False >Thread-15910238::DEBUG::2013-11-19 21:05:37,416::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >libvirtEventLoop::INFO::2013-11-19 21:05:37,417::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: teardownVolume, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15910238::DEBUG::2013-11-19 21:05:37,418::task::495::TaskManager.Task::(_debug) Task 4e53534e-ac5a-4a8c-9b1f-b65da1fc8a0a: ref 0 aborting False >Thread-15910238::INFO::2013-11-19 21:05:37,419::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >libvirtEventLoop::DEBUG::2013-11-19 21:05:37,421::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/libexec/vdsm/hooks/after_vm_destroy/50_vhostmd' (cwd None) >Thread-15910239::DEBUG::2013-11-19 21:05:37,656::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910239::INFO::2013-11-19 21:05:37,659::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910239::DEBUG::2013-11-19 21:05:37,660::task::495::TaskManager.Task::(_debug) Task 4d602ac7-9093-448a-83ae-df5181f00118: moving from state init -> state preparing >Thread-15910239::DEBUG::2013-11-19 21:05:37,662::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3edf6fee-a3a2-405b-9b01-226b5a9e71fe`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910239::DEBUG::2013-11-19 21:05:37,663::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910239::DEBUG::2013-11-19 21:05:37,665::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910239::DEBUG::2013-11-19 21:05:37,666::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3edf6fee-a3a2-405b-9b01-226b5a9e71fe`::Granted request >Thread-15910239::DEBUG::2013-11-19 21:05:37,668::task::495::TaskManager.Task::(_debug) Task 4d602ac7-9093-448a-83ae-df5181f00118: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910239::DEBUG::2013-11-19 21:05:37,669::task::495::TaskManager.Task::(_debug) Task 4d602ac7-9093-448a-83ae-df5181f00118: ref 1 aborting False >Thread-15910239::WARNING::2013-11-19 21:05:37,673::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910239::DEBUG::2013-11-19 21:05:37,681::task::495::TaskManager.Task::(_debug) Task 4d602ac7-9093-448a-83ae-df5181f00118: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910239::DEBUG::2013-11-19 21:05:37,690::task::495::TaskManager.Task::(_debug) Task 4d602ac7-9093-448a-83ae-df5181f00118: moving from state preparing -> state finished >Thread-15910239::DEBUG::2013-11-19 21:05:37,691::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910239::DEBUG::2013-11-19 21:05:37,693::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910239::DEBUG::2013-11-19 21:05:37,693::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910239::DEBUG::2013-11-19 21:05:37,694::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910239::DEBUG::2013-11-19 21:05:37,695::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910239::DEBUG::2013-11-19 21:05:37,695::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910239::DEBUG::2013-11-19 21:05:37,696::task::495::TaskManager.Task::(_debug) Task 4d602ac7-9093-448a-83ae-df5181f00118: ref 0 aborting False >Thread-15910239::INFO::2013-11-19 21:05:37,697::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >libvirtEventLoop::DEBUG::2013-11-19 21:05:38,308::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >libvirtEventLoop::INFO::2013-11-19 21:05:38,309::hooks::51::root::(_runHooksDir) >libvirtEventLoop::DEBUG::2013-11-19 21:05:38,317::vm::742::vm.Vm::(setDownStatus) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Changed state to Down: User shut down >libvirtEventLoop::DEBUG::2013-11-19 21:05:38,318::utils::314::vm.Vm::(stop) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Stop statistics collection >Dummy-15909965::DEBUG::2013-11-19 21:05:38,462::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:05:39,005::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0738998 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15910242::DEBUG::2013-11-19 21:05:39,145::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910242::DEBUG::2013-11-19 21:05:39,146::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Down', 'timeOffset': -1L, 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'exitMessage': 'User shut down', 'exitCode': 0}]} >Thread-15910243::DEBUG::2013-11-19 21:05:39,159::clientIF::54::vds::(wrapper) [192.168.100.135]::call destroy with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15910243::INFO::2013-11-19 21:05:39,160::clientIF::450::vds::(destroy) vmContainerLock acquired by vm 6574d88c-00ac-46f4-99ce-a2726932bfda >Thread-15910243::DEBUG::2013-11-19 21:05:39,160::libvirtvm::1344::vm.Vm::(destroy) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::destroy Called >Thread-15910243::DEBUG::2013-11-19 21:05:39,161::libvirtvm::1339::vm.Vm::(deleteVm) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Total desktops after destroy of 6574d88c-00ac-46f4-99ce-a2726932bfda is 0 >Thread-15910243::DEBUG::2013-11-19 21:05:39,162::clientIF::59::vds::(wrapper) return destroy with {'status': {'message': 'Machine destroyed', 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:05:41,020::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:05:41,573::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0739456 s, 13.8 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:05:42,316::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910245::INFO::2013-11-19 21:05:43,310::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910245::DEBUG::2013-11-19 21:05:43,311::task::495::TaskManager.Task::(_debug) Task 853ba3b9-f6d8-4d1e-935d-75445f77a73d: moving from state init -> state preparing >Thread-15910245::DEBUG::2013-11-19 21:05:43,311::task::495::TaskManager.Task::(_debug) Task 853ba3b9-f6d8-4d1e-935d-75445f77a73d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000452995300293', 'lastCheck': 1384895141.3012021, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00221800804138', 'lastCheck': 1384895137.3147531, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895142.317908, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000633001327515', 'lastCheck': 1384895140.305815, 'valid': True, 'code': 0}} >Thread-15910245::DEBUG::2013-11-19 21:05:43,312::task::495::TaskManager.Task::(_debug) Task 853ba3b9-f6d8-4d1e-935d-75445f77a73d: moving from state preparing -> state finished >Thread-15910245::DEBUG::2013-11-19 21:05:43,313::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910245::DEBUG::2013-11-19 21:05:43,313::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910245::DEBUG::2013-11-19 21:05:43,314::task::495::TaskManager.Task::(_debug) Task 853ba3b9-f6d8-4d1e-935d-75445f77a73d: ref 0 aborting False >Thread-15910245::INFO::2013-11-19 21:05:43,315::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000452995300293', 'lastCheck': 1384895141.3012021, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00221800804138', 'lastCheck': 1384895137.3147531, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895142.317908, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000633001327515', 'lastCheck': 1384895140.305815, 'valid': True, 'code': 0}} >Dummy-15909965::DEBUG::2013-11-19 21:05:43,589::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:05:44,120::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0742872 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:05:46,135::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:05:46,651::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0732189 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15910249::DEBUG::2013-11-19 21:05:47,816::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910249::INFO::2013-11-19 21:05:47,818::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910249::DEBUG::2013-11-19 21:05:47,819::task::495::TaskManager.Task::(_debug) Task 883eca27-1678-4328-ad8d-cedcc347cfca: moving from state init -> state preparing >Thread-15910249::DEBUG::2013-11-19 21:05:47,820::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910249::DEBUG::2013-11-19 21:05:47,821::task::495::TaskManager.Task::(_debug) Task 883eca27-1678-4328-ad8d-cedcc347cfca: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910249::DEBUG::2013-11-19 21:05:47,822::task::495::TaskManager.Task::(_debug) Task 883eca27-1678-4328-ad8d-cedcc347cfca: moving from state preparing -> state finished >Thread-15910249::DEBUG::2013-11-19 21:05:47,822::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910249::DEBUG::2013-11-19 21:05:47,823::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910249::DEBUG::2013-11-19 21:05:47,824::task::495::TaskManager.Task::(_debug) Task 883eca27-1678-4328-ad8d-cedcc347cfca: ref 0 aborting False >Thread-15910249::INFO::2013-11-19 21:05:47,825::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910250::DEBUG::2013-11-19 21:05:47,836::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910250::INFO::2013-11-19 21:05:47,837::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910250::DEBUG::2013-11-19 21:05:47,838::task::495::TaskManager.Task::(_debug) Task 1aa6642b-2929-4be2-8144-17b048d21f0e: moving from state init -> state preparing >Thread-15910250::DEBUG::2013-11-19 21:05:47,839::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`04aea70c-f457-4184-8700-044528fb10c9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910250::DEBUG::2013-11-19 21:05:47,839::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910250::DEBUG::2013-11-19 21:05:47,840::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910250::DEBUG::2013-11-19 21:05:47,841::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`04aea70c-f457-4184-8700-044528fb10c9`::Granted request >Thread-15910250::DEBUG::2013-11-19 21:05:47,844::task::495::TaskManager.Task::(_debug) Task 1aa6642b-2929-4be2-8144-17b048d21f0e: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910250::DEBUG::2013-11-19 21:05:47,845::task::495::TaskManager.Task::(_debug) Task 1aa6642b-2929-4be2-8144-17b048d21f0e: ref 1 aborting False >Thread-15910250::WARNING::2013-11-19 21:05:47,847::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910250::DEBUG::2013-11-19 21:05:47,855::task::495::TaskManager.Task::(_debug) Task 1aa6642b-2929-4be2-8144-17b048d21f0e: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910250::DEBUG::2013-11-19 21:05:47,859::task::495::TaskManager.Task::(_debug) Task 1aa6642b-2929-4be2-8144-17b048d21f0e: moving from state preparing -> state finished >Thread-15910250::DEBUG::2013-11-19 21:05:47,859::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910250::DEBUG::2013-11-19 21:05:47,860::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910250::DEBUG::2013-11-19 21:05:47,860::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910250::DEBUG::2013-11-19 21:05:47,861::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910250::DEBUG::2013-11-19 21:05:47,862::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910250::DEBUG::2013-11-19 21:05:47,862::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910250::DEBUG::2013-11-19 21:05:47,863::task::495::TaskManager.Task::(_debug) Task 1aa6642b-2929-4be2-8144-17b048d21f0e: ref 0 aborting False >Thread-15910250::INFO::2013-11-19 21:05:47,863::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 274}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15909965::DEBUG::2013-11-19 21:05:48,666::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:05:49,215::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0743069 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15909965::DEBUG::2013-11-19 21:05:51,230::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15909965::DEBUG::2013-11-19 21:05:51,754::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0750319 s, 13.6 MB/s\n'; <rc> = 0 >Thread-15910253::DEBUG::2013-11-19 21:05:52,345::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910253::INFO::2013-11-19 21:05:52,352::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksStatuses, args: () >Thread-15910253::DEBUG::2013-11-19 21:05:52,353::task::495::TaskManager.Task::(_debug) Task f73f2d85-3c96-4de2-b35a-30b89271318b: moving from state init -> state preparing >Thread-15910253::DEBUG::2013-11-19 21:05:52,354::taskManager::92::TaskManager::(getAllTasksStatuses) Entry. >Thread-15910253::DEBUG::2013-11-19 21:05:52,354::taskManager::97::TaskManager::(getAllTasksStatuses) Return: {} >Thread-15910253::DEBUG::2013-11-19 21:05:52,355::task::495::TaskManager.Task::(_debug) Task f73f2d85-3c96-4de2-b35a-30b89271318b: finished: {'allTasksStatus': {}} >Thread-15910253::DEBUG::2013-11-19 21:05:52,355::task::495::TaskManager.Task::(_debug) Task f73f2d85-3c96-4de2-b35a-30b89271318b: moving from state preparing -> state finished >Thread-15910253::DEBUG::2013-11-19 21:05:52,356::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910253::DEBUG::2013-11-19 21:05:52,356::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910253::DEBUG::2013-11-19 21:05:52,357::task::495::TaskManager.Task::(_debug) Task f73f2d85-3c96-4de2-b35a-30b89271318b: ref 0 aborting False >Thread-15910253::INFO::2013-11-19 21:05:52,357::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksStatuses, Return response: {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}} >Thread-15910254::DEBUG::2013-11-19 21:05:52,384::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910254::INFO::2013-11-19 21:05:52,385::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: deactivateStorageDomain, args: ( sdUUID=5d7b5178-3790-472c-a13b-b481eb27a61e spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 msdUUID=00000000-0000-0000-0000-000000000000 masterVersion=1) >Thread-15910254::DEBUG::2013-11-19 21:05:52,386::task::495::TaskManager.Task::(_debug) Task fc43eb4b-607e-44ef-b0dc-a0fa60171dd9: moving from state init -> state preparing >Thread-15910254::DEBUG::2013-11-19 21:05:52,387::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`506b49c6-d267-4d73-a8d6-def453f3268f`::Request was made in '/usr/share/vdsm/storage/spm.py' line '1112' at 'public_deactivateStorageDomain' >Thread-15910254::DEBUG::2013-11-19 21:05:52,388::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >Thread-15910254::DEBUG::2013-11-19 21:05:52,389::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >Thread-15910254::DEBUG::2013-11-19 21:05:52,389::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`506b49c6-d267-4d73-a8d6-def453f3268f`::Granted request >Thread-15910254::DEBUG::2013-11-19 21:05:52,395::task::495::TaskManager.Task::(_debug) Task fc43eb4b-607e-44ef-b0dc-a0fa60171dd9: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (exclusive) >Thread-15910254::DEBUG::2013-11-19 21:05:52,395::task::495::TaskManager.Task::(_debug) Task fc43eb4b-607e-44ef-b0dc-a0fa60171dd9: ref 1 aborting False >Thread-15910254::DEBUG::2013-11-19 21:05:52,400::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`501cbbc7-efde-463f-917f-61d451f94c7d`::Request was made in '/usr/share/vdsm/storage/spm.py' line '1113' at 'public_deactivateStorageDomain' >Thread-15910254::DEBUG::2013-11-19 21:05:52,400::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' for lock type 'exclusive' >Thread-15910254::DEBUG::2013-11-19 21:05:52,401::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' is free. Now locking as 'exclusive' (1 active user) >Thread-15910254::DEBUG::2013-11-19 21:05:52,402::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`501cbbc7-efde-463f-917f-61d451f94c7d`::Granted request >Thread-15910254::DEBUG::2013-11-19 21:05:52,410::task::495::TaskManager.Task::(_debug) Task fc43eb4b-607e-44ef-b0dc-a0fa60171dd9: _resourcesAcquired: Storage.5d7b5178-3790-472c-a13b-b481eb27a61e (exclusive) >Thread-15910254::DEBUG::2013-11-19 21:05:52,412::task::495::TaskManager.Task::(_debug) Task fc43eb4b-607e-44ef-b0dc-a0fa60171dd9: ref 1 aborting False >Thread-15910254::INFO::2013-11-19 21:05:52,413::sp::775::Storage.StoragePool::(deactivateSD) sdUUID=5d7b5178-3790-472c-a13b-b481eb27a61e spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 new_msdUUID=00000000-0000-0000-0000-000000000000 >Thread-15910254::INFO::2013-11-19 21:05:52,414::fileSD::176::Storage.StorageDomain::(validate) sdUUID=5d7b5178-3790-472c-a13b-b481eb27a61e >Thread-15910254::DEBUG::2013-11-19 21:05:52,415::persistentDict::204::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=[] >Thread-15910254::WARNING::2013-11-19 21:05:52,415::persistentDict::226::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-15910254::ERROR::2013-11-19 21:05:52,416::task::868::TaskManager.Task::(_setError) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 876, in _run > File "/usr/share/vdsm/storage/spm.py", line 115, in run > File "/usr/share/vdsm/storage/spm.py", line 1115, in public_deactivateStorageDomain > File "/usr/share/vdsm/storage/sp.py", line 798, in deactivateSD > File "/usr/share/vdsm/storage/sd.py", line 666, in isBackup > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910254::DEBUG::2013-11-19 21:05:52,420::task::495::TaskManager.Task::(_debug) Task fc43eb4b-607e-44ef-b0dc-a0fa60171dd9: Task._run: fc43eb4b-607e-44ef-b0dc-a0fa60171dd9 ('5d7b5178-3790-472c-a13b-b481eb27a61e', 'c4e5889a-348d-11e2-83fe-9f72369eab30', '00000000-0000-0000-0000-000000000000', 1) {} failed - stopping task >Thread-15910254::DEBUG::2013-11-19 21:05:52,421::task::495::TaskManager.Task::(_debug) Task fc43eb4b-607e-44ef-b0dc-a0fa60171dd9: stopping in state preparing (force False) >Thread-15910254::DEBUG::2013-11-19 21:05:52,422::task::495::TaskManager.Task::(_debug) Task fc43eb4b-607e-44ef-b0dc-a0fa60171dd9: ref 1 aborting True >Thread-15910254::INFO::2013-11-19 21:05:52,422::task::1171::TaskManager.Task::(prepare) aborting: Task is aborted: "'CLASS'" - code 100 >Thread-15910254::DEBUG::2013-11-19 21:05:52,423::task::495::TaskManager.Task::(_debug) Task fc43eb4b-607e-44ef-b0dc-a0fa60171dd9: Prepare: aborted: 'CLASS' >Thread-15910254::DEBUG::2013-11-19 21:05:52,423::task::495::TaskManager.Task::(_debug) Task fc43eb4b-607e-44ef-b0dc-a0fa60171dd9: ref 0 aborting True >Thread-15910254::DEBUG::2013-11-19 21:05:52,424::task::495::TaskManager.Task::(_debug) Task fc43eb4b-607e-44ef-b0dc-a0fa60171dd9: Task._doAbort: force False >Thread-15910254::DEBUG::2013-11-19 21:05:52,424::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910254::DEBUG::2013-11-19 21:05:52,425::task::495::TaskManager.Task::(_debug) Task fc43eb4b-607e-44ef-b0dc-a0fa60171dd9: moving from state preparing -> state aborting >Thread-15910254::DEBUG::2013-11-19 21:05:52,425::task::495::TaskManager.Task::(_debug) Task fc43eb4b-607e-44ef-b0dc-a0fa60171dd9: _aborting: recover policy none >Thread-15910254::DEBUG::2013-11-19 21:05:52,426::task::495::TaskManager.Task::(_debug) Task fc43eb4b-607e-44ef-b0dc-a0fa60171dd9: moving from state aborting -> state failed >Thread-15910254::DEBUG::2013-11-19 21:05:52,426::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>, 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e': < ResourceRef 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e', isValid: 'True' obj: 'None'>} >Thread-15910254::DEBUG::2013-11-19 21:05:52,430::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910254::DEBUG::2013-11-19 21:05:52,431::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910254::DEBUG::2013-11-19 21:05:52,432::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910254::DEBUG::2013-11-19 21:05:52,432::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910254::DEBUG::2013-11-19 21:05:52,433::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910254::DEBUG::2013-11-19 21:05:52,433::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' >Thread-15910254::DEBUG::2013-11-19 21:05:52,434::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' (0 active users) >Thread-15910254::DEBUG::2013-11-19 21:05:52,434::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' is free, finding out if anyone is waiting for it. >Thread-15910254::DEBUG::2013-11-19 21:05:52,435::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e', Clearing records. >Thread-15910254::ERROR::2013-11-19 21:05:52,436::dispatcher::106::Storage.Dispatcher.Protect::(run) 'CLASS' >Thread-15910254::ERROR::2013-11-19 21:05:52,436::dispatcher::107::Storage.Dispatcher.Protect::(run) Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 96, in run > File "/usr/share/vdsm/storage/task.py", line 1178, in prepare >KeyError: 'CLASS' > >Thread-1024767::ERROR::2013-11-19 21:05:52,441::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910255::DEBUG::2013-11-19 21:05:52,452::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910255::INFO::2013-11-19 21:05:52,453::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910255::DEBUG::2013-11-19 21:05:52,453::task::495::TaskManager.Task::(_debug) Task 123bfd63-731b-4596-b335-c5c1f254af6e: moving from state init -> state preparing >Thread-15910255::DEBUG::2013-11-19 21:05:52,454::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=274 spmId=2 >Thread-15910255::DEBUG::2013-11-19 21:05:52,455::task::495::TaskManager.Task::(_debug) Task 123bfd63-731b-4596-b335-c5c1f254af6e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910255::DEBUG::2013-11-19 21:05:52,455::task::495::TaskManager.Task::(_debug) Task 123bfd63-731b-4596-b335-c5c1f254af6e: moving from state preparing -> state finished >Thread-15910255::DEBUG::2013-11-19 21:05:52,456::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910255::DEBUG::2013-11-19 21:05:52,456::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910255::DEBUG::2013-11-19 21:05:52,457::task::495::TaskManager.Task::(_debug) Task 123bfd63-731b-4596-b335-c5c1f254af6e: ref 0 aborting False >Thread-15910255::INFO::2013-11-19 21:05:52,457::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 274}} >Thread-15910256::DEBUG::2013-11-19 21:05:52,468::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910256::INFO::2013-11-19 21:05:52,469::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksStatuses, args: () >Thread-15910256::DEBUG::2013-11-19 21:05:52,470::task::495::TaskManager.Task::(_debug) Task 327e0066-91e6-4354-a297-d7bcdcd10123: moving from state init -> state preparing >Thread-15910256::DEBUG::2013-11-19 21:05:52,470::taskManager::92::TaskManager::(getAllTasksStatuses) Entry. >Thread-15910256::DEBUG::2013-11-19 21:05:52,471::taskManager::97::TaskManager::(getAllTasksStatuses) Return: {} >Thread-15910256::DEBUG::2013-11-19 21:05:52,471::task::495::TaskManager.Task::(_debug) Task 327e0066-91e6-4354-a297-d7bcdcd10123: finished: {'allTasksStatus': {}} >Thread-15910256::DEBUG::2013-11-19 21:05:52,472::task::495::TaskManager.Task::(_debug) Task 327e0066-91e6-4354-a297-d7bcdcd10123: moving from state preparing -> state finished >Thread-15910256::DEBUG::2013-11-19 21:05:52,473::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910256::DEBUG::2013-11-19 21:05:52,473::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910256::DEBUG::2013-11-19 21:05:52,474::task::495::TaskManager.Task::(_debug) Task 327e0066-91e6-4354-a297-d7bcdcd10123: ref 0 aborting False >Thread-15910256::INFO::2013-11-19 21:05:52,474::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksStatuses, Return response: {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}} >Thread-15910257::DEBUG::2013-11-19 21:05:52,482::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910257::INFO::2013-11-19 21:05:52,487::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: spmStop, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910257::DEBUG::2013-11-19 21:05:52,488::task::495::TaskManager.Task::(_debug) Task 1cee9ac6-7302-4a86-abf3-67b0ce0fabdb: moving from state init -> state preparing >Thread-15910257::DEBUG::2013-11-19 21:05:52,488::spm::547::Storage.SPM::(public_spmStop) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 >Thread-15910257::DEBUG::2013-11-19 21:05:52,489::taskManager::105::TaskManager::(getAllTasks) Entry. >Thread-15910257::DEBUG::2013-11-19 21:05:52,489::taskManager::110::TaskManager::(getAllTasks) Return: {} >Thread-15910257::DEBUG::2013-11-19 21:05:52,490::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f225f88c-6d25-445e-81af-17a5da96a59a`::Request was made in '/usr/share/vdsm/storage/spm.py' line '573' at 'public_spmStop' >Thread-15910257::DEBUG::2013-11-19 21:05:52,491::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >Thread-15910257::DEBUG::2013-11-19 21:05:52,492::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >Thread-15910257::DEBUG::2013-11-19 21:05:52,492::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f225f88c-6d25-445e-81af-17a5da96a59a`::Granted request >Thread-15910257::DEBUG::2013-11-19 21:05:52,499::task::495::TaskManager.Task::(_debug) Task 1cee9ac6-7302-4a86-abf3-67b0ce0fabdb: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (exclusive) >Thread-15910257::DEBUG::2013-11-19 21:05:52,499::task::495::TaskManager.Task::(_debug) Task 1cee9ac6-7302-4a86-abf3-67b0ce0fabdb: ref 1 aborting False >Thread-15910257::DEBUG::2013-11-19 21:05:52,500::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`cd3e0a6d-6148-4742-bb1f-6c9ee5fdb6d6`::Request was made in '/usr/share/vdsm/storage/spm.py' line '584' at '_stop' >Thread-15910257::DEBUG::2013-11-19 21:05:52,501::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >Thread-15910257::DEBUG::2013-11-19 21:05:52,501::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >Thread-15910257::DEBUG::2013-11-19 21:05:52,502::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`cd3e0a6d-6148-4742-bb1f-6c9ee5fdb6d6`::Granted request >Thread-15910257::DEBUG::2013-11-19 21:05:52,505::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`4243a79a-03e3-41d0-8176-fa99c9e084e8`::Request was made in '/usr/share/vdsm/storage/spm.py' line '601' at '_stop' >Thread-15910257::DEBUG::2013-11-19 21:05:52,506::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' for lock type 'exclusive' >Thread-15910257::DEBUG::2013-11-19 21:05:52,506::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' is free. Now locking as 'exclusive' (1 active user) >Thread-15910257::DEBUG::2013-11-19 21:05:52,513::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`4243a79a-03e3-41d0-8176-fa99c9e084e8`::Granted request >Thread-15910257::DEBUG::2013-11-19 21:05:52,520::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' >Thread-15910257::DEBUG::2013-11-19 21:05:52,523::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' (0 active users) >Thread-15910257::DEBUG::2013-11-19 21:05:52,523::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' is free, finding out if anyone is waiting for it. >Thread-15910257::DEBUG::2013-11-19 21:05:52,524::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e', Clearing records. >Thread-15910257::DEBUG::2013-11-19 21:05:52,525::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910257::DEBUG::2013-11-19 21:05:52,525::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910257::DEBUG::2013-11-19 21:05:52,526::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910257::DEBUG::2013-11-19 21:05:52,526::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910257::DEBUG::2013-11-19 21:05:52,527::taskManager::118::TaskManager::(unloadTasks) Entry. >Thread-15910257::DEBUG::2013-11-19 21:05:52,527::taskManager::122::TaskManager::(unloadTasks) Return >Thread-15910257::DEBUG::2013-11-19 21:05:52,529::spm::195::Storage.SPM::(__cleanupMasterMount) unmounting /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master >Thread-15910257::DEBUG::2013-11-19 21:05:52,655::fileUtils::112::Storage.Misc.excCmd::(umount) '/usr/bin/sudo -n /bin/umount -f /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15910257::DEBUG::2013-11-19 21:05:53,057::fileUtils::112::Storage.Misc.excCmd::(umount) SUCCESS: <err> = ''; <rc> = 0 >Thread-15910257::DEBUG::2013-11-19 21:05:53,060::spm::290::Storage.SPM::(__cleanupSPM) cleaning up SPM: c4e5889a-348d-11e2-83fe-9f72369eab30 >Thread-15910257::DEBUG::2013-11-19 21:05:53,061::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >Thread-15910257::DEBUG::2013-11-19 21:05:53,062::persistentDict::146::Storage.PersistentDict::(transaction) Flushing changes >Thread-15910257::DEBUG::2013-11-19 21:05:53,063::persistentDict::265::Storage.PersistentDict::(flush) about to write lines (VGTagMetadataRW)=[u'CLASS=Data', u'DESCRIPTION=OS', u'IOOPTIMEOUTSEC=10', u'LEASERETRIES=3', u'LEASETIMESEC=60', u'LOCKPOLICY=', u'LOCKRENEWALINTERVALSEC=5', u'LOGBLKSIZE=512', u'MASTER_VERSION=1', u'PHYBLKSIZE=512', u'POOL_DESCRIPTION=Default', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_ID=-1', u'POOL_SPM_LVER=274', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'ROLE=Master', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'TYPE=FCP', u'VERSION=2', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', '_SHA_CKSUM=a631aa7004ad6a1f7b79a293dfe44576de53980f'] >Thread-15910257::DEBUG::2013-11-19 21:05:53,075::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15910257::DEBUG::2013-11-19 21:05:53,076::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15910257::DEBUG::2013-11-19 21:05:53,077::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15910257::DEBUG::2013-11-19 21:05:53,078::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15910257::DEBUG::2013-11-19 21:05:53,079::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15910257::DEBUG::2013-11-19 21:05:53,086::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15910258::INFO::2013-11-19 21:05:53,482::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910258::DEBUG::2013-11-19 21:05:53,483::task::495::TaskManager.Task::(_debug) Task ccfafc69-876c-4daf-abf7-72b838fe597f: moving from state init -> state preparing >Thread-15910258::DEBUG::2013-11-19 21:05:53,484::task::495::TaskManager.Task::(_debug) Task ccfafc69-876c-4daf-abf7-72b838fe597f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000458955764771', 'lastCheck': 1384895151.474242, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147604942322', 'lastCheck': 1384895147.3423131, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895152.442081, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000319004058838', 'lastCheck': 1384895150.340637, 'valid': True, 'code': 0}} >Thread-15910258::DEBUG::2013-11-19 21:05:53,490::task::495::TaskManager.Task::(_debug) Task ccfafc69-876c-4daf-abf7-72b838fe597f: moving from state preparing -> state finished >Thread-15910258::DEBUG::2013-11-19 21:05:53,491::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910258::DEBUG::2013-11-19 21:05:53,495::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910258::DEBUG::2013-11-19 21:05:53,496::task::495::TaskManager.Task::(_debug) Task ccfafc69-876c-4daf-abf7-72b838fe597f: ref 0 aborting False >Thread-15910258::INFO::2013-11-19 21:05:53,500::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000458955764771', 'lastCheck': 1384895151.474242, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147604942322', 'lastCheck': 1384895147.3423131, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895152.442081, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000319004058838', 'lastCheck': 1384895150.340637, 'valid': True, 'code': 0}} >Thread-15910257::DEBUG::2013-11-19 21:05:53,764::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15909965::INFO::2013-11-19 21:05:53,766::storage_mailbox::673::Storage.MailBox.SpmMailMonitor::(run) SPM_MailMonitor - Incoming mail monitoring thread stopped >Thread-15910257::DEBUG::2013-11-19 21:05:53,790::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15910257::DEBUG::2013-11-19 21:05:53,792::blockSD::145::storage.Metadata.VGTagMetadataRW::(writelines) Updating metadata adding=MDT_POOL_SPM_ID=-1, MDT__SHA_CKSUM=a631aa7004ad6a1f7b79a293dfe44576de53980f removing=MDT__SHA_CKSUM=e643d303df7878b6d71c193b5f1f2469245114b5, MDT_POOL_SPM_ID=2 >Thread-15910257::DEBUG::2013-11-19 21:05:53,793::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --deltag MDT_POOL_SPM_ID=2 --deltag MDT__SHA_CKSUM=e643d303df7878b6d71c193b5f1f2469245114b5 --addtag MDT_POOL_SPM_ID=-1 --addtag MDT__SHA_CKSUM=a631aa7004ad6a1f7b79a293dfe44576de53980f 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15910257::DEBUG::2013-11-19 21:05:54,449::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15910257::DEBUG::2013-11-19 21:05:54,451::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15910257::DEBUG::2013-11-19 21:05:54,452::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15910257::DEBUG::2013-11-19 21:05:54,453::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >Thread-15910257::INFO::2013-11-19 21:05:54,454::safelease::83::ClusterLock::(release) Releasing cluster lock for domain 6b892067-ac7e-4039-a7be-5b7387525b16 >Thread-15910257::DEBUG::2013-11-19 21:05:54,459::safelease::84::Storage.Misc.excCmd::(release) '/usr/libexec/vdsm/spmstop.sh 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd /usr/libexec/vdsm/) >Thread-15910257::DEBUG::2013-11-19 21:05:55,946::safelease::84::Storage.Misc.excCmd::(release) SUCCESS: <err> = ''; <rc> = 0 >Thread-15910257::DEBUG::2013-11-19 21:05:55,947::safelease::87::ClusterLock::(release) Cluster lock released successfully >Thread-15910257::DEBUG::2013-11-19 21:05:55,948::task::495::TaskManager.Task::(_debug) Task 1cee9ac6-7302-4a86-abf3-67b0ce0fabdb: finished: None >Thread-15910257::DEBUG::2013-11-19 21:05:55,949::task::495::TaskManager.Task::(_debug) Task 1cee9ac6-7302-4a86-abf3-67b0ce0fabdb: moving from state preparing -> state finished >Thread-15910257::DEBUG::2013-11-19 21:05:55,949::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910257::DEBUG::2013-11-19 21:05:55,950::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910257::DEBUG::2013-11-19 21:05:55,951::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910257::DEBUG::2013-11-19 21:05:55,952::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910257::DEBUG::2013-11-19 21:05:55,952::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910257::DEBUG::2013-11-19 21:05:55,953::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910257::DEBUG::2013-11-19 21:05:55,953::task::495::TaskManager.Task::(_debug) Task 1cee9ac6-7302-4a86-abf3-67b0ce0fabdb: ref 0 aborting False >Thread-15910257::INFO::2013-11-19 21:05:55,954::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: spmStop, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-42838::DEBUG::2013-11-19 21:06:01,681::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-42838::DEBUG::2013-11-19 21:06:01,683::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 21:06:02,295::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-42838::DEBUG::2013-11-19 21:06:02,299::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1024767::ERROR::2013-11-19 21:06:02,454::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:06:12,468::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:06:22,482::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:06:32,496::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:06:42,510::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:06:52,524::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 21:07:00,523::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 21:07:01,247::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:07:02,751::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:07:12,765::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:07:22,779::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:07:32,795::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:07:42,809::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:07:52,822::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:08:02,837::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:08:12,851::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:08:22,864::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:08:32,888::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910264::INFO::2013-11-19 21:08:42,470::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910264::DEBUG::2013-11-19 21:08:42,473::task::495::TaskManager.Task::(_debug) Task ab4b5d65-7852-4b9d-abc1-59466c2d300e: moving from state init -> state preparing >Thread-15910264::DEBUG::2013-11-19 21:08:42,475::task::495::TaskManager.Task::(_debug) Task ab4b5d65-7852-4b9d-abc1-59466c2d300e: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000408172607422', 'lastCheck': 1384895312.5005269, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00215601921082', 'lastCheck': 1384895317.9003539, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895312.889158, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00042986869812', 'lastCheck': 1384895321.363343, 'valid': True, 'code': 0}} >Thread-15910264::DEBUG::2013-11-19 21:08:42,476::task::495::TaskManager.Task::(_debug) Task ab4b5d65-7852-4b9d-abc1-59466c2d300e: moving from state preparing -> state finished >Thread-15910264::DEBUG::2013-11-19 21:08:42,478::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910264::DEBUG::2013-11-19 21:08:42,479::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910264::DEBUG::2013-11-19 21:08:42,479::task::495::TaskManager.Task::(_debug) Task ab4b5d65-7852-4b9d-abc1-59466c2d300e: ref 0 aborting False >Thread-15910264::INFO::2013-11-19 21:08:42,480::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000408172607422', 'lastCheck': 1384895312.5005269, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00215601921082', 'lastCheck': 1384895317.9003539, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895312.889158, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00042986869812', 'lastCheck': 1384895321.363343, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:08:42,902::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910270::INFO::2013-11-19 21:08:52,612::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910270::DEBUG::2013-11-19 21:08:52,613::task::495::TaskManager.Task::(_debug) Task b91d27b9-3fdc-4d79-ac58-6040fb8ed9fd: moving from state init -> state preparing >Thread-15910270::DEBUG::2013-11-19 21:08:52,614::task::495::TaskManager.Task::(_debug) Task b91d27b9-3fdc-4d79-ac58-6040fb8ed9fd: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000324964523315', 'lastCheck': 1384895332.5263181, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00136399269104', 'lastCheck': 1384895327.915976, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895322.903131, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000418186187744', 'lastCheck': 1384895331.3750341, 'valid': True, 'code': 0}} >Thread-15910270::DEBUG::2013-11-19 21:08:52,615::task::495::TaskManager.Task::(_debug) Task b91d27b9-3fdc-4d79-ac58-6040fb8ed9fd: moving from state preparing -> state finished >Thread-15910270::DEBUG::2013-11-19 21:08:52,615::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910270::DEBUG::2013-11-19 21:08:52,616::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910270::DEBUG::2013-11-19 21:08:52,616::task::495::TaskManager.Task::(_debug) Task b91d27b9-3fdc-4d79-ac58-6040fb8ed9fd: ref 0 aborting False >Thread-15910270::INFO::2013-11-19 21:08:52,617::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000324964523315', 'lastCheck': 1384895332.5263181, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00136399269104', 'lastCheck': 1384895327.915976, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895322.903131, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000418186187744', 'lastCheck': 1384895331.3750341, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:08:52,916::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910276::INFO::2013-11-19 21:09:02,734::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910276::DEBUG::2013-11-19 21:09:02,735::task::495::TaskManager.Task::(_debug) Task b0e26d18-3490-43e4-b7c6-e9b3c7452586: moving from state init -> state preparing >Thread-15910276::DEBUG::2013-11-19 21:09:02,735::task::495::TaskManager.Task::(_debug) Task b0e26d18-3490-43e4-b7c6-e9b3c7452586: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000302076339722', 'lastCheck': 1384895342.5392871, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00125598907471', 'lastCheck': 1384895337.9319, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895332.9171131, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000450134277344', 'lastCheck': 1384895341.3871281, 'valid': True, 'code': 0}} >Thread-15910276::DEBUG::2013-11-19 21:09:02,736::task::495::TaskManager.Task::(_debug) Task b0e26d18-3490-43e4-b7c6-e9b3c7452586: moving from state preparing -> state finished >Thread-15910276::DEBUG::2013-11-19 21:09:02,737::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910276::DEBUG::2013-11-19 21:09:02,737::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910276::DEBUG::2013-11-19 21:09:02,738::task::495::TaskManager.Task::(_debug) Task b0e26d18-3490-43e4-b7c6-e9b3c7452586: ref 0 aborting False >Thread-15910276::INFO::2013-11-19 21:09:02,739::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000302076339722', 'lastCheck': 1384895342.5392871, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00125598907471', 'lastCheck': 1384895337.9319, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895332.9171131, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000450134277344', 'lastCheck': 1384895341.3871281, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:09:02,930::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910282::INFO::2013-11-19 21:09:12,899::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910282::DEBUG::2013-11-19 21:09:12,900::task::495::TaskManager.Task::(_debug) Task 7326c44d-23b2-4aad-be1b-4aa9265459a7: moving from state init -> state preparing >Thread-15910282::DEBUG::2013-11-19 21:09:12,900::task::495::TaskManager.Task::(_debug) Task 7326c44d-23b2-4aad-be1b-4aa9265459a7: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000398874282837', 'lastCheck': 1384895352.5523, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137686729431', 'lastCheck': 1384895347.9473341, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895342.931057, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000427007675171', 'lastCheck': 1384895351.398876, 'valid': True, 'code': 0}} >Thread-15910282::DEBUG::2013-11-19 21:09:12,901::task::495::TaskManager.Task::(_debug) Task 7326c44d-23b2-4aad-be1b-4aa9265459a7: moving from state preparing -> state finished >Thread-15910282::DEBUG::2013-11-19 21:09:12,902::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910282::DEBUG::2013-11-19 21:09:12,902::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910282::DEBUG::2013-11-19 21:09:12,903::task::495::TaskManager.Task::(_debug) Task 7326c44d-23b2-4aad-be1b-4aa9265459a7: ref 0 aborting False >Thread-15910282::INFO::2013-11-19 21:09:12,903::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000398874282837', 'lastCheck': 1384895352.5523, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137686729431', 'lastCheck': 1384895347.9473341, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895342.931057, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000427007675171', 'lastCheck': 1384895351.398876, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:09:12,944::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 21:09:22,563::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:09:22,960::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910288::INFO::2013-11-19 21:09:23,055::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910288::DEBUG::2013-11-19 21:09:23,055::task::495::TaskManager.Task::(_debug) Task deebe857-c003-45ea-a57f-4ef99642b367: moving from state init -> state preparing >Thread-15910288::DEBUG::2013-11-19 21:09:23,056::task::495::TaskManager.Task::(_debug) Task deebe857-c003-45ea-a57f-4ef99642b367: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000398874282837', 'lastCheck': 1384895352.5523, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00136208534241', 'lastCheck': 1384895357.962534, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895362.9622829, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000322103500366', 'lastCheck': 1384895361.4105051, 'valid': True, 'code': 0}} >Thread-15910288::DEBUG::2013-11-19 21:09:23,057::task::495::TaskManager.Task::(_debug) Task deebe857-c003-45ea-a57f-4ef99642b367: moving from state preparing -> state finished >Thread-15910288::DEBUG::2013-11-19 21:09:23,058::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910288::DEBUG::2013-11-19 21:09:23,058::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910288::DEBUG::2013-11-19 21:09:23,059::task::495::TaskManager.Task::(_debug) Task deebe857-c003-45ea-a57f-4ef99642b367: ref 0 aborting False >Thread-15910288::INFO::2013-11-19 21:09:23,059::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000398874282837', 'lastCheck': 1384895352.5523, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00136208534241', 'lastCheck': 1384895357.962534, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895362.9622829, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000322103500366', 'lastCheck': 1384895361.4105051, 'valid': True, 'code': 0}} >Thread-42838::DEBUG::2013-11-19 21:09:23,198::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:09:32,975::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910294::INFO::2013-11-19 21:09:33,189::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910294::DEBUG::2013-11-19 21:09:33,190::task::495::TaskManager.Task::(_debug) Task d69fbd7c-1698-4d00-9207-13609135a8e3: moving from state init -> state preparing >Thread-15910294::DEBUG::2013-11-19 21:09:33,191::task::495::TaskManager.Task::(_debug) Task d69fbd7c-1698-4d00-9207-13609135a8e3: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000361919403076', 'lastCheck': 1384895363.2013409, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00292801856995', 'lastCheck': 1384895367.984206, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895372.9765589, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000463962554932', 'lastCheck': 1384895371.4224789, 'valid': True, 'code': 0}} >Thread-15910294::DEBUG::2013-11-19 21:09:33,192::task::495::TaskManager.Task::(_debug) Task d69fbd7c-1698-4d00-9207-13609135a8e3: moving from state preparing -> state finished >Thread-15910294::DEBUG::2013-11-19 21:09:33,192::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910294::DEBUG::2013-11-19 21:09:33,193::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910294::DEBUG::2013-11-19 21:09:33,193::task::495::TaskManager.Task::(_debug) Task d69fbd7c-1698-4d00-9207-13609135a8e3: ref 0 aborting False >Thread-15910294::INFO::2013-11-19 21:09:33,194::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000361919403076', 'lastCheck': 1384895363.2013409, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00292801856995', 'lastCheck': 1384895367.984206, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895372.9765589, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000463962554932', 'lastCheck': 1384895371.4224789, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:09:42,992::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910300::INFO::2013-11-19 21:09:43,363::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910300::DEBUG::2013-11-19 21:09:43,364::task::495::TaskManager.Task::(_debug) Task baafe264-10fa-46ea-afa3-fffe3bf1a188: moving from state init -> state preparing >Thread-15910300::DEBUG::2013-11-19 21:09:43,365::task::495::TaskManager.Task::(_debug) Task baafe264-10fa-46ea-afa3-fffe3bf1a188: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000962018966675', 'lastCheck': 1384895383.226661, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00221610069275', 'lastCheck': 1384895378.001792, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895382.9941361, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000440835952759', 'lastCheck': 1384895381.434324, 'valid': True, 'code': 0}} >Thread-15910300::DEBUG::2013-11-19 21:09:43,365::task::495::TaskManager.Task::(_debug) Task baafe264-10fa-46ea-afa3-fffe3bf1a188: moving from state preparing -> state finished >Thread-15910300::DEBUG::2013-11-19 21:09:43,366::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910300::DEBUG::2013-11-19 21:09:43,367::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910300::DEBUG::2013-11-19 21:09:43,367::task::495::TaskManager.Task::(_debug) Task baafe264-10fa-46ea-afa3-fffe3bf1a188: ref 0 aborting False >Thread-15910300::INFO::2013-11-19 21:09:43,368::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000962018966675', 'lastCheck': 1384895383.226661, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00221610069275', 'lastCheck': 1384895378.001792, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895382.9941361, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000440835952759', 'lastCheck': 1384895381.434324, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:09:53,005::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910306::INFO::2013-11-19 21:09:53,543::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910306::DEBUG::2013-11-19 21:09:53,544::task::495::TaskManager.Task::(_debug) Task 231dcf0a-1bec-4674-9ac3-e1465f549f8b: moving from state init -> state preparing >Thread-15910306::DEBUG::2013-11-19 21:09:53,544::task::495::TaskManager.Task::(_debug) Task 231dcf0a-1bec-4674-9ac3-e1465f549f8b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000321865081787', 'lastCheck': 1384895393.2395229, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00140905380249', 'lastCheck': 1384895388.0171759, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895393.006654, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000323057174683', 'lastCheck': 1384895391.445971, 'valid': True, 'code': 0}} >Thread-15910306::DEBUG::2013-11-19 21:09:53,545::task::495::TaskManager.Task::(_debug) Task 231dcf0a-1bec-4674-9ac3-e1465f549f8b: moving from state preparing -> state finished >Thread-15910306::DEBUG::2013-11-19 21:09:53,546::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910306::DEBUG::2013-11-19 21:09:53,546::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910306::DEBUG::2013-11-19 21:09:53,547::task::495::TaskManager.Task::(_debug) Task 231dcf0a-1bec-4674-9ac3-e1465f549f8b: ref 0 aborting False >Thread-15910306::INFO::2013-11-19 21:09:53,547::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000321865081787', 'lastCheck': 1384895393.2395229, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00140905380249', 'lastCheck': 1384895388.0171759, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895393.006654, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000323057174683', 'lastCheck': 1384895391.445971, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:10:03,020::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910312::INFO::2013-11-19 21:10:03,723::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910312::DEBUG::2013-11-19 21:10:03,724::task::495::TaskManager.Task::(_debug) Task e9f40114-a5f4-47b2-8fce-e78b781acc9f: moving from state init -> state preparing >Thread-15910312::DEBUG::2013-11-19 21:10:03,733::task::495::TaskManager.Task::(_debug) Task e9f40114-a5f4-47b2-8fce-e78b781acc9f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000370979309082', 'lastCheck': 1384895403.2527349, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00118780136108', 'lastCheck': 1384895398.03247, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895403.0211179, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000417947769165', 'lastCheck': 1384895401.4578071, 'valid': True, 'code': 0}} >Thread-15910312::DEBUG::2013-11-19 21:10:03,734::task::495::TaskManager.Task::(_debug) Task e9f40114-a5f4-47b2-8fce-e78b781acc9f: moving from state preparing -> state finished >Thread-15910312::DEBUG::2013-11-19 21:10:03,735::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910312::DEBUG::2013-11-19 21:10:03,735::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910312::DEBUG::2013-11-19 21:10:03,736::task::495::TaskManager.Task::(_debug) Task e9f40114-a5f4-47b2-8fce-e78b781acc9f: ref 0 aborting False >Thread-15910312::INFO::2013-11-19 21:10:03,736::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000370979309082', 'lastCheck': 1384895403.2527349, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00118780136108', 'lastCheck': 1384895398.03247, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895403.0211179, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000417947769165', 'lastCheck': 1384895401.4578071, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:10:13,034::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910318::INFO::2013-11-19 21:10:13,871::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910318::DEBUG::2013-11-19 21:10:13,872::task::495::TaskManager.Task::(_debug) Task e4e303f5-5190-429d-bcb7-b1a1b05b0e01: moving from state init -> state preparing >Thread-15910318::DEBUG::2013-11-19 21:10:13,881::task::495::TaskManager.Task::(_debug) Task e4e303f5-5190-429d-bcb7-b1a1b05b0e01: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000344038009644', 'lastCheck': 1384895413.2657571, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00155305862427', 'lastCheck': 1384895408.0483191, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895413.034914, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000426054000854', 'lastCheck': 1384895411.4693689, 'valid': True, 'code': 0}} >Thread-15910318::DEBUG::2013-11-19 21:10:13,882::task::495::TaskManager.Task::(_debug) Task e4e303f5-5190-429d-bcb7-b1a1b05b0e01: moving from state preparing -> state finished >Thread-15910318::DEBUG::2013-11-19 21:10:13,882::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910318::DEBUG::2013-11-19 21:10:13,883::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910318::DEBUG::2013-11-19 21:10:13,884::task::495::TaskManager.Task::(_debug) Task e4e303f5-5190-429d-bcb7-b1a1b05b0e01: ref 0 aborting False >Thread-15910318::INFO::2013-11-19 21:10:13,884::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000344038009644', 'lastCheck': 1384895413.2657571, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00155305862427', 'lastCheck': 1384895408.0483191, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895413.034914, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000426054000854', 'lastCheck': 1384895411.4693689, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:10:23,047::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910324::INFO::2013-11-19 21:10:23,999::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910324::DEBUG::2013-11-19 21:10:24,000::task::495::TaskManager.Task::(_debug) Task b209c23f-e4e8-4ffb-a380-09863d825916: moving from state init -> state preparing >Thread-15910324::DEBUG::2013-11-19 21:10:24,001::task::495::TaskManager.Task::(_debug) Task b209c23f-e4e8-4ffb-a380-09863d825916: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000339984893799', 'lastCheck': 1384895423.2786729, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00136399269104', 'lastCheck': 1384895418.063956, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895423.048322, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000330924987793', 'lastCheck': 1384895421.4808969, 'valid': True, 'code': 0}} >Thread-15910324::DEBUG::2013-11-19 21:10:24,001::task::495::TaskManager.Task::(_debug) Task b209c23f-e4e8-4ffb-a380-09863d825916: moving from state preparing -> state finished >Thread-15910324::DEBUG::2013-11-19 21:10:24,002::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910324::DEBUG::2013-11-19 21:10:24,002::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910324::DEBUG::2013-11-19 21:10:24,003::task::495::TaskManager.Task::(_debug) Task b209c23f-e4e8-4ffb-a380-09863d825916: ref 0 aborting False >Thread-15910324::INFO::2013-11-19 21:10:24,004::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000339984893799', 'lastCheck': 1384895423.2786729, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00136399269104', 'lastCheck': 1384895418.063956, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895423.048322, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000330924987793', 'lastCheck': 1384895421.4808969, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:10:33,061::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910330::INFO::2013-11-19 21:10:34,166::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910330::DEBUG::2013-11-19 21:10:34,167::task::495::TaskManager.Task::(_debug) Task bec9db4b-e505-43ac-88da-3ddee52b18f3: moving from state init -> state preparing >Thread-15910330::DEBUG::2013-11-19 21:10:34,167::task::495::TaskManager.Task::(_debug) Task bec9db4b-e505-43ac-88da-3ddee52b18f3: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000358104705811', 'lastCheck': 1384895433.2918279, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137400627136', 'lastCheck': 1384895428.079247, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895433.062007, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000414848327637', 'lastCheck': 1384895431.4925799, 'valid': True, 'code': 0}} >Thread-15910330::DEBUG::2013-11-19 21:10:34,168::task::495::TaskManager.Task::(_debug) Task bec9db4b-e505-43ac-88da-3ddee52b18f3: moving from state preparing -> state finished >Thread-15910330::DEBUG::2013-11-19 21:10:34,169::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910330::DEBUG::2013-11-19 21:10:34,169::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910330::DEBUG::2013-11-19 21:10:34,170::task::495::TaskManager.Task::(_debug) Task bec9db4b-e505-43ac-88da-3ddee52b18f3: ref 0 aborting False >Thread-15910330::INFO::2013-11-19 21:10:34,170::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000358104705811', 'lastCheck': 1384895433.2918279, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137400627136', 'lastCheck': 1384895428.079247, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895433.062007, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000414848327637', 'lastCheck': 1384895431.4925799, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:10:43,075::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910336::INFO::2013-11-19 21:10:44,305::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910336::DEBUG::2013-11-19 21:10:44,306::task::495::TaskManager.Task::(_debug) Task 32eae202-3c9d-46d7-aa60-484f22cf6401: moving from state init -> state preparing >Thread-15910336::DEBUG::2013-11-19 21:10:44,307::task::495::TaskManager.Task::(_debug) Task 32eae202-3c9d-46d7-aa60-484f22cf6401: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000369071960449', 'lastCheck': 1384895443.3044679, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00225591659546', 'lastCheck': 1384895438.0960181, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895443.075911, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000434875488281', 'lastCheck': 1384895441.5043221, 'valid': True, 'code': 0}} >Thread-15910336::DEBUG::2013-11-19 21:10:44,308::task::495::TaskManager.Task::(_debug) Task 32eae202-3c9d-46d7-aa60-484f22cf6401: moving from state preparing -> state finished >Thread-15910336::DEBUG::2013-11-19 21:10:44,308::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910336::DEBUG::2013-11-19 21:10:44,309::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910336::DEBUG::2013-11-19 21:10:44,309::task::495::TaskManager.Task::(_debug) Task 32eae202-3c9d-46d7-aa60-484f22cf6401: ref 0 aborting False >Thread-15910336::INFO::2013-11-19 21:10:44,310::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000369071960449', 'lastCheck': 1384895443.3044679, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00225591659546', 'lastCheck': 1384895438.0960181, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895443.075911, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000434875488281', 'lastCheck': 1384895441.5043221, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:10:53,088::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910342::INFO::2013-11-19 21:10:54,432::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910342::DEBUG::2013-11-19 21:10:54,432::task::495::TaskManager.Task::(_debug) Task d0a3799a-6bab-451e-9b83-2442d2ac378f: moving from state init -> state preparing >Thread-15910342::DEBUG::2013-11-19 21:10:54,433::task::495::TaskManager.Task::(_debug) Task d0a3799a-6bab-451e-9b83-2442d2ac378f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000319957733154', 'lastCheck': 1384895453.3178811, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00140619277954', 'lastCheck': 1384895448.1113989, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895453.0896139, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000417947769165', 'lastCheck': 1384895451.5159969, 'valid': True, 'code': 0}} >Thread-15910342::DEBUG::2013-11-19 21:10:54,434::task::495::TaskManager.Task::(_debug) Task d0a3799a-6bab-451e-9b83-2442d2ac378f: moving from state preparing -> state finished >Thread-15910342::DEBUG::2013-11-19 21:10:54,434::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910342::DEBUG::2013-11-19 21:10:54,435::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910342::DEBUG::2013-11-19 21:10:54,435::task::495::TaskManager.Task::(_debug) Task d0a3799a-6bab-451e-9b83-2442d2ac378f: ref 0 aborting False >Thread-15910342::INFO::2013-11-19 21:10:54,436::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000319957733154', 'lastCheck': 1384895453.3178811, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00140619277954', 'lastCheck': 1384895448.1113989, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895453.0896139, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000417947769165', 'lastCheck': 1384895451.5159969, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:11:03,102::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910348::INFO::2013-11-19 21:11:04,669::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910348::DEBUG::2013-11-19 21:11:04,669::task::495::TaskManager.Task::(_debug) Task a5ea5607-815a-473a-92bd-cd00e35ee5ec: moving from state init -> state preparing >Thread-15910348::DEBUG::2013-11-19 21:11:04,670::task::495::TaskManager.Task::(_debug) Task a5ea5607-815a-473a-92bd-cd00e35ee5ec: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000321865081787', 'lastCheck': 1384895463.3308849, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00136303901672', 'lastCheck': 1384895458.126931, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895463.1031461, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000327110290527', 'lastCheck': 1384895461.5276811, 'valid': True, 'code': 0}} >Thread-15910348::DEBUG::2013-11-19 21:11:04,671::task::495::TaskManager.Task::(_debug) Task a5ea5607-815a-473a-92bd-cd00e35ee5ec: moving from state preparing -> state finished >Thread-15910348::DEBUG::2013-11-19 21:11:04,671::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910348::DEBUG::2013-11-19 21:11:04,672::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910348::DEBUG::2013-11-19 21:11:04,672::task::495::TaskManager.Task::(_debug) Task a5ea5607-815a-473a-92bd-cd00e35ee5ec: ref 0 aborting False >Thread-15910348::INFO::2013-11-19 21:11:04,673::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000321865081787', 'lastCheck': 1384895463.3308849, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00136303901672', 'lastCheck': 1384895458.126931, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895463.1031461, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000327110290527', 'lastCheck': 1384895461.5276811, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:11:13,115::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910354::INFO::2013-11-19 21:11:14,825::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910354::DEBUG::2013-11-19 21:11:14,825::task::495::TaskManager.Task::(_debug) Task f48a06fe-2197-4a5f-9282-40347a0a5b94: moving from state init -> state preparing >Thread-15910354::DEBUG::2013-11-19 21:11:14,826::task::495::TaskManager.Task::(_debug) Task f48a06fe-2197-4a5f-9282-40347a0a5b94: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000309944152832', 'lastCheck': 1384895473.343817, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147008895874', 'lastCheck': 1384895468.1423869, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895473.1166539, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000341176986694', 'lastCheck': 1384895471.5393701, 'valid': True, 'code': 0}} >Thread-15910354::DEBUG::2013-11-19 21:11:14,827::task::495::TaskManager.Task::(_debug) Task f48a06fe-2197-4a5f-9282-40347a0a5b94: moving from state preparing -> state finished >Thread-15910354::DEBUG::2013-11-19 21:11:14,827::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910354::DEBUG::2013-11-19 21:11:14,828::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910354::DEBUG::2013-11-19 21:11:14,828::task::495::TaskManager.Task::(_debug) Task f48a06fe-2197-4a5f-9282-40347a0a5b94: ref 0 aborting False >Thread-15910354::INFO::2013-11-19 21:11:14,836::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000309944152832', 'lastCheck': 1384895473.343817, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147008895874', 'lastCheck': 1384895468.1423869, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895473.1166539, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000341176986694', 'lastCheck': 1384895471.5393701, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:11:23,128::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910360::INFO::2013-11-19 21:11:25,071::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910360::DEBUG::2013-11-19 21:11:25,072::task::495::TaskManager.Task::(_debug) Task c3c39101-5f46-4ae8-b9be-71242af0bbbf: moving from state init -> state preparing >Thread-15910360::DEBUG::2013-11-19 21:11:25,072::task::495::TaskManager.Task::(_debug) Task c3c39101-5f46-4ae8-b9be-71242af0bbbf: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000325918197632', 'lastCheck': 1384895483.3567221, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137090682983', 'lastCheck': 1384895478.158097, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895483.1291201, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000318050384521', 'lastCheck': 1384895481.5509479, 'valid': True, 'code': 0}} >Thread-15910360::DEBUG::2013-11-19 21:11:25,073::task::495::TaskManager.Task::(_debug) Task c3c39101-5f46-4ae8-b9be-71242af0bbbf: moving from state preparing -> state finished >Thread-15910360::DEBUG::2013-11-19 21:11:25,074::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910360::DEBUG::2013-11-19 21:11:25,074::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910360::DEBUG::2013-11-19 21:11:25,075::task::495::TaskManager.Task::(_debug) Task c3c39101-5f46-4ae8-b9be-71242af0bbbf: ref 0 aborting False >Thread-15910360::INFO::2013-11-19 21:11:25,075::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000325918197632', 'lastCheck': 1384895483.3567221, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137090682983', 'lastCheck': 1384895478.158097, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895483.1291201, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000318050384521', 'lastCheck': 1384895481.5509479, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:11:33,142::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910366::INFO::2013-11-19 21:11:35,189::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910366::DEBUG::2013-11-19 21:11:35,190::task::495::TaskManager.Task::(_debug) Task eb2365a4-a243-40a7-a2dc-ed2e411ecd5b: moving from state init -> state preparing >Thread-15910366::DEBUG::2013-11-19 21:11:35,191::task::495::TaskManager.Task::(_debug) Task eb2365a4-a243-40a7-a2dc-ed2e411ecd5b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000344038009644', 'lastCheck': 1384895493.3696339, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00138211250305', 'lastCheck': 1384895488.17326, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895493.142915, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000350952148438', 'lastCheck': 1384895491.562614, 'valid': True, 'code': 0}} >Thread-15910366::DEBUG::2013-11-19 21:11:35,191::task::495::TaskManager.Task::(_debug) Task eb2365a4-a243-40a7-a2dc-ed2e411ecd5b: moving from state preparing -> state finished >Thread-15910366::DEBUG::2013-11-19 21:11:35,192::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910366::DEBUG::2013-11-19 21:11:35,193::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910366::DEBUG::2013-11-19 21:11:35,193::task::495::TaskManager.Task::(_debug) Task eb2365a4-a243-40a7-a2dc-ed2e411ecd5b: ref 0 aborting False >Thread-15910366::INFO::2013-11-19 21:11:35,194::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000344038009644', 'lastCheck': 1384895493.3696339, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00138211250305', 'lastCheck': 1384895488.17326, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895493.142915, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000350952148438', 'lastCheck': 1384895491.562614, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:11:43,155::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910372::INFO::2013-11-19 21:11:45,334::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910372::DEBUG::2013-11-19 21:11:45,334::task::495::TaskManager.Task::(_debug) Task ef3f0ed1-f27b-4979-8da1-f5be3b56ddda: moving from state init -> state preparing >Thread-15910372::DEBUG::2013-11-19 21:11:45,335::task::495::TaskManager.Task::(_debug) Task ef3f0ed1-f27b-4979-8da1-f5be3b56ddda: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000320911407471', 'lastCheck': 1384895503.3827341, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00226998329163', 'lastCheck': 1384895498.1895151, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895503.156425, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000459909439087', 'lastCheck': 1384895501.5744121, 'valid': True, 'code': 0}} >Thread-15910372::DEBUG::2013-11-19 21:11:45,336::task::495::TaskManager.Task::(_debug) Task ef3f0ed1-f27b-4979-8da1-f5be3b56ddda: moving from state preparing -> state finished >Thread-15910372::DEBUG::2013-11-19 21:11:45,336::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910372::DEBUG::2013-11-19 21:11:45,337::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910372::DEBUG::2013-11-19 21:11:45,337::task::495::TaskManager.Task::(_debug) Task ef3f0ed1-f27b-4979-8da1-f5be3b56ddda: ref 0 aborting False >Thread-15910372::INFO::2013-11-19 21:11:45,338::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000320911407471', 'lastCheck': 1384895503.3827341, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00226998329163', 'lastCheck': 1384895498.1895151, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895503.156425, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000459909439087', 'lastCheck': 1384895501.5744121, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:11:53,168::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910378::INFO::2013-11-19 21:11:55,459::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910378::DEBUG::2013-11-19 21:11:55,459::task::495::TaskManager.Task::(_debug) Task 2753572b-2b91-4b35-8272-ad5c6f214e04: moving from state init -> state preparing >Thread-15910378::DEBUG::2013-11-19 21:11:55,460::task::495::TaskManager.Task::(_debug) Task 2753572b-2b91-4b35-8272-ad5c6f214e04: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000318050384521', 'lastCheck': 1384895513.395658, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00134205818176', 'lastCheck': 1384895508.2047901, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895513.169647, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000324010848999', 'lastCheck': 1384895511.585988, 'valid': True, 'code': 0}} >Thread-15910378::DEBUG::2013-11-19 21:11:55,461::task::495::TaskManager.Task::(_debug) Task 2753572b-2b91-4b35-8272-ad5c6f214e04: moving from state preparing -> state finished >Thread-15910378::DEBUG::2013-11-19 21:11:55,461::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910378::DEBUG::2013-11-19 21:11:55,462::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910378::DEBUG::2013-11-19 21:11:55,462::task::495::TaskManager.Task::(_debug) Task 2753572b-2b91-4b35-8272-ad5c6f214e04: ref 0 aborting False >Thread-15910378::INFO::2013-11-19 21:11:55,463::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000318050384521', 'lastCheck': 1384895513.395658, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00134205818176', 'lastCheck': 1384895508.2047901, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895513.169647, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000324010848999', 'lastCheck': 1384895511.585988, 'valid': True, 'code': 0}} >Thread-42840::DEBUG::2013-11-19 21:12:01,597::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 21:12:02,214::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:12:03,181::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910384::INFO::2013-11-19 21:12:05,626::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910384::DEBUG::2013-11-19 21:12:05,627::task::495::TaskManager.Task::(_debug) Task 1f943a22-502b-4eb5-9969-a304790c98b9: moving from state init -> state preparing >Thread-15910384::DEBUG::2013-11-19 21:12:05,628::task::495::TaskManager.Task::(_debug) Task 1f943a22-502b-4eb5-9969-a304790c98b9: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00042200088501', 'lastCheck': 1384895523.4093571, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00135207176208', 'lastCheck': 1384895518.2201409, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895523.1826639, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000316143035889', 'lastCheck': 1384895522.2175879, 'valid': True, 'code': 0}} >Thread-15910384::DEBUG::2013-11-19 21:12:05,629::task::495::TaskManager.Task::(_debug) Task 1f943a22-502b-4eb5-9969-a304790c98b9: moving from state preparing -> state finished >Thread-15910384::DEBUG::2013-11-19 21:12:05,629::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910384::DEBUG::2013-11-19 21:12:05,630::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910384::DEBUG::2013-11-19 21:12:05,630::task::495::TaskManager.Task::(_debug) Task 1f943a22-502b-4eb5-9969-a304790c98b9: ref 0 aborting False >Thread-15910384::INFO::2013-11-19 21:12:05,631::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00042200088501', 'lastCheck': 1384895523.4093571, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00135207176208', 'lastCheck': 1384895518.2201409, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895523.1826639, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000316143035889', 'lastCheck': 1384895522.2175879, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:12:13,195::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910390::INFO::2013-11-19 21:12:15,797::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910390::DEBUG::2013-11-19 21:12:15,797::task::495::TaskManager.Task::(_debug) Task e3e71470-c449-49c8-a8d4-71616dc5ded7: moving from state init -> state preparing >Thread-15910390::DEBUG::2013-11-19 21:12:15,798::task::495::TaskManager.Task::(_debug) Task e3e71470-c449-49c8-a8d4-71616dc5ded7: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000313997268677', 'lastCheck': 1384895533.4224761, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152707099915', 'lastCheck': 1384895528.2353671, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895533.196115, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000421047210693', 'lastCheck': 1384895532.229032, 'valid': True, 'code': 0}} >Thread-15910390::DEBUG::2013-11-19 21:12:15,799::task::495::TaskManager.Task::(_debug) Task e3e71470-c449-49c8-a8d4-71616dc5ded7: moving from state preparing -> state finished >Thread-15910390::DEBUG::2013-11-19 21:12:15,799::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910390::DEBUG::2013-11-19 21:12:15,800::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910390::DEBUG::2013-11-19 21:12:15,800::task::495::TaskManager.Task::(_debug) Task e3e71470-c449-49c8-a8d4-71616dc5ded7: ref 0 aborting False >Thread-15910390::INFO::2013-11-19 21:12:15,805::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000313997268677', 'lastCheck': 1384895533.4224761, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152707099915', 'lastCheck': 1384895528.2353671, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895533.196115, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000421047210693', 'lastCheck': 1384895532.229032, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:12:23,208::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910396::INFO::2013-11-19 21:12:25,984::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910396::DEBUG::2013-11-19 21:12:25,985::task::495::TaskManager.Task::(_debug) Task 1192731e-a6c8-48ab-9dd9-8b661e651d49: moving from state init -> state preparing >Thread-15910396::DEBUG::2013-11-19 21:12:25,985::task::495::TaskManager.Task::(_debug) Task 1192731e-a6c8-48ab-9dd9-8b661e651d49: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000319004058838', 'lastCheck': 1384895543.4355631, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137305259705', 'lastCheck': 1384895538.2509029, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895543.2094221, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000339984893799', 'lastCheck': 1384895542.24038, 'valid': True, 'code': 0}} >Thread-15910396::DEBUG::2013-11-19 21:12:25,986::task::495::TaskManager.Task::(_debug) Task 1192731e-a6c8-48ab-9dd9-8b661e651d49: moving from state preparing -> state finished >Thread-15910396::DEBUG::2013-11-19 21:12:25,987::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910396::DEBUG::2013-11-19 21:12:25,987::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910396::DEBUG::2013-11-19 21:12:25,988::task::495::TaskManager.Task::(_debug) Task 1192731e-a6c8-48ab-9dd9-8b661e651d49: ref 0 aborting False >Thread-15910396::INFO::2013-11-19 21:12:25,992::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000319004058838', 'lastCheck': 1384895543.4355631, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137305259705', 'lastCheck': 1384895538.2509029, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895543.2094221, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000339984893799', 'lastCheck': 1384895542.24038, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:12:33,222::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910402::INFO::2013-11-19 21:12:36,122::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910402::DEBUG::2013-11-19 21:12:36,123::task::495::TaskManager.Task::(_debug) Task b413f516-df14-41f8-b476-45dcc838844c: moving from state init -> state preparing >Thread-15910402::DEBUG::2013-11-19 21:12:36,124::task::495::TaskManager.Task::(_debug) Task b413f516-df14-41f8-b476-45dcc838844c: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000382900238037', 'lastCheck': 1384895553.4489319, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00140404701233', 'lastCheck': 1384895548.2661879, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895553.222811, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000334024429321', 'lastCheck': 1384895552.251694, 'valid': True, 'code': 0}} >Thread-15910402::DEBUG::2013-11-19 21:12:36,124::task::495::TaskManager.Task::(_debug) Task b413f516-df14-41f8-b476-45dcc838844c: moving from state preparing -> state finished >Thread-15910402::DEBUG::2013-11-19 21:12:36,125::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910402::DEBUG::2013-11-19 21:12:36,126::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910402::DEBUG::2013-11-19 21:12:36,126::task::495::TaskManager.Task::(_debug) Task b413f516-df14-41f8-b476-45dcc838844c: ref 0 aborting False >Thread-15910402::INFO::2013-11-19 21:12:36,127::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000382900238037', 'lastCheck': 1384895553.4489319, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00140404701233', 'lastCheck': 1384895548.2661879, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895553.222811, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000334024429321', 'lastCheck': 1384895552.251694, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:12:43,235::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910408::INFO::2013-11-19 21:12:46,238::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910408::DEBUG::2013-11-19 21:12:46,238::task::495::TaskManager.Task::(_debug) Task d8e2ed60-cdf7-4cc6-a4ba-991e77cb8453: moving from state init -> state preparing >Thread-15910408::DEBUG::2013-11-19 21:12:46,239::task::495::TaskManager.Task::(_debug) Task d8e2ed60-cdf7-4cc6-a4ba-991e77cb8453: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000336885452271', 'lastCheck': 1384895563.4620769, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00220012664795', 'lastCheck': 1384895558.282846, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895563.2362289, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000450134277344', 'lastCheck': 1384895562.263191, 'valid': True, 'code': 0}} >Thread-15910408::DEBUG::2013-11-19 21:12:46,240::task::495::TaskManager.Task::(_debug) Task d8e2ed60-cdf7-4cc6-a4ba-991e77cb8453: moving from state preparing -> state finished >Thread-15910408::DEBUG::2013-11-19 21:12:46,241::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910408::DEBUG::2013-11-19 21:12:46,241::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910408::DEBUG::2013-11-19 21:12:46,242::task::495::TaskManager.Task::(_debug) Task d8e2ed60-cdf7-4cc6-a4ba-991e77cb8453: ref 0 aborting False >Thread-15910408::INFO::2013-11-19 21:12:46,242::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000336885452271', 'lastCheck': 1384895563.4620769, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00220012664795', 'lastCheck': 1384895558.282846, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895563.2362289, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000450134277344', 'lastCheck': 1384895562.263191, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:12:53,249::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910414::INFO::2013-11-19 21:12:56,349::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910414::DEBUG::2013-11-19 21:12:56,350::task::495::TaskManager.Task::(_debug) Task 9ea9a0c5-d016-4499-ad81-6ab25a38a53b: moving from state init -> state preparing >Thread-15910414::DEBUG::2013-11-19 21:12:56,351::task::495::TaskManager.Task::(_debug) Task 9ea9a0c5-d016-4499-ad81-6ab25a38a53b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000330924987793', 'lastCheck': 1384895573.475178, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00134682655334', 'lastCheck': 1384895568.2990451, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895573.2495561, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000304937362671', 'lastCheck': 1384895572.274518, 'valid': True, 'code': 0}} >Thread-15910414::DEBUG::2013-11-19 21:12:56,351::task::495::TaskManager.Task::(_debug) Task 9ea9a0c5-d016-4499-ad81-6ab25a38a53b: moving from state preparing -> state finished >Thread-15910414::DEBUG::2013-11-19 21:12:56,352::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910414::DEBUG::2013-11-19 21:12:56,352::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910414::DEBUG::2013-11-19 21:12:56,353::task::495::TaskManager.Task::(_debug) Task 9ea9a0c5-d016-4499-ad81-6ab25a38a53b: ref 0 aborting False >Thread-15910414::INFO::2013-11-19 21:12:56,354::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000330924987793', 'lastCheck': 1384895573.475178, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00134682655334', 'lastCheck': 1384895568.2990451, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895573.2495561, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000304937362671', 'lastCheck': 1384895572.274518, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:13:03,262::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910420::INFO::2013-11-19 21:13:06,488::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910420::DEBUG::2013-11-19 21:13:06,489::task::495::TaskManager.Task::(_debug) Task e28b579c-e1fe-455d-8bab-128df95d5243: moving from state init -> state preparing >Thread-15910420::DEBUG::2013-11-19 21:13:06,490::task::495::TaskManager.Task::(_debug) Task e28b579c-e1fe-455d-8bab-128df95d5243: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000352144241333', 'lastCheck': 1384895583.4882181, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139284133911', 'lastCheck': 1384895578.314255, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895583.2632959, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000420093536377', 'lastCheck': 1384895582.2859011, 'valid': True, 'code': 0}} >Thread-15910420::DEBUG::2013-11-19 21:13:06,492::task::495::TaskManager.Task::(_debug) Task e28b579c-e1fe-455d-8bab-128df95d5243: moving from state preparing -> state finished >Thread-15910420::DEBUG::2013-11-19 21:13:06,492::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910420::DEBUG::2013-11-19 21:13:06,493::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910420::DEBUG::2013-11-19 21:13:06,494::task::495::TaskManager.Task::(_debug) Task e28b579c-e1fe-455d-8bab-128df95d5243: ref 0 aborting False >Thread-15910420::INFO::2013-11-19 21:13:06,496::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000352144241333', 'lastCheck': 1384895583.4882181, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139284133911', 'lastCheck': 1384895578.314255, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895583.2632959, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000420093536377', 'lastCheck': 1384895582.2859011, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:13:13,275::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910426::INFO::2013-11-19 21:13:16,583::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910426::DEBUG::2013-11-19 21:13:16,584::task::495::TaskManager.Task::(_debug) Task fff5f27d-7fdf-46b7-b996-bf9a751a82d4: moving from state init -> state preparing >Thread-15910426::DEBUG::2013-11-19 21:13:16,585::task::495::TaskManager.Task::(_debug) Task fff5f27d-7fdf-46b7-b996-bf9a751a82d4: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000400066375732', 'lastCheck': 1384895593.501384, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00136494636536', 'lastCheck': 1384895588.329561, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895593.276644, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000404834747314', 'lastCheck': 1384895592.297287, 'valid': True, 'code': 0}} >Thread-15910426::DEBUG::2013-11-19 21:13:16,586::task::495::TaskManager.Task::(_debug) Task fff5f27d-7fdf-46b7-b996-bf9a751a82d4: moving from state preparing -> state finished >Thread-15910426::DEBUG::2013-11-19 21:13:16,586::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910426::DEBUG::2013-11-19 21:13:16,587::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910426::DEBUG::2013-11-19 21:13:16,587::task::495::TaskManager.Task::(_debug) Task fff5f27d-7fdf-46b7-b996-bf9a751a82d4: ref 0 aborting False >Thread-15910426::INFO::2013-11-19 21:13:16,588::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000400066375732', 'lastCheck': 1384895593.501384, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00136494636536', 'lastCheck': 1384895588.329561, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895593.276644, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000404834747314', 'lastCheck': 1384895592.297287, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:13:23,289::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910432::INFO::2013-11-19 21:13:26,733::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910432::DEBUG::2013-11-19 21:13:26,734::task::495::TaskManager.Task::(_debug) Task 9e87bf66-b718-46e9-990f-3326e938cd8d: moving from state init -> state preparing >Thread-15910432::DEBUG::2013-11-19 21:13:26,735::task::495::TaskManager.Task::(_debug) Task 9e87bf66-b718-46e9-990f-3326e938cd8d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000308990478516', 'lastCheck': 1384895603.514235, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139784812927', 'lastCheck': 1384895598.345098, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895603.290184, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000298976898193', 'lastCheck': 1384895602.308568, 'valid': True, 'code': 0}} >Thread-15910432::DEBUG::2013-11-19 21:13:26,735::task::495::TaskManager.Task::(_debug) Task 9e87bf66-b718-46e9-990f-3326e938cd8d: moving from state preparing -> state finished >Thread-15910432::DEBUG::2013-11-19 21:13:26,736::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910432::DEBUG::2013-11-19 21:13:26,737::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910432::DEBUG::2013-11-19 21:13:26,737::task::495::TaskManager.Task::(_debug) Task 9e87bf66-b718-46e9-990f-3326e938cd8d: ref 0 aborting False >Thread-15910432::INFO::2013-11-19 21:13:26,738::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000308990478516', 'lastCheck': 1384895603.514235, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139784812927', 'lastCheck': 1384895598.345098, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895603.290184, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000298976898193', 'lastCheck': 1384895602.308568, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:13:33,302::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910438::INFO::2013-11-19 21:13:36,820::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910438::DEBUG::2013-11-19 21:13:36,821::task::495::TaskManager.Task::(_debug) Task 6db58ab8-804a-41f7-832e-0758b2d19bdb: moving from state init -> state preparing >Thread-15910438::DEBUG::2013-11-19 21:13:36,822::task::495::TaskManager.Task::(_debug) Task 6db58ab8-804a-41f7-832e-0758b2d19bdb: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000391960144043', 'lastCheck': 1384895613.5273261, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00134181976318', 'lastCheck': 1384895608.36025, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895613.3036339, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000415086746216', 'lastCheck': 1384895612.3199761, 'valid': True, 'code': 0}} >Thread-15910438::DEBUG::2013-11-19 21:13:36,823::task::495::TaskManager.Task::(_debug) Task 6db58ab8-804a-41f7-832e-0758b2d19bdb: moving from state preparing -> state finished >Thread-15910438::DEBUG::2013-11-19 21:13:36,823::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910438::DEBUG::2013-11-19 21:13:36,824::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910438::DEBUG::2013-11-19 21:13:36,824::task::495::TaskManager.Task::(_debug) Task 6db58ab8-804a-41f7-832e-0758b2d19bdb: ref 0 aborting False >Thread-15910438::INFO::2013-11-19 21:13:36,825::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000391960144043', 'lastCheck': 1384895613.5273261, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00134181976318', 'lastCheck': 1384895608.36025, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895613.3036339, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000415086746216', 'lastCheck': 1384895612.3199761, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:13:43,316::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910444::INFO::2013-11-19 21:13:46,954::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910444::DEBUG::2013-11-19 21:13:46,955::task::495::TaskManager.Task::(_debug) Task 1ae61d88-0bee-4aaa-b98d-eb696c39a390: moving from state init -> state preparing >Thread-15910444::DEBUG::2013-11-19 21:13:46,956::task::495::TaskManager.Task::(_debug) Task 1ae61d88-0bee-4aaa-b98d-eb696c39a390: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000324010848999', 'lastCheck': 1384895623.540354, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00223207473755', 'lastCheck': 1384895618.3769441, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895623.3171301, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000436067581177', 'lastCheck': 1384895622.331425, 'valid': True, 'code': 0}} >Thread-15910444::DEBUG::2013-11-19 21:13:46,957::task::495::TaskManager.Task::(_debug) Task 1ae61d88-0bee-4aaa-b98d-eb696c39a390: moving from state preparing -> state finished >Thread-15910444::DEBUG::2013-11-19 21:13:46,957::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910444::DEBUG::2013-11-19 21:13:46,958::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910444::DEBUG::2013-11-19 21:13:46,958::task::495::TaskManager.Task::(_debug) Task 1ae61d88-0bee-4aaa-b98d-eb696c39a390: ref 0 aborting False >Thread-15910444::INFO::2013-11-19 21:13:46,959::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000324010848999', 'lastCheck': 1384895623.540354, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00223207473755', 'lastCheck': 1384895618.3769441, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895623.3171301, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000436067581177', 'lastCheck': 1384895622.331425, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:13:53,329::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910450::INFO::2013-11-19 21:13:57,086::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910450::DEBUG::2013-11-19 21:13:57,086::task::495::TaskManager.Task::(_debug) Task 0ae757a5-c2c1-4c74-9767-47ee2783777e: moving from state init -> state preparing >Thread-15910450::DEBUG::2013-11-19 21:13:57,087::task::495::TaskManager.Task::(_debug) Task 0ae757a5-c2c1-4c74-9767-47ee2783777e: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000345945358276', 'lastCheck': 1384895633.553391, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142908096313', 'lastCheck': 1384895628.3922441, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895633.3305371, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000321865081787', 'lastCheck': 1384895632.342752, 'valid': True, 'code': 0}} >Thread-15910450::DEBUG::2013-11-19 21:13:57,088::task::495::TaskManager.Task::(_debug) Task 0ae757a5-c2c1-4c74-9767-47ee2783777e: moving from state preparing -> state finished >Thread-15910450::DEBUG::2013-11-19 21:13:57,089::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910450::DEBUG::2013-11-19 21:13:57,089::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910450::DEBUG::2013-11-19 21:13:57,090::task::495::TaskManager.Task::(_debug) Task 0ae757a5-c2c1-4c74-9767-47ee2783777e: ref 0 aborting False >Thread-15910450::INFO::2013-11-19 21:13:57,091::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000345945358276', 'lastCheck': 1384895633.553391, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142908096313', 'lastCheck': 1384895628.3922441, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895633.3305371, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000321865081787', 'lastCheck': 1384895632.342752, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:14:03,343::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910456::INFO::2013-11-19 21:14:07,193::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910456::DEBUG::2013-11-19 21:14:07,194::task::495::TaskManager.Task::(_debug) Task b88d54c4-c7d0-4208-b83b-0de4e0ffd361: moving from state init -> state preparing >Thread-15910456::DEBUG::2013-11-19 21:14:07,195::task::495::TaskManager.Task::(_debug) Task b88d54c4-c7d0-4208-b83b-0de4e0ffd361: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00037693977356', 'lastCheck': 1384895643.5665641, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00138711929321', 'lastCheck': 1384895638.4074831, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895643.3443849, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000458955764771', 'lastCheck': 1384895642.3542509, 'valid': True, 'code': 0}} >Thread-15910456::DEBUG::2013-11-19 21:14:07,195::task::495::TaskManager.Task::(_debug) Task b88d54c4-c7d0-4208-b83b-0de4e0ffd361: moving from state preparing -> state finished >Thread-15910456::DEBUG::2013-11-19 21:14:07,196::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910456::DEBUG::2013-11-19 21:14:07,197::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910456::DEBUG::2013-11-19 21:14:07,197::task::495::TaskManager.Task::(_debug) Task b88d54c4-c7d0-4208-b83b-0de4e0ffd361: ref 0 aborting False >Thread-15910456::INFO::2013-11-19 21:14:07,198::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00037693977356', 'lastCheck': 1384895643.5665641, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00138711929321', 'lastCheck': 1384895638.4074831, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895643.3443849, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000458955764771', 'lastCheck': 1384895642.3542509, 'valid': True, 'code': 0}} >Thread-15910458::DEBUG::2013-11-19 21:14:12,764::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910458::INFO::2013-11-19 21:14:12,765::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910458::DEBUG::2013-11-19 21:14:12,775::task::495::TaskManager.Task::(_debug) Task 9715bbe1-1522-4c9f-8e63-2c2beffe8d45: moving from state init -> state preparing >Thread-15910458::DEBUG::2013-11-19 21:14:12,776::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=Free spmLver=274 spmId=-1 >Thread-15910458::DEBUG::2013-11-19 21:14:12,776::task::495::TaskManager.Task::(_debug) Task 9715bbe1-1522-4c9f-8e63-2c2beffe8d45: finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 274}} >Thread-15910458::DEBUG::2013-11-19 21:14:12,777::task::495::TaskManager.Task::(_debug) Task 9715bbe1-1522-4c9f-8e63-2c2beffe8d45: moving from state preparing -> state finished >Thread-15910458::DEBUG::2013-11-19 21:14:12,778::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910458::DEBUG::2013-11-19 21:14:12,779::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910458::DEBUG::2013-11-19 21:14:12,779::task::495::TaskManager.Task::(_debug) Task 9715bbe1-1522-4c9f-8e63-2c2beffe8d45: ref 0 aborting False >Thread-15910458::INFO::2013-11-19 21:14:12,780::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 274}} >Thread-15910459::DEBUG::2013-11-19 21:14:12,809::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910459::INFO::2013-11-19 21:14:12,812::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: spmStart, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 prevID=-1 prevLVER=274 recoveryMode=0 scsiFencing=false maxHostID=250 domVersion=2) >Thread-15910459::DEBUG::2013-11-19 21:14:12,813::task::495::TaskManager.Task::(_debug) Task 88cc2f68-e802-4615-bced-c7a8c4aefa33: moving from state init -> state preparing >Thread-15910459::DEBUG::2013-11-19 21:14:12,814::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5b6b8e38-b3df-4bd9-a75b-bba6ca4b10ea`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '330' at 'public_spmStart' >Thread-15910459::DEBUG::2013-11-19 21:14:12,815::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >Thread-15910459::DEBUG::2013-11-19 21:14:12,815::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >Thread-15910459::DEBUG::2013-11-19 21:14:12,816::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5b6b8e38-b3df-4bd9-a75b-bba6ca4b10ea`::Granted request >Thread-15910459::DEBUG::2013-11-19 21:14:12,817::task::495::TaskManager.Task::(_debug) Task 88cc2f68-e802-4615-bced-c7a8c4aefa33: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (exclusive) >Thread-15910459::DEBUG::2013-11-19 21:14:12,817::task::495::TaskManager.Task::(_debug) Task 88cc2f68-e802-4615-bced-c7a8c4aefa33: ref 1 aborting False >Thread-15910459::DEBUG::2013-11-19 21:14:12,818::task::495::TaskManager.Task::(_debug) Task 88cc2f68-e802-4615-bced-c7a8c4aefa33: Prepare: 1 jobs exist, move to acquiring >Thread-15910459::DEBUG::2013-11-19 21:14:12,819::task::495::TaskManager.Task::(_debug) Task 88cc2f68-e802-4615-bced-c7a8c4aefa33: moving from state preparing -> state acquiring >Thread-15910459::DEBUG::2013-11-19 21:14:12,819::task::495::TaskManager.Task::(_debug) Task 88cc2f68-e802-4615-bced-c7a8c4aefa33: moving from state acquiring -> state queued >Thread-15910459::DEBUG::2013-11-19 21:14:12,820::taskManager::37::TaskManager::(_queueTask) queueing task: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910459::DEBUG::2013-11-19 21:14:12,820::taskManager::43::TaskManager::(_queueTask) task queued: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >57fe3c04-4010-4c2b-b99c-03efcc17c092::DEBUG::2013-11-19 21:14:12,821::threadPool::47::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-15910459::DEBUG::2013-11-19 21:14:12,821::task::1183::TaskManager.Task::(prepare) returning >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:12,822::threadPool::192::Misc.ThreadPool.WorkerThread::(run) Task: 88cc2f68-e802-4615-bced-c7a8c4aefa33 running: <bound method Task.commit of <storage.task.Task instance at 0x7f487c13ad40>> with: None >Thread-15910459::DEBUG::2013-11-19 21:14:12,823::task::495::TaskManager.Task::(_debug) Task 88cc2f68-e802-4615-bced-c7a8c4aefa33: ref 0 aborting False >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:12,823::task::1195::TaskManager.Task::(commit) Commiting task: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910459::INFO::2013-11-19 21:14:12,824::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: spmStart, Return response: {'status': {'message': 'OK', 'code': 0}, 'uuid': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:12,825::task::495::TaskManager.Task::(_debug) Task 88cc2f68-e802-4615-bced-c7a8c4aefa33: moving from state queued -> state running >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:12,828::task::907::TaskManager.Task::(_runJobs) Task.run: running job 0: spmStart: <bound method SPM.start of <storage.spm.SPM instance at 0x1bf9f38>> (args: ('c4e5889a-348d-11e2-83fe-9f72369eab30', -1, '274', 0, 'false', 250, 2) kwargs: {}) >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:12,829::task::300::TaskManager.Task::(run) Job.run: running spmStart: <bound method SPM.start of <storage.spm.SPM instance at 0x1bf9f38>> (args: ('c4e5889a-348d-11e2-83fe-9f72369eab30', -1, '274', 0, 'false', 250, 2) kwargs: {}) callback None >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:12,830::spm::433::Storage.SPM::(start) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:12,830::misc::1010::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:12,831::misc::1012::SamplingMethod::(__call__) Got in to sampling method >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:12,831::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:12,832::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:12,833::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:12,833::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:12,834::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:12,835::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:14:13,358::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:13,548::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:13,551::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:13,552::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=276', u'POOL_SPM_ID=-1', u'_SHA_CKSUM=d1ce634f167d7f0ddd58cf2ebbdf434d45f6ef67'] >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:13,553::misc::1020::SamplingMethod::(__call__) Returning last result >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:13,554::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:13,555::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:13,555::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:13,556::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:13,557::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:13,558::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 21:14:13,775::lvm::439::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-42838::DEBUG::2013-11-19 21:14:13,781::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15910460::DEBUG::2013-11-19 21:14:14,130::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910460::INFO::2013-11-19 21:14:14,133::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910460::DEBUG::2013-11-19 21:14:14,133::task::495::TaskManager.Task::(_debug) Task c64690c8-6eb8-444a-ac09-334f327564a6: moving from state init -> state preparing >Thread-15910460::DEBUG::2013-11-19 21:14:14,134::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910460::DEBUG::2013-11-19 21:14:14,135::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910460::DEBUG::2013-11-19 21:14:14,136::task::495::TaskManager.Task::(_debug) Task c64690c8-6eb8-444a-ac09-334f327564a6: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910460::DEBUG::2013-11-19 21:14:14,136::task::495::TaskManager.Task::(_debug) Task c64690c8-6eb8-444a-ac09-334f327564a6: moving from state preparing -> state finished >Thread-15910460::DEBUG::2013-11-19 21:14:14,137::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910460::DEBUG::2013-11-19 21:14:14,137::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910460::DEBUG::2013-11-19 21:14:14,138::task::495::TaskManager.Task::(_debug) Task c64690c8-6eb8-444a-ac09-334f327564a6: ref 0 aborting False >Thread-15910460::INFO::2013-11-19 21:14:14,139::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:14,333::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:14,336::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=276', u'POOL_SPM_ID=-1', u'_SHA_CKSUM=d1ce634f167d7f0ddd58cf2ebbdf434d45f6ef67'] >88cc2f68-e802-4615-bced-c7a8c4aefa33::INFO::2013-11-19 21:14:14,338::spm::457::Storage.SPM::(start) expected previd:-1 lver:276 got request for previd:-1 lver:274 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:14,339::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting... >Thread-42838::DEBUG::2013-11-19 21:14:14,567::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-42838::DEBUG::2013-11-19 21:14:14,570::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:14,571::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:14,572::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:14,574::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:14,576::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:14,576::persistentDict::163::Storage.PersistentDict::(__init__) Created a persistant dict with LvMetadataRW backend >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:14,577::blockSD::164::Storage.Misc.excCmd::(readlines) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:14,973::blockSD::164::Storage.Misc.excCmd::(readlines) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000205295 s, 10.0 MB/s\n'; <rc> = 0 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:14,976::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000205295 s, 10.0 MB/s'], size: 2048 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:14,977::persistentDict::204::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >88cc2f68-e802-4615-bced-c7a8c4aefa33::WARNING::2013-11-19 21:14:14,977::persistentDict::226::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:14,978::persistentDict::163::Storage.PersistentDict::(__init__) Created a persistant dict with VGTagMetadataRW backend >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:14,979::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:14,979::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:14,980::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:14,981::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:14,981::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:14,983::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15910461::DEBUG::2013-11-19 21:14:15,188::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910461::INFO::2013-11-19 21:14:15,191::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910461::DEBUG::2013-11-19 21:14:15,192::task::495::TaskManager.Task::(_debug) Task f10b365d-7645-4780-baec-05ea83407463: moving from state init -> state preparing >Thread-15910461::DEBUG::2013-11-19 21:14:15,193::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910461::DEBUG::2013-11-19 21:14:15,193::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910461::DEBUG::2013-11-19 21:14:15,194::task::495::TaskManager.Task::(_debug) Task f10b365d-7645-4780-baec-05ea83407463: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910461::DEBUG::2013-11-19 21:14:15,195::task::495::TaskManager.Task::(_debug) Task f10b365d-7645-4780-baec-05ea83407463: moving from state preparing -> state finished >Thread-15910461::DEBUG::2013-11-19 21:14:15,196::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910461::DEBUG::2013-11-19 21:14:15,197::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910461::DEBUG::2013-11-19 21:14:15,198::task::495::TaskManager.Task::(_debug) Task f10b365d-7645-4780-baec-05ea83407463: ref 0 aborting False >Thread-15910461::INFO::2013-11-19 21:14:15,199::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:15,638::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:15,641::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:15,642::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=276', u'POOL_SPM_ID=-1', u'_SHA_CKSUM=d1ce634f167d7f0ddd58cf2ebbdf434d45f6ef67'] >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:15,643::safelease::52::ClusterLock::(acquire) Acquiring cluster lock for domain 6b892067-ac7e-4039-a7be-5b7387525b16 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:15,644::safelease::61::Storage.Misc.excCmd::(acquire) '/usr/bin/sudo -n /usr/bin/setsid /usr/bin/ionice -c1 -n0 /bin/su vdsm -s /bin/sh -c "/usr/libexec/vdsm/spmprotect.sh start 6b892067-ac7e-4039-a7be-5b7387525b16 2 5 /dev/6b892067-ac7e-4039-a7be-5b7387525b16/leases 60000 10000 3"' (cwd /usr/libexec/vdsm/) >Thread-15910462::DEBUG::2013-11-19 21:14:16,223::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910462::INFO::2013-11-19 21:14:16,225::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910462::DEBUG::2013-11-19 21:14:16,226::task::495::TaskManager.Task::(_debug) Task 9aa45c3d-79c7-451a-ad8a-018cb1eb5810: moving from state init -> state preparing >Thread-15910462::DEBUG::2013-11-19 21:14:16,226::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910462::DEBUG::2013-11-19 21:14:16,227::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910462::DEBUG::2013-11-19 21:14:16,227::task::495::TaskManager.Task::(_debug) Task 9aa45c3d-79c7-451a-ad8a-018cb1eb5810: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910462::DEBUG::2013-11-19 21:14:16,228::task::495::TaskManager.Task::(_debug) Task 9aa45c3d-79c7-451a-ad8a-018cb1eb5810: moving from state preparing -> state finished >Thread-15910462::DEBUG::2013-11-19 21:14:16,229::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910462::DEBUG::2013-11-19 21:14:16,229::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910462::DEBUG::2013-11-19 21:14:16,230::task::495::TaskManager.Task::(_debug) Task 9aa45c3d-79c7-451a-ad8a-018cb1eb5810: ref 0 aborting False >Thread-15910462::INFO::2013-11-19 21:14:16,230::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910463::DEBUG::2013-11-19 21:14:17,239::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910463::INFO::2013-11-19 21:14:17,240::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910463::DEBUG::2013-11-19 21:14:17,240::task::495::TaskManager.Task::(_debug) Task 55e70696-5317-4733-b499-880e08c0de0c: moving from state init -> state preparing >Thread-15910463::DEBUG::2013-11-19 21:14:17,241::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910463::DEBUG::2013-11-19 21:14:17,241::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910463::DEBUG::2013-11-19 21:14:17,242::task::495::TaskManager.Task::(_debug) Task 55e70696-5317-4733-b499-880e08c0de0c: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910463::DEBUG::2013-11-19 21:14:17,243::task::495::TaskManager.Task::(_debug) Task 55e70696-5317-4733-b499-880e08c0de0c: moving from state preparing -> state finished >Thread-15910463::DEBUG::2013-11-19 21:14:17,243::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910463::DEBUG::2013-11-19 21:14:17,244::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910463::DEBUG::2013-11-19 21:14:17,244::task::495::TaskManager.Task::(_debug) Task 55e70696-5317-4733-b499-880e08c0de0c: ref 0 aborting False >Thread-15910463::INFO::2013-11-19 21:14:17,245::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910464::DEBUG::2013-11-19 21:14:18,263::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910464::INFO::2013-11-19 21:14:18,263::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910464::DEBUG::2013-11-19 21:14:18,264::task::495::TaskManager.Task::(_debug) Task cf61f899-ed74-4e0d-ab6f-0f51e9a9be92: moving from state init -> state preparing >Thread-15910464::DEBUG::2013-11-19 21:14:18,265::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910464::DEBUG::2013-11-19 21:14:18,265::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910464::DEBUG::2013-11-19 21:14:18,266::task::495::TaskManager.Task::(_debug) Task cf61f899-ed74-4e0d-ab6f-0f51e9a9be92: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910464::DEBUG::2013-11-19 21:14:18,266::task::495::TaskManager.Task::(_debug) Task cf61f899-ed74-4e0d-ab6f-0f51e9a9be92: moving from state preparing -> state finished >Thread-15910464::DEBUG::2013-11-19 21:14:18,267::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910464::DEBUG::2013-11-19 21:14:18,267::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910464::DEBUG::2013-11-19 21:14:18,268::task::495::TaskManager.Task::(_debug) Task cf61f899-ed74-4e0d-ab6f-0f51e9a9be92: ref 0 aborting False >Thread-15910464::INFO::2013-11-19 21:14:18,268::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910465::DEBUG::2013-11-19 21:14:19,281::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910465::INFO::2013-11-19 21:14:19,282::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910465::DEBUG::2013-11-19 21:14:19,283::task::495::TaskManager.Task::(_debug) Task 4113e920-ed0b-4391-b3aa-051f91318fc2: moving from state init -> state preparing >Thread-15910465::DEBUG::2013-11-19 21:14:19,284::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910465::DEBUG::2013-11-19 21:14:19,284::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910465::DEBUG::2013-11-19 21:14:19,285::task::495::TaskManager.Task::(_debug) Task 4113e920-ed0b-4391-b3aa-051f91318fc2: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910465::DEBUG::2013-11-19 21:14:19,285::task::495::TaskManager.Task::(_debug) Task 4113e920-ed0b-4391-b3aa-051f91318fc2: moving from state preparing -> state finished >Thread-15910465::DEBUG::2013-11-19 21:14:19,286::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910465::DEBUG::2013-11-19 21:14:19,286::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910465::DEBUG::2013-11-19 21:14:19,287::task::495::TaskManager.Task::(_debug) Task 4113e920-ed0b-4391-b3aa-051f91318fc2: ref 0 aborting False >Thread-15910465::INFO::2013-11-19 21:14:19,287::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910466::DEBUG::2013-11-19 21:14:20,304::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910466::INFO::2013-11-19 21:14:20,305::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910466::DEBUG::2013-11-19 21:14:20,306::task::495::TaskManager.Task::(_debug) Task f0a3828e-3564-474b-a6e5-4421549393ae: moving from state init -> state preparing >Thread-15910466::DEBUG::2013-11-19 21:14:20,306::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910466::DEBUG::2013-11-19 21:14:20,307::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910466::DEBUG::2013-11-19 21:14:20,307::task::495::TaskManager.Task::(_debug) Task f0a3828e-3564-474b-a6e5-4421549393ae: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910466::DEBUG::2013-11-19 21:14:20,308::task::495::TaskManager.Task::(_debug) Task f0a3828e-3564-474b-a6e5-4421549393ae: moving from state preparing -> state finished >Thread-15910466::DEBUG::2013-11-19 21:14:20,308::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910466::DEBUG::2013-11-19 21:14:20,309::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910466::DEBUG::2013-11-19 21:14:20,310::task::495::TaskManager.Task::(_debug) Task f0a3828e-3564-474b-a6e5-4421549393ae: ref 0 aborting False >Thread-15910466::INFO::2013-11-19 21:14:20,310::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910467::DEBUG::2013-11-19 21:14:21,329::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910467::INFO::2013-11-19 21:14:21,330::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910467::DEBUG::2013-11-19 21:14:21,331::task::495::TaskManager.Task::(_debug) Task d575f139-6d0a-4b85-a560-5e4549635c81: moving from state init -> state preparing >Thread-15910467::DEBUG::2013-11-19 21:14:21,331::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910467::DEBUG::2013-11-19 21:14:21,332::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910467::DEBUG::2013-11-19 21:14:21,332::task::495::TaskManager.Task::(_debug) Task d575f139-6d0a-4b85-a560-5e4549635c81: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910467::DEBUG::2013-11-19 21:14:21,333::task::495::TaskManager.Task::(_debug) Task d575f139-6d0a-4b85-a560-5e4549635c81: moving from state preparing -> state finished >Thread-15910467::DEBUG::2013-11-19 21:14:21,334::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910467::DEBUG::2013-11-19 21:14:21,334::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910467::DEBUG::2013-11-19 21:14:21,335::task::495::TaskManager.Task::(_debug) Task d575f139-6d0a-4b85-a560-5e4549635c81: ref 0 aborting False >Thread-15910467::INFO::2013-11-19 21:14:21,335::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910468::DEBUG::2013-11-19 21:14:22,341::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910468::INFO::2013-11-19 21:14:22,342::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910468::DEBUG::2013-11-19 21:14:22,342::task::495::TaskManager.Task::(_debug) Task eca382ad-7607-4261-ab0c-62de58c1d69f: moving from state init -> state preparing >Thread-15910468::DEBUG::2013-11-19 21:14:22,343::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910468::DEBUG::2013-11-19 21:14:22,344::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910468::DEBUG::2013-11-19 21:14:22,347::task::495::TaskManager.Task::(_debug) Task eca382ad-7607-4261-ab0c-62de58c1d69f: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910468::DEBUG::2013-11-19 21:14:22,348::task::495::TaskManager.Task::(_debug) Task eca382ad-7607-4261-ab0c-62de58c1d69f: moving from state preparing -> state finished >Thread-15910468::DEBUG::2013-11-19 21:14:22,348::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910468::DEBUG::2013-11-19 21:14:22,349::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910468::DEBUG::2013-11-19 21:14:22,349::task::495::TaskManager.Task::(_debug) Task eca382ad-7607-4261-ab0c-62de58c1d69f: ref 0 aborting False >Thread-15910468::INFO::2013-11-19 21:14:22,350::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910469::DEBUG::2013-11-19 21:14:23,367::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910469::INFO::2013-11-19 21:14:23,368::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910469::DEBUG::2013-11-19 21:14:23,369::task::495::TaskManager.Task::(_debug) Task 3253eb4f-7c21-4a45-a99e-9d9deedb8290: moving from state init -> state preparing >Thread-15910469::DEBUG::2013-11-19 21:14:23,369::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910469::DEBUG::2013-11-19 21:14:23,370::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910469::DEBUG::2013-11-19 21:14:23,372::task::495::TaskManager.Task::(_debug) Task 3253eb4f-7c21-4a45-a99e-9d9deedb8290: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910469::DEBUG::2013-11-19 21:14:23,373::task::495::TaskManager.Task::(_debug) Task 3253eb4f-7c21-4a45-a99e-9d9deedb8290: moving from state preparing -> state finished >Thread-1024767::ERROR::2013-11-19 21:14:23,373::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910469::DEBUG::2013-11-19 21:14:23,374::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910469::DEBUG::2013-11-19 21:14:23,375::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910469::DEBUG::2013-11-19 21:14:23,376::task::495::TaskManager.Task::(_debug) Task 3253eb4f-7c21-4a45-a99e-9d9deedb8290: ref 0 aborting False >Thread-15910469::INFO::2013-11-19 21:14:23,376::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910470::DEBUG::2013-11-19 21:14:24,382::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910470::INFO::2013-11-19 21:14:24,383::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910470::DEBUG::2013-11-19 21:14:24,384::task::495::TaskManager.Task::(_debug) Task d19617ad-3633-4722-818d-3ee568334099: moving from state init -> state preparing >Thread-15910470::DEBUG::2013-11-19 21:14:24,384::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910470::DEBUG::2013-11-19 21:14:24,385::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910470::DEBUG::2013-11-19 21:14:24,386::task::495::TaskManager.Task::(_debug) Task d19617ad-3633-4722-818d-3ee568334099: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910470::DEBUG::2013-11-19 21:14:24,386::task::495::TaskManager.Task::(_debug) Task d19617ad-3633-4722-818d-3ee568334099: moving from state preparing -> state finished >Thread-15910470::DEBUG::2013-11-19 21:14:24,387::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910470::DEBUG::2013-11-19 21:14:24,387::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910470::DEBUG::2013-11-19 21:14:24,388::task::495::TaskManager.Task::(_debug) Task d19617ad-3633-4722-818d-3ee568334099: ref 0 aborting False >Thread-15910470::INFO::2013-11-19 21:14:24,388::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-42838::DEBUG::2013-11-19 21:14:24,587::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 21:14:25,221::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15910471::DEBUG::2013-11-19 21:14:25,401::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910471::INFO::2013-11-19 21:14:25,402::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910471::DEBUG::2013-11-19 21:14:25,403::task::495::TaskManager.Task::(_debug) Task b89e0ab3-9b58-4116-b596-e353742c3fe7: moving from state init -> state preparing >Thread-15910471::DEBUG::2013-11-19 21:14:25,403::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910471::DEBUG::2013-11-19 21:14:25,404::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910471::DEBUG::2013-11-19 21:14:25,405::task::495::TaskManager.Task::(_debug) Task b89e0ab3-9b58-4116-b596-e353742c3fe7: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910471::DEBUG::2013-11-19 21:14:25,405::task::495::TaskManager.Task::(_debug) Task b89e0ab3-9b58-4116-b596-e353742c3fe7: moving from state preparing -> state finished >Thread-15910471::DEBUG::2013-11-19 21:14:25,406::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910471::DEBUG::2013-11-19 21:14:25,406::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910471::DEBUG::2013-11-19 21:14:25,407::task::495::TaskManager.Task::(_debug) Task b89e0ab3-9b58-4116-b596-e353742c3fe7: ref 0 aborting False >Thread-15910471::INFO::2013-11-19 21:14:25,407::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910472::DEBUG::2013-11-19 21:14:26,418::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910472::INFO::2013-11-19 21:14:26,419::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910472::DEBUG::2013-11-19 21:14:26,426::task::495::TaskManager.Task::(_debug) Task c20af5db-d874-4bfa-aa5d-0c3db7e24e87: moving from state init -> state preparing >Thread-15910472::DEBUG::2013-11-19 21:14:26,427::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910472::DEBUG::2013-11-19 21:14:26,427::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910472::DEBUG::2013-11-19 21:14:26,428::task::495::TaskManager.Task::(_debug) Task c20af5db-d874-4bfa-aa5d-0c3db7e24e87: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910472::DEBUG::2013-11-19 21:14:26,429::task::495::TaskManager.Task::(_debug) Task c20af5db-d874-4bfa-aa5d-0c3db7e24e87: moving from state preparing -> state finished >Thread-15910472::DEBUG::2013-11-19 21:14:26,429::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910472::DEBUG::2013-11-19 21:14:26,430::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910472::DEBUG::2013-11-19 21:14:26,430::task::495::TaskManager.Task::(_debug) Task c20af5db-d874-4bfa-aa5d-0c3db7e24e87: ref 0 aborting False >Thread-15910472::INFO::2013-11-19 21:14:26,431::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910473::DEBUG::2013-11-19 21:14:27,437::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910473::INFO::2013-11-19 21:14:27,438::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910473::DEBUG::2013-11-19 21:14:27,438::task::495::TaskManager.Task::(_debug) Task 7bbd9c7f-0b7b-420a-8220-a07f38a2c6e4: moving from state init -> state preparing >Thread-15910473::DEBUG::2013-11-19 21:14:27,439::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910473::DEBUG::2013-11-19 21:14:27,439::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910473::DEBUG::2013-11-19 21:14:27,440::task::495::TaskManager.Task::(_debug) Task 7bbd9c7f-0b7b-420a-8220-a07f38a2c6e4: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910473::DEBUG::2013-11-19 21:14:27,441::task::495::TaskManager.Task::(_debug) Task 7bbd9c7f-0b7b-420a-8220-a07f38a2c6e4: moving from state preparing -> state finished >Thread-15910473::DEBUG::2013-11-19 21:14:27,441::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910473::DEBUG::2013-11-19 21:14:27,442::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910473::DEBUG::2013-11-19 21:14:27,442::task::495::TaskManager.Task::(_debug) Task 7bbd9c7f-0b7b-420a-8220-a07f38a2c6e4: ref 0 aborting False >Thread-15910473::INFO::2013-11-19 21:14:27,443::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910474::DEBUG::2013-11-19 21:14:28,460::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910474::INFO::2013-11-19 21:14:28,461::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910474::DEBUG::2013-11-19 21:14:28,461::task::495::TaskManager.Task::(_debug) Task 689ceecd-8576-40f5-9ce8-c91121bb6be3: moving from state init -> state preparing >Thread-15910474::DEBUG::2013-11-19 21:14:28,462::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910474::DEBUG::2013-11-19 21:14:28,462::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910474::DEBUG::2013-11-19 21:14:28,463::task::495::TaskManager.Task::(_debug) Task 689ceecd-8576-40f5-9ce8-c91121bb6be3: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910474::DEBUG::2013-11-19 21:14:28,464::task::495::TaskManager.Task::(_debug) Task 689ceecd-8576-40f5-9ce8-c91121bb6be3: moving from state preparing -> state finished >Thread-15910474::DEBUG::2013-11-19 21:14:28,464::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910474::DEBUG::2013-11-19 21:14:28,465::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910474::DEBUG::2013-11-19 21:14:28,465::task::495::TaskManager.Task::(_debug) Task 689ceecd-8576-40f5-9ce8-c91121bb6be3: ref 0 aborting False >Thread-15910474::INFO::2013-11-19 21:14:28,466::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910475::DEBUG::2013-11-19 21:14:29,480::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910475::INFO::2013-11-19 21:14:29,481::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910475::DEBUG::2013-11-19 21:14:29,481::task::495::TaskManager.Task::(_debug) Task dae02429-78b9-453f-94f3-01231893f908: moving from state init -> state preparing >Thread-15910475::DEBUG::2013-11-19 21:14:29,482::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910475::DEBUG::2013-11-19 21:14:29,482::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910475::DEBUG::2013-11-19 21:14:29,483::task::495::TaskManager.Task::(_debug) Task dae02429-78b9-453f-94f3-01231893f908: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910475::DEBUG::2013-11-19 21:14:29,484::task::495::TaskManager.Task::(_debug) Task dae02429-78b9-453f-94f3-01231893f908: moving from state preparing -> state finished >Thread-15910475::DEBUG::2013-11-19 21:14:29,484::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910475::DEBUG::2013-11-19 21:14:29,485::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910475::DEBUG::2013-11-19 21:14:29,485::task::495::TaskManager.Task::(_debug) Task dae02429-78b9-453f-94f3-01231893f908: ref 0 aborting False >Thread-15910475::INFO::2013-11-19 21:14:29,486::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910476::DEBUG::2013-11-19 21:14:30,492::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910476::INFO::2013-11-19 21:14:30,494::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910476::DEBUG::2013-11-19 21:14:30,494::task::495::TaskManager.Task::(_debug) Task 2cea97ee-4bc7-4fb2-9463-66879a8c2d1a: moving from state init -> state preparing >Thread-15910476::DEBUG::2013-11-19 21:14:30,495::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910476::DEBUG::2013-11-19 21:14:30,496::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910476::DEBUG::2013-11-19 21:14:30,497::task::495::TaskManager.Task::(_debug) Task 2cea97ee-4bc7-4fb2-9463-66879a8c2d1a: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910476::DEBUG::2013-11-19 21:14:30,499::task::495::TaskManager.Task::(_debug) Task 2cea97ee-4bc7-4fb2-9463-66879a8c2d1a: moving from state preparing -> state finished >Thread-15910476::DEBUG::2013-11-19 21:14:30,500::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910476::DEBUG::2013-11-19 21:14:30,501::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910476::DEBUG::2013-11-19 21:14:30,503::task::495::TaskManager.Task::(_debug) Task 2cea97ee-4bc7-4fb2-9463-66879a8c2d1a: ref 0 aborting False >Thread-15910476::INFO::2013-11-19 21:14:30,505::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910477::DEBUG::2013-11-19 21:14:31,518::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910477::INFO::2013-11-19 21:14:31,519::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910477::DEBUG::2013-11-19 21:14:31,524::task::495::TaskManager.Task::(_debug) Task 507bfd38-35ee-4e88-86c0-8b1a3c5c9c9c: moving from state init -> state preparing >Thread-15910477::DEBUG::2013-11-19 21:14:31,524::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910477::DEBUG::2013-11-19 21:14:31,525::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910477::DEBUG::2013-11-19 21:14:31,525::task::495::TaskManager.Task::(_debug) Task 507bfd38-35ee-4e88-86c0-8b1a3c5c9c9c: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910477::DEBUG::2013-11-19 21:14:31,526::task::495::TaskManager.Task::(_debug) Task 507bfd38-35ee-4e88-86c0-8b1a3c5c9c9c: moving from state preparing -> state finished >Thread-15910477::DEBUG::2013-11-19 21:14:31,527::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910477::DEBUG::2013-11-19 21:14:31,527::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910477::DEBUG::2013-11-19 21:14:31,528::task::495::TaskManager.Task::(_debug) Task 507bfd38-35ee-4e88-86c0-8b1a3c5c9c9c: ref 0 aborting False >Thread-15910477::INFO::2013-11-19 21:14:31,528::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910478::DEBUG::2013-11-19 21:14:32,534::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910478::INFO::2013-11-19 21:14:32,535::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910478::DEBUG::2013-11-19 21:14:32,536::task::495::TaskManager.Task::(_debug) Task 7eca6f46-ddfc-4986-ac2d-d0e9cdcd4d4c: moving from state init -> state preparing >Thread-15910478::DEBUG::2013-11-19 21:14:32,537::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910478::DEBUG::2013-11-19 21:14:32,537::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910478::DEBUG::2013-11-19 21:14:32,538::task::495::TaskManager.Task::(_debug) Task 7eca6f46-ddfc-4986-ac2d-d0e9cdcd4d4c: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910478::DEBUG::2013-11-19 21:14:32,538::task::495::TaskManager.Task::(_debug) Task 7eca6f46-ddfc-4986-ac2d-d0e9cdcd4d4c: moving from state preparing -> state finished >Thread-15910478::DEBUG::2013-11-19 21:14:32,539::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910478::DEBUG::2013-11-19 21:14:32,539::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910478::DEBUG::2013-11-19 21:14:32,540::task::495::TaskManager.Task::(_debug) Task 7eca6f46-ddfc-4986-ac2d-d0e9cdcd4d4c: ref 0 aborting False >Thread-15910478::INFO::2013-11-19 21:14:32,541::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-1024767::ERROR::2013-11-19 21:14:33,388::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910479::DEBUG::2013-11-19 21:14:33,546::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910479::INFO::2013-11-19 21:14:33,547::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910479::DEBUG::2013-11-19 21:14:33,548::task::495::TaskManager.Task::(_debug) Task 364d24e3-d387-4238-9b7d-bdb88d49528f: moving from state init -> state preparing >Thread-15910479::DEBUG::2013-11-19 21:14:33,549::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910479::DEBUG::2013-11-19 21:14:33,549::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910479::DEBUG::2013-11-19 21:14:33,550::task::495::TaskManager.Task::(_debug) Task 364d24e3-d387-4238-9b7d-bdb88d49528f: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910479::DEBUG::2013-11-19 21:14:33,550::task::495::TaskManager.Task::(_debug) Task 364d24e3-d387-4238-9b7d-bdb88d49528f: moving from state preparing -> state finished >Thread-15910479::DEBUG::2013-11-19 21:14:33,551::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910479::DEBUG::2013-11-19 21:14:33,551::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910479::DEBUG::2013-11-19 21:14:33,552::task::495::TaskManager.Task::(_debug) Task 364d24e3-d387-4238-9b7d-bdb88d49528f: ref 0 aborting False >Thread-15910479::INFO::2013-11-19 21:14:33,553::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910480::DEBUG::2013-11-19 21:14:34,564::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910480::INFO::2013-11-19 21:14:34,565::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910480::DEBUG::2013-11-19 21:14:34,566::task::495::TaskManager.Task::(_debug) Task 06c4428a-99fe-4902-bcd7-5fdfa886a738: moving from state init -> state preparing >Thread-15910480::DEBUG::2013-11-19 21:14:34,566::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910480::DEBUG::2013-11-19 21:14:34,567::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910480::DEBUG::2013-11-19 21:14:34,567::task::495::TaskManager.Task::(_debug) Task 06c4428a-99fe-4902-bcd7-5fdfa886a738: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910480::DEBUG::2013-11-19 21:14:34,568::task::495::TaskManager.Task::(_debug) Task 06c4428a-99fe-4902-bcd7-5fdfa886a738: moving from state preparing -> state finished >Thread-15910480::DEBUG::2013-11-19 21:14:34,568::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910480::DEBUG::2013-11-19 21:14:34,569::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910480::DEBUG::2013-11-19 21:14:34,570::task::495::TaskManager.Task::(_debug) Task 06c4428a-99fe-4902-bcd7-5fdfa886a738: ref 0 aborting False >Thread-15910480::INFO::2013-11-19 21:14:34,570::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910481::DEBUG::2013-11-19 21:14:35,576::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910481::INFO::2013-11-19 21:14:35,577::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910481::DEBUG::2013-11-19 21:14:35,578::task::495::TaskManager.Task::(_debug) Task 45ed58d2-bcd1-4d94-8a88-0d15204429fc: moving from state init -> state preparing >Thread-15910481::DEBUG::2013-11-19 21:14:35,578::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910481::DEBUG::2013-11-19 21:14:35,579::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910481::DEBUG::2013-11-19 21:14:35,579::task::495::TaskManager.Task::(_debug) Task 45ed58d2-bcd1-4d94-8a88-0d15204429fc: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910481::DEBUG::2013-11-19 21:14:35,580::task::495::TaskManager.Task::(_debug) Task 45ed58d2-bcd1-4d94-8a88-0d15204429fc: moving from state preparing -> state finished >Thread-15910481::DEBUG::2013-11-19 21:14:35,580::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910481::DEBUG::2013-11-19 21:14:35,581::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910481::DEBUG::2013-11-19 21:14:35,581::task::495::TaskManager.Task::(_debug) Task 45ed58d2-bcd1-4d94-8a88-0d15204429fc: ref 0 aborting False >Thread-15910481::INFO::2013-11-19 21:14:35,582::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:36,082::safelease::61::Storage.Misc.excCmd::(acquire) SUCCESS: <err> = ''; <rc> = 0 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:36,083::safelease::65::ClusterLock::(acquire) Clustered lock acquired successfully >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:36,084::spm::468::Storage.SPM::(start) spm lock acquired successfully >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:36,084::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:36,085::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:36,086::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:36,086::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:36,087::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:36,088::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15910482::DEBUG::2013-11-19 21:14:36,594::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910482::INFO::2013-11-19 21:14:36,596::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910482::DEBUG::2013-11-19 21:14:36,612::task::495::TaskManager.Task::(_debug) Task 31ae7293-5be4-47cb-83e8-da0d6335f8de: moving from state init -> state preparing >Thread-15910482::DEBUG::2013-11-19 21:14:36,613::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910482::DEBUG::2013-11-19 21:14:36,613::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910482::DEBUG::2013-11-19 21:14:36,614::task::495::TaskManager.Task::(_debug) Task 31ae7293-5be4-47cb-83e8-da0d6335f8de: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910482::DEBUG::2013-11-19 21:14:36,615::task::495::TaskManager.Task::(_debug) Task 31ae7293-5be4-47cb-83e8-da0d6335f8de: moving from state preparing -> state finished >Thread-15910482::DEBUG::2013-11-19 21:14:36,615::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910482::DEBUG::2013-11-19 21:14:36,616::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910482::DEBUG::2013-11-19 21:14:36,616::task::495::TaskManager.Task::(_debug) Task 31ae7293-5be4-47cb-83e8-da0d6335f8de: ref 0 aborting False >Thread-15910482::INFO::2013-11-19 21:14:36,617::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:36,752::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:36,755::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:36,756::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=276', u'POOL_SPM_ID=-1', u'_SHA_CKSUM=d1ce634f167d7f0ddd58cf2ebbdf434d45f6ef67'] >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:36,757::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:36,758::persistentDict::146::Storage.PersistentDict::(transaction) Flushing changes >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:36,759::persistentDict::265::Storage.PersistentDict::(flush) about to write lines (VGTagMetadataRW)=[u'CLASS=Data', u'DESCRIPTION=OS', u'IOOPTIMEOUTSEC=10', u'LEASERETRIES=3', u'LEASETIMESEC=60', u'LOCKPOLICY=', u'LOCKRENEWALINTERVALSEC=5', u'LOGBLKSIZE=512', u'MASTER_VERSION=1', u'PHYBLKSIZE=512', u'POOL_DESCRIPTION=Default', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_ID=2', u'POOL_SPM_LVER=277', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'ROLE=Master', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'TYPE=FCP', u'VERSION=2', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', '_SHA_CKSUM=a3ed62b0970fa02db432bafe8169c17849a2719b'] >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:36,759::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:36,760::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:36,761::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:36,761::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:36,762::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:36,763::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:37,416::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:37,419::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:37,420::blockSD::145::storage.Metadata.VGTagMetadataRW::(writelines) Updating metadata adding=MDT_POOL_SPM_LVER=277, MDT__SHA_CKSUM=a3ed62b0970fa02db432bafe8169c17849a2719b, MDT_POOL_SPM_ID=2 removing=MDT_POOL_SPM_ID=-1, MDT_POOL_SPM_LVER=276, MDT__SHA_CKSUM=d1ce634f167d7f0ddd58cf2ebbdf434d45f6ef67 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:37,421::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --deltag MDT_POOL_SPM_LVER=276 --deltag MDT_POOL_SPM_ID=-1 --deltag MDT__SHA_CKSUM=d1ce634f167d7f0ddd58cf2ebbdf434d45f6ef67 --addtag MDT__SHA_CKSUM=a3ed62b0970fa02db432bafe8169c17849a2719b --addtag MDT_POOL_SPM_ID=2 --addtag MDT_POOL_SPM_LVER=277 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15910483::DEBUG::2013-11-19 21:14:37,645::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910483::INFO::2013-11-19 21:14:37,648::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910483::DEBUG::2013-11-19 21:14:37,649::task::495::TaskManager.Task::(_debug) Task 03a78abe-6dc0-40fe-b840-bb7b1fae9ef7: moving from state init -> state preparing >Thread-15910483::DEBUG::2013-11-19 21:14:37,649::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910483::DEBUG::2013-11-19 21:14:37,650::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910483::DEBUG::2013-11-19 21:14:37,651::task::495::TaskManager.Task::(_debug) Task 03a78abe-6dc0-40fe-b840-bb7b1fae9ef7: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910483::DEBUG::2013-11-19 21:14:37,651::task::495::TaskManager.Task::(_debug) Task 03a78abe-6dc0-40fe-b840-bb7b1fae9ef7: moving from state preparing -> state finished >Thread-15910483::DEBUG::2013-11-19 21:14:37,652::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910483::DEBUG::2013-11-19 21:14:37,652::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910483::DEBUG::2013-11-19 21:14:37,653::task::495::TaskManager.Task::(_debug) Task 03a78abe-6dc0-40fe-b840-bb7b1fae9ef7: ref 0 aborting False >Thread-15910483::INFO::2013-11-19 21:14:37,654::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,127::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,128::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,129::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,130::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,131::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`67122734-677a-4233-ba17-e4c9cde2c78c`::Request was made in '/usr/share/vdsm/storage/spm.py' line '330' at '_upgradePool' >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,132::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,133::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,133::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`67122734-677a-4233-ba17-e4c9cde2c78c`::Granted request >88cc2f68-e802-4615-bced-c7a8c4aefa33::INFO::2013-11-19 21:14:38,134::spm::336::Storage.SPM::(_upgradePool) Trying to upgrade master domain `6b892067-ac7e-4039-a7be-5b7387525b16` >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,135::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ce6bff3b-5f97-48a2-b4f1-b2b55f16516b`::Request was made in '/usr/share/vdsm/storage/spm.py' line '337' at '_upgradePool' >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,135::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'exclusive' >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,136::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'exclusive' (1 active user) >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,136::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ce6bff3b-5f97-48a2-b4f1-b2b55f16516b`::Granted request >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,138::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,138::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,139::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,139::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,140::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,141::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15910484::DEBUG::2013-11-19 21:14:38,667::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910484::INFO::2013-11-19 21:14:38,669::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910484::DEBUG::2013-11-19 21:14:38,670::task::495::TaskManager.Task::(_debug) Task d25b5b9b-4767-4c17-ba1a-a942fdcd80de: moving from state init -> state preparing >Thread-15910484::DEBUG::2013-11-19 21:14:38,670::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910484::DEBUG::2013-11-19 21:14:38,671::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910484::DEBUG::2013-11-19 21:14:38,671::task::495::TaskManager.Task::(_debug) Task d25b5b9b-4767-4c17-ba1a-a942fdcd80de: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910484::DEBUG::2013-11-19 21:14:38,672::task::495::TaskManager.Task::(_debug) Task d25b5b9b-4767-4c17-ba1a-a942fdcd80de: moving from state preparing -> state finished >Thread-15910484::DEBUG::2013-11-19 21:14:38,673::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910484::DEBUG::2013-11-19 21:14:38,673::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910484::DEBUG::2013-11-19 21:14:38,674::task::495::TaskManager.Task::(_debug) Task d25b5b9b-4767-4c17-ba1a-a942fdcd80de: ref 0 aborting False >Thread-15910484::INFO::2013-11-19 21:14:38,674::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,790::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,793::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,794::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'_SHA_CKSUM=a3ed62b0970fa02db432bafe8169c17849a2719b', u'POOL_SPM_ID=2', u'POOL_SPM_LVER=277'] >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,795::blockSD::625::Storage.StorageDomain::(upgrade) Trying to upgrade domain `6b892067-ac7e-4039-a7be-5b7387525b16` from version 2 to version 2 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,795::blockSD::631::Storage.StorageDomain::(upgrade) No need to upgrade domain `6b892067-ac7e-4039-a7be-5b7387525b16`, leaving unchanged >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,796::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,797::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,797::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,798::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,798::spm::340::Storage.SPM::(_upgradePool) Marking all domains for upgrade >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,799::spm::347::Storage.SPM::(_upgradePool) Registering with state change event >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,800::spm::349::Storage.SPM::(_upgradePool) Running initial domain upgrade threads >Thread-15910485::DEBUG::2013-11-19 21:14:38,800::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 45cdff8b-89c0-4926-a8c7-80354064bfa6 >Thread-15910485::DEBUG::2013-11-19 21:14:38,805::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`b8588e7b-60b6-4f67-af89-a20223c7b28d`::Request was made in '/usr/share/vdsm/storage/spm.py' line '376' at '_upgradePoolDomain' >Thread-15910486::DEBUG::2013-11-19 21:14:38,806::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 772472cb-2429-466d-96cd-00b3301ef78a >Thread-15910485::DEBUG::2013-11-19 21:14:38,807::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' for lock type 'exclusive' >Thread-15910486::DEBUG::2013-11-19 21:14:38,808::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`8fc8ef6f-f04a-48c0-bfb3-64a869b65da2`::Request was made in '/usr/share/vdsm/storage/spm.py' line '376' at '_upgradePoolDomain' >Thread-15910487::DEBUG::2013-11-19 21:14:38,809::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 5d7b5178-3790-472c-a13b-b481eb27a61e >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,809::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910485::DEBUG::2013-11-19 21:14:38,810::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' is free. Now locking as 'exclusive' (1 active user) >Thread-15910486::DEBUG::2013-11-19 21:14:38,811::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'exclusive' >Thread-15910487::ERROR::2013-11-19 21:14:38,814::spm::373::Storage.SPM::(_upgradePoolDomain) Error while preparing domain `5d7b5178-3790-472c-a13b-b481eb27a61e` upgrade >Traceback (most recent call last): > File "/usr/share/vdsm/storage/spm.py", line 369, in _upgradePoolDomain >IndexError: list index out of range >Thread-15910485::DEBUG::2013-11-19 21:14:38,815::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`b8588e7b-60b6-4f67-af89-a20223c7b28d`::Granted request >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,818::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910485::DEBUG::2013-11-19 21:14:38,819::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`176bc5dc-18d7-4e40-9250-e1b5ac7d93d2`::Request was made in '/usr/share/vdsm/storage/spm.py' line '377' at '_upgradePoolDomain' >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,820::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910485::DEBUG::2013-11-19 21:14:38,820::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' for lock type 'exclusive' >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,821::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910486::DEBUG::2013-11-19 21:14:38,822::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'exclusive' (1 active user) >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:38,823::blockSD::791::Storage.Misc.excCmd::(mountMaster) '/usr/bin/sudo -n /sbin/e2fsck -p /dev/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15910486::DEBUG::2013-11-19 21:14:38,824::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`8fc8ef6f-f04a-48c0-bfb3-64a869b65da2`::Granted request >Thread-15910485::DEBUG::2013-11-19 21:14:38,825::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' is free. Now locking as 'exclusive' (1 active user) >Thread-15910486::DEBUG::2013-11-19 21:14:39,033::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`9efd327d-86d6-446b-9413-3a93bd99b0de`::Request was made in '/usr/share/vdsm/storage/spm.py' line '377' at '_upgradePoolDomain' >Thread-15910486::DEBUG::2013-11-19 21:14:39,036::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'exclusive' >Thread-15910485::DEBUG::2013-11-19 21:14:39,036::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`176bc5dc-18d7-4e40-9250-e1b5ac7d93d2`::Granted request >Thread-15910486::DEBUG::2013-11-19 21:14:39,040::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'exclusive' (1 active user) >Thread-15910486::DEBUG::2013-11-19 21:14:39,042::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`9efd327d-86d6-446b-9413-3a93bd99b0de`::Granted request >Thread-15910485::DEBUG::2013-11-19 21:14:39,042::spm::385::Storage.SPM::(_upgradePoolDomain) Domain `45cdff8b-89c0-4926-a8c7-80354064bfa6` is not a data domain it is an 2 domain, not upgrading >Thread-15910485::DEBUG::2013-11-19 21:14:39,052::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' >Thread-15910485::DEBUG::2013-11-19 21:14:39,054::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' (0 active users) >Thread-15910486::DEBUG::2013-11-19 21:14:39,055::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15910485::DEBUG::2013-11-19 21:14:39,055::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' is free, finding out if anyone is waiting for it. >Thread-15910486::DEBUG::2013-11-19 21:14:39,057::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15910485::DEBUG::2013-11-19 21:14:39,060::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6', Clearing records. >Thread-15910486::DEBUG::2013-11-19 21:14:39,066::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15910486::DEBUG::2013-11-19 21:14:39,069::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15910485::DEBUG::2013-11-19 21:14:39,069::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' >Thread-15910486::DEBUG::2013-11-19 21:14:39,070::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15910485::DEBUG::2013-11-19 21:14:39,071::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' (0 active users) >Thread-15910486::DEBUG::2013-11-19 21:14:39,073::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15910485::DEBUG::2013-11-19 21:14:39,074::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' is free, finding out if anyone is waiting for it. >Thread-15910485::DEBUG::2013-11-19 21:14:39,079::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6', Clearing records. >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:39,395::blockSD::791::Storage.Misc.excCmd::(mountMaster) SUCCESS: <err> = ''; <rc> = 0 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:39,396::blockSD::816::Storage.Misc.excCmd::(mountMaster) '/usr/bin/sudo -n /sbin/tune2fs -j /dev/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15910488::DEBUG::2013-11-19 21:14:39,800::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910488::INFO::2013-11-19 21:14:39,802::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910488::DEBUG::2013-11-19 21:14:39,802::task::495::TaskManager.Task::(_debug) Task f15d3920-3338-4349-8a04-900826b13937: moving from state init -> state preparing >Thread-15910488::DEBUG::2013-11-19 21:14:39,803::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910488::DEBUG::2013-11-19 21:14:39,803::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910488::DEBUG::2013-11-19 21:14:39,804::task::495::TaskManager.Task::(_debug) Task f15d3920-3338-4349-8a04-900826b13937: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910488::DEBUG::2013-11-19 21:14:39,804::task::495::TaskManager.Task::(_debug) Task f15d3920-3338-4349-8a04-900826b13937: moving from state preparing -> state finished >Thread-15910488::DEBUG::2013-11-19 21:14:39,804::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910488::DEBUG::2013-11-19 21:14:39,805::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910488::DEBUG::2013-11-19 21:14:39,805::task::495::TaskManager.Task::(_debug) Task f15d3920-3338-4349-8a04-900826b13937: ref 0 aborting False >Thread-15910488::INFO::2013-11-19 21:14:39,805::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:39,935::blockSD::816::Storage.Misc.excCmd::(mountMaster) FAILED: <err> = 'The filesystem already has a journal.\n'; <rc> = 1 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:39,938::fileUtils::83::Storage.Misc.excCmd::(mount) '/usr/bin/sudo -n /bin/mount /dev/6b892067-ac7e-4039-a7be-5b7387525b16/master /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15910486::DEBUG::2013-11-19 21:14:40,230::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15910486::DEBUG::2013-11-19 21:14:40,233::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15910486::DEBUG::2013-11-19 21:14:40,234::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'SDUUID=772472cb-2429-466d-96cd-00b3301ef78a', u'VERSION=2', u'LOCKPOLICY=', u'VGUUID=XTb294-NYv6-dVpO-bB1n-4f9P-d0GL-dhd2DI', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'LEASETIMESEC=5', u'LOCKRENEWALINTERVALSEC=5', u'ROLE=Regular', u'PHYBLKSIZE=512', u'DESCRIPTION=Database', u'IOOPTIMEOUTSEC=1', u'CLASS=Data', u'TYPE=FCP', u'PV0=pv:3600c0ff0001520c7414f605001000000,uuid:NujM33-vNiD-sBHY-mrPo-EQ2f-1qsz-HXfn0q,pestart:0,pecount:8929,mapoffset:0', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'MASTER_VERSION=0', u'_SHA_CKSUM=ef8840d9fdef24516a56fd970bed1960d2b1ae92'] >Thread-15910486::DEBUG::2013-11-19 21:14:40,235::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15910486::DEBUG::2013-11-19 21:14:40,236::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15910486::DEBUG::2013-11-19 21:14:40,236::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15910486::DEBUG::2013-11-19 21:14:40,236::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15910486::DEBUG::2013-11-19 21:14:40,237::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15910486::DEBUG::2013-11-19 21:14:40,237::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15910486::DEBUG::2013-11-19 21:14:40,238::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15910486::DEBUG::2013-11-19 21:14:40,238::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,415::fileUtils::83::Storage.Misc.excCmd::(mount) SUCCESS: <err> = ''; <rc> = 0 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,416::blockSD::833::Storage.Misc.excCmd::(mountMaster) '/usr/bin/sudo -n /bin/chown vdsm:kvm /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15910489::DEBUG::2013-11-19 21:14:40,814::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910489::INFO::2013-11-19 21:14:40,816::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910489::DEBUG::2013-11-19 21:14:40,817::task::495::TaskManager.Task::(_debug) Task 19474d41-a570-4cc4-af58-f9ef6607c694: moving from state init -> state preparing >Thread-15910489::DEBUG::2013-11-19 21:14:40,818::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910489::DEBUG::2013-11-19 21:14:40,818::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910489::DEBUG::2013-11-19 21:14:40,819::task::495::TaskManager.Task::(_debug) Task 19474d41-a570-4cc4-af58-f9ef6607c694: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910489::DEBUG::2013-11-19 21:14:40,820::task::495::TaskManager.Task::(_debug) Task 19474d41-a570-4cc4-af58-f9ef6607c694: moving from state preparing -> state finished >Thread-15910489::DEBUG::2013-11-19 21:14:40,820::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910489::DEBUG::2013-11-19 21:14:40,821::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910489::DEBUG::2013-11-19 21:14:40,821::task::495::TaskManager.Task::(_debug) Task 19474d41-a570-4cc4-af58-f9ef6607c694: ref 0 aborting False >Thread-15910489::INFO::2013-11-19 21:14:40,822::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,843::blockSD::833::Storage.Misc.excCmd::(mountMaster) SUCCESS: <err> = ''; <rc> = 0 >88cc2f68-e802-4615-bced-c7a8c4aefa33::WARNING::2013-11-19 21:14:40,871::spm::490::Storage.SPM::(start) Backup domain validation failed >Traceback (most recent call last): > File "/usr/share/vdsm/storage/spm.py", line 488, in start > File "/usr/share/vdsm/storage/sp.py", line 1208, in checkBackupDomain > File "/usr/share/vdsm/storage/sd.py", line 666, in isBackup > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,872::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,873::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,873::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,874::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,874::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,875::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,875::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,876::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,876::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,877::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,877::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,878::persistentDict::146::Storage.PersistentDict::(transaction) Flushing changes >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,878::persistentDict::265::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['MASTER_VERSION=0', 'ROLE=Regular', '_SHA_CKSUM=1deb73127f89496e2cb17e51bbfae1df6c1879d1'] >88cc2f68-e802-4615-bced-c7a8c4aefa33::WARNING::2013-11-19 21:14:40,880::persistentDict::150::Storage.PersistentDict::(transaction) Error in transaction, rolling back changes >Traceback (most recent call last): > File "/usr/share/vdsm/storage/persistentDict.py", line 147, in transaction > File "/usr/share/vdsm/storage/persistentDict.py", line 266, in flush > File "/usr/share/vdsm/storage/fileSD.py", line 58, in writelines > File "/usr/share/vdsm/storage/processPool.py", line 33, in wrapper > File "/usr/share/vdsm/storage/processPool.py", line 44, in runExternally > File "/usr/share/vdsm/storage/processPool.py", line 127, in runExternally >IOError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata.new' >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,881::persistentDict::265::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['_SHA_CKSUM=da39a3ee5e6b4b0d3255bfef95601890afd80709'] >88cc2f68-e802-4615-bced-c7a8c4aefa33::ERROR::2013-11-19 21:14:40,882::spm::526::Storage.SPM::(start) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/spm.py", line 493, in start > File "/usr/share/vdsm/storage/sp.py", line 637, in _regulariseOutdatedMasters > File "/usr/share/vdsm/storage/sp.py", line 631, in _setRegularSD > File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ > File "/usr/share/vdsm/storage/persistentDict.py", line 152, in transaction > File "/usr/share/vdsm/storage/persistentDict.py", line 266, in flush > File "/usr/share/vdsm/storage/fileSD.py", line 58, in writelines > File "/usr/share/vdsm/storage/processPool.py", line 33, in wrapper > File "/usr/share/vdsm/storage/processPool.py", line 44, in runExternally > File "/usr/share/vdsm/storage/processPool.py", line 127, in runExternally >IOError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata.new' >88cc2f68-e802-4615-bced-c7a8c4aefa33::ERROR::2013-11-19 21:14:40,883::spm::527::Storage.SPM::(start) failed: [Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata.new' >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,884::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`61d8fc27-466f-443d-aa88-41b16bda908b`::Request was made in '/usr/share/vdsm/storage/spm.py' line '584' at '_stop' >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,884::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,885::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,885::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`61d8fc27-466f-443d-aa88-41b16bda908b`::Granted request >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,887::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`1908c5be-2e2c-4523-9f6c-f2705c921ff4`::Request was made in '/usr/share/vdsm/storage/spm.py' line '601' at '_stop' >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,887::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' for lock type 'exclusive' >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,888::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' is free. Now locking as 'exclusive' (1 active user) >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,889::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`1908c5be-2e2c-4523-9f6c-f2705c921ff4`::Granted request >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,889::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,890::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' (0 active users) >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,891::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' is free, finding out if anyone is waiting for it. >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,891::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e', Clearing records. >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,892::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,893::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,893::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,893::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,894::taskManager::118::TaskManager::(unloadTasks) Entry. >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,895::taskManager::122::TaskManager::(unloadTasks) Return >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:40,897::spm::195::Storage.SPM::(__cleanupMasterMount) unmounting /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:41,011::fileUtils::112::Storage.Misc.excCmd::(umount) '/usr/bin/sudo -n /bin/umount -f /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:41,392::fileUtils::112::Storage.Misc.excCmd::(umount) SUCCESS: <err> = ''; <rc> = 0 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:41,396::spm::290::Storage.SPM::(__cleanupSPM) cleaning up SPM: c4e5889a-348d-11e2-83fe-9f72369eab30 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:41,396::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:41,397::persistentDict::146::Storage.PersistentDict::(transaction) Flushing changes >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:41,398::persistentDict::265::Storage.PersistentDict::(flush) about to write lines (VGTagMetadataRW)=[u'CLASS=Data', u'DESCRIPTION=OS', u'IOOPTIMEOUTSEC=10', u'LEASERETRIES=3', u'LEASETIMESEC=60', u'LOCKPOLICY=', u'LOCKRENEWALINTERVALSEC=5', u'LOGBLKSIZE=512', u'MASTER_VERSION=1', u'PHYBLKSIZE=512', u'POOL_DESCRIPTION=Default', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_ID=-1', u'POOL_SPM_LVER=277', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'ROLE=Master', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'TYPE=FCP', u'VERSION=2', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', '_SHA_CKSUM=44c47fe1672a8deeb5c0f134ce9469f0ec72525b'] >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:41,399::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:41,400::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:41,400::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:41,401::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:41,402::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:41,403::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15910490::DEBUG::2013-11-19 21:14:41,830::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910490::INFO::2013-11-19 21:14:41,831::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910490::DEBUG::2013-11-19 21:14:41,832::task::495::TaskManager.Task::(_debug) Task 4fd4c0ae-07da-4e8d-aeae-03d2a80a80b0: moving from state init -> state preparing >Thread-15910490::DEBUG::2013-11-19 21:14:41,833::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910490::DEBUG::2013-11-19 21:14:41,833::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910490::DEBUG::2013-11-19 21:14:41,834::task::495::TaskManager.Task::(_debug) Task 4fd4c0ae-07da-4e8d-aeae-03d2a80a80b0: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910490::DEBUG::2013-11-19 21:14:41,835::task::495::TaskManager.Task::(_debug) Task 4fd4c0ae-07da-4e8d-aeae-03d2a80a80b0: moving from state preparing -> state finished >Thread-15910490::DEBUG::2013-11-19 21:14:41,835::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910490::DEBUG::2013-11-19 21:14:41,836::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910490::DEBUG::2013-11-19 21:14:41,836::task::495::TaskManager.Task::(_debug) Task 4fd4c0ae-07da-4e8d-aeae-03d2a80a80b0: ref 0 aborting False >Thread-15910490::INFO::2013-11-19 21:14:41,837::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:42,075::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:42,078::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:42,079::blockSD::145::storage.Metadata.VGTagMetadataRW::(writelines) Updating metadata adding=MDT__SHA_CKSUM=44c47fe1672a8deeb5c0f134ce9469f0ec72525b, MDT_POOL_SPM_ID=-1 removing=MDT__SHA_CKSUM=a3ed62b0970fa02db432bafe8169c17849a2719b, MDT_POOL_SPM_ID=2 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:42,080::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --deltag MDT__SHA_CKSUM=a3ed62b0970fa02db432bafe8169c17849a2719b --deltag MDT_POOL_SPM_ID=2 --addtag MDT_POOL_SPM_ID=-1 --addtag MDT__SHA_CKSUM=44c47fe1672a8deeb5c0f134ce9469f0ec72525b 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:42,776::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:42,778::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:42,779::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:42,781::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >88cc2f68-e802-4615-bced-c7a8c4aefa33::INFO::2013-11-19 21:14:42,782::safelease::83::ClusterLock::(release) Releasing cluster lock for domain 6b892067-ac7e-4039-a7be-5b7387525b16 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:42,784::safelease::84::Storage.Misc.excCmd::(release) '/usr/libexec/vdsm/spmstop.sh 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd /usr/libexec/vdsm/) >Thread-15910491::DEBUG::2013-11-19 21:14:43,017::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910491::INFO::2013-11-19 21:14:43,020::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910491::DEBUG::2013-11-19 21:14:43,020::task::495::TaskManager.Task::(_debug) Task e35dce8b-8aa7-42a4-98e4-7d7f32c4dac9: moving from state init -> state preparing >Thread-15910491::DEBUG::2013-11-19 21:14:43,021::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910491::DEBUG::2013-11-19 21:14:43,022::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910491::DEBUG::2013-11-19 21:14:43,023::task::495::TaskManager.Task::(_debug) Task e35dce8b-8aa7-42a4-98e4-7d7f32c4dac9: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910491::DEBUG::2013-11-19 21:14:43,023::task::495::TaskManager.Task::(_debug) Task e35dce8b-8aa7-42a4-98e4-7d7f32c4dac9: moving from state preparing -> state finished >Thread-15910491::DEBUG::2013-11-19 21:14:43,024::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910491::DEBUG::2013-11-19 21:14:43,025::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910491::DEBUG::2013-11-19 21:14:43,025::task::495::TaskManager.Task::(_debug) Task e35dce8b-8aa7-42a4-98e4-7d7f32c4dac9: ref 0 aborting False >Thread-15910491::INFO::2013-11-19 21:14:43,026::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-1024767::ERROR::2013-11-19 21:14:43,403::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910492::DEBUG::2013-11-19 21:14:44,032::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910492::INFO::2013-11-19 21:14:44,034::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910492::DEBUG::2013-11-19 21:14:44,034::task::495::TaskManager.Task::(_debug) Task fd5fbdf2-c480-4d37-97ee-c42712c1c6f9: moving from state init -> state preparing >Thread-15910492::DEBUG::2013-11-19 21:14:44,035::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910492::DEBUG::2013-11-19 21:14:44,035::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910492::DEBUG::2013-11-19 21:14:44,036::task::495::TaskManager.Task::(_debug) Task fd5fbdf2-c480-4d37-97ee-c42712c1c6f9: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910492::DEBUG::2013-11-19 21:14:44,037::task::495::TaskManager.Task::(_debug) Task fd5fbdf2-c480-4d37-97ee-c42712c1c6f9: moving from state preparing -> state finished >Thread-15910492::DEBUG::2013-11-19 21:14:44,037::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910492::DEBUG::2013-11-19 21:14:44,038::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910492::DEBUG::2013-11-19 21:14:44,038::task::495::TaskManager.Task::(_debug) Task fd5fbdf2-c480-4d37-97ee-c42712c1c6f9: ref 0 aborting False >Thread-15910492::INFO::2013-11-19 21:14:44,039::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,300::safelease::84::Storage.Misc.excCmd::(release) SUCCESS: <err> = ''; <rc> = 0 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,301::safelease::87::ClusterLock::(release) Cluster lock released successfully >88cc2f68-e802-4615-bced-c7a8c4aefa33::ERROR::2013-11-19 21:14:44,301::task::868::TaskManager.Task::(_setError) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 876, in _run > File "/usr/share/vdsm/storage/task.py", line 303, in run > File "/usr/share/vdsm/storage/spm.py", line 493, in start > File "/usr/share/vdsm/storage/sp.py", line 637, in _regulariseOutdatedMasters > File "/usr/share/vdsm/storage/sp.py", line 631, in _setRegularSD > File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ > File "/usr/share/vdsm/storage/persistentDict.py", line 152, in transaction > File "/usr/share/vdsm/storage/persistentDict.py", line 266, in flush > File "/usr/share/vdsm/storage/fileSD.py", line 58, in writelines > File "/usr/share/vdsm/storage/processPool.py", line 33, in wrapper > File "/usr/share/vdsm/storage/processPool.py", line 44, in runExternally > File "/usr/share/vdsm/storage/processPool.py", line 127, in runExternally >IOError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata.new' >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,302::task::495::TaskManager.Task::(_debug) Task 88cc2f68-e802-4615-bced-c7a8c4aefa33: Task._run: 88cc2f68-e802-4615-bced-c7a8c4aefa33 () {} failed - stopping task >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,303::task::495::TaskManager.Task::(_debug) Task 88cc2f68-e802-4615-bced-c7a8c4aefa33: stopping in state running (force False) >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,304::task::495::TaskManager.Task::(_debug) Task 88cc2f68-e802-4615-bced-c7a8c4aefa33: ref 1 aborting True >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,304::task::918::TaskManager.Task::(_runJobs) aborting: Task is aborted: "[Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata.new'" - code 100 >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,305::task::495::TaskManager.Task::(_debug) Task 88cc2f68-e802-4615-bced-c7a8c4aefa33: ref 0 aborting True >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,305::task::495::TaskManager.Task::(_debug) Task 88cc2f68-e802-4615-bced-c7a8c4aefa33: Task._doAbort: force False >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,306::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,306::task::495::TaskManager.Task::(_debug) Task 88cc2f68-e802-4615-bced-c7a8c4aefa33: moving from state running -> state aborting >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,307::task::495::TaskManager.Task::(_debug) Task 88cc2f68-e802-4615-bced-c7a8c4aefa33: _aborting: recover policy auto >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,307::task::495::TaskManager.Task::(_debug) Task 88cc2f68-e802-4615-bced-c7a8c4aefa33: moving from state aborting -> state racquiring >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,308::task::495::TaskManager.Task::(_debug) Task 88cc2f68-e802-4615-bced-c7a8c4aefa33: moving from state racquiring -> state recovering >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,308::task::495::TaskManager.Task::(_debug) Task 88cc2f68-e802-4615-bced-c7a8c4aefa33: _recover >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,309::task::495::TaskManager.Task::(_debug) Task 88cc2f68-e802-4615-bced-c7a8c4aefa33: running recovery None >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,309::task::495::TaskManager.Task::(_debug) Task 88cc2f68-e802-4615-bced-c7a8c4aefa33: Recover Done: state recovering >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,310::task::495::TaskManager.Task::(_debug) Task 88cc2f68-e802-4615-bced-c7a8c4aefa33: moving from state recovering -> state recovered >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,310::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,311::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,312::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,312::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,313::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,313::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >88cc2f68-e802-4615-bced-c7a8c4aefa33::DEBUG::2013-11-19 21:14:44,314::threadPool::47::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 >Thread-15910493::DEBUG::2013-11-19 21:14:45,045::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910493::INFO::2013-11-19 21:14:45,045::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910493::DEBUG::2013-11-19 21:14:45,046::task::495::TaskManager.Task::(_debug) Task 237abb41-7fb6-40d5-97c4-c7273261f8ee: moving from state init -> state preparing >Thread-15910493::DEBUG::2013-11-19 21:14:45,047::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910493::DEBUG::2013-11-19 21:14:45,047::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 100, 'message': "[Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata.new'", 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'} >Thread-15910493::DEBUG::2013-11-19 21:14:45,048::task::495::TaskManager.Task::(_debug) Task 237abb41-7fb6-40d5-97c4-c7273261f8ee: finished: {'taskStatus': {'code': 100, 'message': "[Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata.new'", 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910493::DEBUG::2013-11-19 21:14:45,048::task::495::TaskManager.Task::(_debug) Task 237abb41-7fb6-40d5-97c4-c7273261f8ee: moving from state preparing -> state finished >Thread-15910493::DEBUG::2013-11-19 21:14:45,049::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910493::DEBUG::2013-11-19 21:14:45,049::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910493::DEBUG::2013-11-19 21:14:45,050::task::495::TaskManager.Task::(_debug) Task 237abb41-7fb6-40d5-97c4-c7273261f8ee: ref 0 aborting False >Thread-15910493::INFO::2013-11-19 21:14:45,051::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 100, 'message': "[Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata.new'", 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '88cc2f68-e802-4615-bced-c7a8c4aefa33'}} >Thread-15910494::DEBUG::2013-11-19 21:14:45,055::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910494::INFO::2013-11-19 21:14:45,056::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910494::DEBUG::2013-11-19 21:14:45,057::task::495::TaskManager.Task::(_debug) Task 1ded1f08-bbb7-48db-b736-c5bc95488ef9: moving from state init -> state preparing >Thread-15910494::DEBUG::2013-11-19 21:14:45,057::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=Free spmLver=277 spmId=-1 >Thread-15910494::DEBUG::2013-11-19 21:14:45,058::task::495::TaskManager.Task::(_debug) Task 1ded1f08-bbb7-48db-b736-c5bc95488ef9: finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 277}} >Thread-15910494::DEBUG::2013-11-19 21:14:45,059::task::495::TaskManager.Task::(_debug) Task 1ded1f08-bbb7-48db-b736-c5bc95488ef9: moving from state preparing -> state finished >Thread-15910494::DEBUG::2013-11-19 21:14:45,059::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910494::DEBUG::2013-11-19 21:14:45,060::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910494::DEBUG::2013-11-19 21:14:45,060::task::495::TaskManager.Task::(_debug) Task 1ded1f08-bbb7-48db-b736-c5bc95488ef9: ref 0 aborting False >Thread-15910494::INFO::2013-11-19 21:14:45,061::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 277}} >Thread-15910495::DEBUG::2013-11-19 21:14:45,066::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910495::INFO::2013-11-19 21:14:45,067::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: clearTask, args: ( taskID=88cc2f68-e802-4615-bced-c7a8c4aefa33) >Thread-15910495::DEBUG::2013-11-19 21:14:45,068::task::495::TaskManager.Task::(_debug) Task 7cf64625-d991-4e01-b46a-2377c6bb7ddf: moving from state init -> state preparing >Thread-15910495::DEBUG::2013-11-19 21:14:45,068::taskManager::146::TaskManager::(clearTask) Entry. taskID: 88cc2f68-e802-4615-bced-c7a8c4aefa33 >Thread-15910495::DEBUG::2013-11-19 21:14:45,069::taskManager::151::TaskManager::(clearTask) Return. >Thread-15910495::DEBUG::2013-11-19 21:14:45,069::task::495::TaskManager.Task::(_debug) Task 7cf64625-d991-4e01-b46a-2377c6bb7ddf: finished: None >Thread-15910495::DEBUG::2013-11-19 21:14:45,070::task::495::TaskManager.Task::(_debug) Task 7cf64625-d991-4e01-b46a-2377c6bb7ddf: moving from state preparing -> state finished >Thread-15910495::DEBUG::2013-11-19 21:14:45,070::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910495::DEBUG::2013-11-19 21:14:45,071::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910495::DEBUG::2013-11-19 21:14:45,071::task::495::TaskManager.Task::(_debug) Task 7cf64625-d991-4e01-b46a-2377c6bb7ddf: ref 0 aborting False >Thread-15910495::INFO::2013-11-19 21:14:45,072::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: clearTask, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15910496::DEBUG::2013-11-19 21:14:45,161::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910496::INFO::2013-11-19 21:14:45,162::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910496::DEBUG::2013-11-19 21:14:45,162::task::495::TaskManager.Task::(_debug) Task 9b1e06af-b390-42fd-a143-377c9ba478b5: moving from state init -> state preparing >Thread-15910496::DEBUG::2013-11-19 21:14:45,163::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=Free spmLver=277 spmId=-1 >Thread-15910496::DEBUG::2013-11-19 21:14:45,163::task::495::TaskManager.Task::(_debug) Task 9b1e06af-b390-42fd-a143-377c9ba478b5: finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 277}} >Thread-15910496::DEBUG::2013-11-19 21:14:45,164::task::495::TaskManager.Task::(_debug) Task 9b1e06af-b390-42fd-a143-377c9ba478b5: moving from state preparing -> state finished >Thread-15910496::DEBUG::2013-11-19 21:14:45,164::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910496::DEBUG::2013-11-19 21:14:45,165::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910496::DEBUG::2013-11-19 21:14:45,165::task::495::TaskManager.Task::(_debug) Task 9b1e06af-b390-42fd-a143-377c9ba478b5: ref 0 aborting False >Thread-15910496::INFO::2013-11-19 21:14:45,166::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 277}} >Thread-15910497::DEBUG::2013-11-19 21:14:45,189::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910497::INFO::2013-11-19 21:14:45,190::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: spmStart, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 prevID=-1 prevLVER=277 recoveryMode=0 scsiFencing=false maxHostID=250 domVersion=2) >Thread-15910497::DEBUG::2013-11-19 21:14:45,190::task::495::TaskManager.Task::(_debug) Task 21b4b46d-58c8-40ca-8f03-a8063e50f1b9: moving from state init -> state preparing >Thread-15910497::DEBUG::2013-11-19 21:14:45,191::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`979b0547-8ba9-4f75-9403-87c9442ca01f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '330' at 'public_spmStart' >Thread-15910497::DEBUG::2013-11-19 21:14:45,192::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >Thread-15910497::DEBUG::2013-11-19 21:14:45,193::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >Thread-15910497::DEBUG::2013-11-19 21:14:45,193::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`979b0547-8ba9-4f75-9403-87c9442ca01f`::Granted request >Thread-15910497::DEBUG::2013-11-19 21:14:45,194::task::495::TaskManager.Task::(_debug) Task 21b4b46d-58c8-40ca-8f03-a8063e50f1b9: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (exclusive) >Thread-15910497::DEBUG::2013-11-19 21:14:45,195::task::495::TaskManager.Task::(_debug) Task 21b4b46d-58c8-40ca-8f03-a8063e50f1b9: ref 1 aborting False >Thread-15910497::DEBUG::2013-11-19 21:14:45,195::task::495::TaskManager.Task::(_debug) Task 21b4b46d-58c8-40ca-8f03-a8063e50f1b9: Prepare: 1 jobs exist, move to acquiring >Thread-15910497::DEBUG::2013-11-19 21:14:45,196::task::495::TaskManager.Task::(_debug) Task 21b4b46d-58c8-40ca-8f03-a8063e50f1b9: moving from state preparing -> state acquiring >Thread-15910497::DEBUG::2013-11-19 21:14:45,196::task::495::TaskManager.Task::(_debug) Task 21b4b46d-58c8-40ca-8f03-a8063e50f1b9: moving from state acquiring -> state queued >Thread-15910497::DEBUG::2013-11-19 21:14:45,197::taskManager::37::TaskManager::(_queueTask) queueing task: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910497::DEBUG::2013-11-19 21:14:45,197::taskManager::43::TaskManager::(_queueTask) task queued: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >16fdd9ed-6b31-4b43-9b39-df445f20d9f6::DEBUG::2013-11-19 21:14:45,198::threadPool::47::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-15910497::DEBUG::2013-11-19 21:14:45,199::task::1183::TaskManager.Task::(prepare) returning >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:45,199::threadPool::192::Misc.ThreadPool.WorkerThread::(run) Task: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 running: <bound method Task.commit of <storage.task.Task instance at 0x7f487c121ef0>> with: None >Thread-15910497::DEBUG::2013-11-19 21:14:45,200::task::495::TaskManager.Task::(_debug) Task 21b4b46d-58c8-40ca-8f03-a8063e50f1b9: ref 0 aborting False >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:45,200::task::1195::TaskManager.Task::(commit) Commiting task: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910497::INFO::2013-11-19 21:14:45,201::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: spmStart, Return response: {'status': {'message': 'OK', 'code': 0}, 'uuid': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:45,202::task::495::TaskManager.Task::(_debug) Task 21b4b46d-58c8-40ca-8f03-a8063e50f1b9: moving from state queued -> state running >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:45,203::task::907::TaskManager.Task::(_runJobs) Task.run: running job 0: spmStart: <bound method SPM.start of <storage.spm.SPM instance at 0x1bf9f38>> (args: ('c4e5889a-348d-11e2-83fe-9f72369eab30', -1, '277', 0, 'false', 250, 2) kwargs: {}) >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:45,204::task::300::TaskManager.Task::(run) Job.run: running spmStart: <bound method SPM.start of <storage.spm.SPM instance at 0x1bf9f38>> (args: ('c4e5889a-348d-11e2-83fe-9f72369eab30', -1, '277', 0, 'false', 250, 2) kwargs: {}) callback None >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:45,204::spm::433::Storage.SPM::(start) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:45,205::misc::1010::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:45,205::misc::1012::SamplingMethod::(__call__) Got in to sampling method >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:45,206::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:45,207::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:45,207::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:45,208::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:45,209::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:45,210::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 21:14:45,418::lvm::439::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-42838::DEBUG::2013-11-19 21:14:45,420::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:46,002::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:46,006::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=277', u'POOL_SPM_ID=-1', u'_SHA_CKSUM=44c47fe1672a8deeb5c0f134ce9469f0ec72525b'] >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:46,008::misc::1020::SamplingMethod::(__call__) Returning last result >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:46,008::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting... >Thread-42838::DEBUG::2013-11-19 21:14:46,190::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-42838::DEBUG::2013-11-19 21:14:46,194::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:46,196::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:46,199::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:46,201::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:46,202::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:46,204::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:46,206::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15910498::DEBUG::2013-11-19 21:14:46,216::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910498::INFO::2013-11-19 21:14:46,220::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910498::DEBUG::2013-11-19 21:14:46,461::task::495::TaskManager.Task::(_debug) Task 6a853d10-609d-42ec-a8a8-7d4f0ca40aeb: moving from state init -> state preparing >Thread-15910498::DEBUG::2013-11-19 21:14:46,482::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910498::DEBUG::2013-11-19 21:14:46,483::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910498::DEBUG::2013-11-19 21:14:46,484::task::495::TaskManager.Task::(_debug) Task 6a853d10-609d-42ec-a8a8-7d4f0ca40aeb: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910498::DEBUG::2013-11-19 21:14:46,485::task::495::TaskManager.Task::(_debug) Task 6a853d10-609d-42ec-a8a8-7d4f0ca40aeb: moving from state preparing -> state finished >Thread-15910498::DEBUG::2013-11-19 21:14:46,485::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910498::DEBUG::2013-11-19 21:14:46,486::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910498::DEBUG::2013-11-19 21:14:46,486::task::495::TaskManager.Task::(_debug) Task 6a853d10-609d-42ec-a8a8-7d4f0ca40aeb: ref 0 aborting False >Thread-15910498::INFO::2013-11-19 21:14:46,487::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:46,848::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:46,851::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:46,852::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=277', u'POOL_SPM_ID=-1', u'_SHA_CKSUM=44c47fe1672a8deeb5c0f134ce9469f0ec72525b'] >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:46,855::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:46,857::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:46,858::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:46,859::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:46,859::persistentDict::163::Storage.PersistentDict::(__init__) Created a persistant dict with LvMetadataRW backend >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:46,860::blockSD::164::Storage.Misc.excCmd::(readlines) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:47,261::blockSD::164::Storage.Misc.excCmd::(readlines) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000302292 s, 6.8 MB/s\n'; <rc> = 0 >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:47,262::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000302292 s, 6.8 MB/s'], size: 2048 >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:47,263::persistentDict::204::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::WARNING::2013-11-19 21:14:47,264::persistentDict::226::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:47,264::persistentDict::163::Storage.PersistentDict::(__init__) Created a persistant dict with VGTagMetadataRW backend >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:47,265::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:47,266::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:47,266::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:47,267::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:47,268::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:47,269::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15910500::DEBUG::2013-11-19 21:14:47,497::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910500::INFO::2013-11-19 21:14:47,500::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910500::DEBUG::2013-11-19 21:14:47,500::task::495::TaskManager.Task::(_debug) Task c24a96c2-2059-46ad-acbf-636f7fda04ce: moving from state init -> state preparing >Thread-15910500::DEBUG::2013-11-19 21:14:47,501::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910500::DEBUG::2013-11-19 21:14:47,502::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910500::DEBUG::2013-11-19 21:14:47,502::task::495::TaskManager.Task::(_debug) Task c24a96c2-2059-46ad-acbf-636f7fda04ce: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910500::DEBUG::2013-11-19 21:14:47,503::task::495::TaskManager.Task::(_debug) Task c24a96c2-2059-46ad-acbf-636f7fda04ce: moving from state preparing -> state finished >Thread-15910500::DEBUG::2013-11-19 21:14:47,504::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910500::DEBUG::2013-11-19 21:14:47,504::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910500::DEBUG::2013-11-19 21:14:47,505::task::495::TaskManager.Task::(_debug) Task c24a96c2-2059-46ad-acbf-636f7fda04ce: ref 0 aborting False >Thread-15910500::INFO::2013-11-19 21:14:47,506::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:47,931::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:47,938::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:47,939::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=277', u'POOL_SPM_ID=-1', u'_SHA_CKSUM=44c47fe1672a8deeb5c0f134ce9469f0ec72525b'] >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:47,940::safelease::52::ClusterLock::(acquire) Acquiring cluster lock for domain 6b892067-ac7e-4039-a7be-5b7387525b16 >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:14:47,941::safelease::61::Storage.Misc.excCmd::(acquire) '/usr/bin/sudo -n /usr/bin/setsid /usr/bin/ionice -c1 -n0 /bin/su vdsm -s /bin/sh -c "/usr/libexec/vdsm/spmprotect.sh start 6b892067-ac7e-4039-a7be-5b7387525b16 2 5 /dev/6b892067-ac7e-4039-a7be-5b7387525b16/leases 60000 10000 3"' (cwd /usr/libexec/vdsm/) >Thread-15910501::DEBUG::2013-11-19 21:14:48,514::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910501::INFO::2013-11-19 21:14:48,515::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910501::DEBUG::2013-11-19 21:14:48,516::task::495::TaskManager.Task::(_debug) Task 354b09fe-08cd-449b-be62-b5a08279cea7: moving from state init -> state preparing >Thread-15910501::DEBUG::2013-11-19 21:14:48,517::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910501::DEBUG::2013-11-19 21:14:48,517::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910501::DEBUG::2013-11-19 21:14:48,518::task::495::TaskManager.Task::(_debug) Task 354b09fe-08cd-449b-be62-b5a08279cea7: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910501::DEBUG::2013-11-19 21:14:48,519::task::495::TaskManager.Task::(_debug) Task 354b09fe-08cd-449b-be62-b5a08279cea7: moving from state preparing -> state finished >Thread-15910501::DEBUG::2013-11-19 21:14:48,519::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910501::DEBUG::2013-11-19 21:14:48,520::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910501::DEBUG::2013-11-19 21:14:48,520::task::495::TaskManager.Task::(_debug) Task 354b09fe-08cd-449b-be62-b5a08279cea7: ref 0 aborting False >Thread-15910501::INFO::2013-11-19 21:14:48,521::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910503::DEBUG::2013-11-19 21:14:49,527::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910503::INFO::2013-11-19 21:14:49,528::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910503::DEBUG::2013-11-19 21:14:49,528::task::495::TaskManager.Task::(_debug) Task 96dcbc6c-6587-403a-8941-ebae2206d48e: moving from state init -> state preparing >Thread-15910503::DEBUG::2013-11-19 21:14:49,529::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910503::DEBUG::2013-11-19 21:14:49,529::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910503::DEBUG::2013-11-19 21:14:49,530::task::495::TaskManager.Task::(_debug) Task 96dcbc6c-6587-403a-8941-ebae2206d48e: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910503::DEBUG::2013-11-19 21:14:49,531::task::495::TaskManager.Task::(_debug) Task 96dcbc6c-6587-403a-8941-ebae2206d48e: moving from state preparing -> state finished >Thread-15910503::DEBUG::2013-11-19 21:14:49,531::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910503::DEBUG::2013-11-19 21:14:49,532::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910503::DEBUG::2013-11-19 21:14:49,532::task::495::TaskManager.Task::(_debug) Task 96dcbc6c-6587-403a-8941-ebae2206d48e: ref 0 aborting False >Thread-15910503::INFO::2013-11-19 21:14:49,533::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910504::DEBUG::2013-11-19 21:14:50,539::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910504::INFO::2013-11-19 21:14:50,539::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910504::DEBUG::2013-11-19 21:14:50,540::task::495::TaskManager.Task::(_debug) Task eb36fad6-180a-4abc-8f7b-9c3d950412ef: moving from state init -> state preparing >Thread-15910504::DEBUG::2013-11-19 21:14:50,541::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910504::DEBUG::2013-11-19 21:14:50,541::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910504::DEBUG::2013-11-19 21:14:50,542::task::495::TaskManager.Task::(_debug) Task eb36fad6-180a-4abc-8f7b-9c3d950412ef: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910504::DEBUG::2013-11-19 21:14:50,542::task::495::TaskManager.Task::(_debug) Task eb36fad6-180a-4abc-8f7b-9c3d950412ef: moving from state preparing -> state finished >Thread-15910504::DEBUG::2013-11-19 21:14:50,543::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910504::DEBUG::2013-11-19 21:14:50,543::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910504::DEBUG::2013-11-19 21:14:50,544::task::495::TaskManager.Task::(_debug) Task eb36fad6-180a-4abc-8f7b-9c3d950412ef: ref 0 aborting False >Thread-15910504::INFO::2013-11-19 21:14:50,544::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910506::DEBUG::2013-11-19 21:14:51,550::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910506::INFO::2013-11-19 21:14:51,551::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910506::DEBUG::2013-11-19 21:14:51,552::task::495::TaskManager.Task::(_debug) Task 639fed28-b7e5-4827-bd65-ffa22b298a6d: moving from state init -> state preparing >Thread-15910506::DEBUG::2013-11-19 21:14:51,552::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910506::DEBUG::2013-11-19 21:14:51,553::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910506::DEBUG::2013-11-19 21:14:51,554::task::495::TaskManager.Task::(_debug) Task 639fed28-b7e5-4827-bd65-ffa22b298a6d: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910506::DEBUG::2013-11-19 21:14:51,554::task::495::TaskManager.Task::(_debug) Task 639fed28-b7e5-4827-bd65-ffa22b298a6d: moving from state preparing -> state finished >Thread-15910506::DEBUG::2013-11-19 21:14:51,555::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910506::DEBUG::2013-11-19 21:14:51,555::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910506::DEBUG::2013-11-19 21:14:51,556::task::495::TaskManager.Task::(_debug) Task 639fed28-b7e5-4827-bd65-ffa22b298a6d: ref 0 aborting False >Thread-15910506::INFO::2013-11-19 21:14:51,556::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910507::DEBUG::2013-11-19 21:14:52,562::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910507::INFO::2013-11-19 21:14:52,563::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910507::DEBUG::2013-11-19 21:14:52,564::task::495::TaskManager.Task::(_debug) Task db428b00-7540-40b1-a649-ecc2fb7d3ce8: moving from state init -> state preparing >Thread-15910507::DEBUG::2013-11-19 21:14:52,564::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910507::DEBUG::2013-11-19 21:14:52,565::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910507::DEBUG::2013-11-19 21:14:52,565::task::495::TaskManager.Task::(_debug) Task db428b00-7540-40b1-a649-ecc2fb7d3ce8: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910507::DEBUG::2013-11-19 21:14:52,566::task::495::TaskManager.Task::(_debug) Task db428b00-7540-40b1-a649-ecc2fb7d3ce8: moving from state preparing -> state finished >Thread-15910507::DEBUG::2013-11-19 21:14:52,566::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910507::DEBUG::2013-11-19 21:14:52,567::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910507::DEBUG::2013-11-19 21:14:52,568::task::495::TaskManager.Task::(_debug) Task db428b00-7540-40b1-a649-ecc2fb7d3ce8: ref 0 aborting False >Thread-15910507::INFO::2013-11-19 21:14:52,568::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-1024767::ERROR::2013-11-19 21:14:53,497::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910509::DEBUG::2013-11-19 21:14:53,574::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910509::INFO::2013-11-19 21:14:53,575::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910509::DEBUG::2013-11-19 21:14:53,575::task::495::TaskManager.Task::(_debug) Task 01ab3932-9a46-45bf-be00-bf197ed23e32: moving from state init -> state preparing >Thread-15910509::DEBUG::2013-11-19 21:14:53,576::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910509::DEBUG::2013-11-19 21:14:53,576::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910509::DEBUG::2013-11-19 21:14:53,577::task::495::TaskManager.Task::(_debug) Task 01ab3932-9a46-45bf-be00-bf197ed23e32: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910509::DEBUG::2013-11-19 21:14:53,577::task::495::TaskManager.Task::(_debug) Task 01ab3932-9a46-45bf-be00-bf197ed23e32: moving from state preparing -> state finished >Thread-15910509::DEBUG::2013-11-19 21:14:53,578::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910509::DEBUG::2013-11-19 21:14:53,578::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910509::DEBUG::2013-11-19 21:14:53,579::task::495::TaskManager.Task::(_debug) Task 01ab3932-9a46-45bf-be00-bf197ed23e32: ref 0 aborting False >Thread-15910509::INFO::2013-11-19 21:14:53,580::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910510::DEBUG::2013-11-19 21:14:54,585::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910510::INFO::2013-11-19 21:14:54,586::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910510::DEBUG::2013-11-19 21:14:54,587::task::495::TaskManager.Task::(_debug) Task 1b87215e-b204-45de-a131-b8140bd477f0: moving from state init -> state preparing >Thread-15910510::DEBUG::2013-11-19 21:14:54,587::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910510::DEBUG::2013-11-19 21:14:54,588::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910510::DEBUG::2013-11-19 21:14:54,588::task::495::TaskManager.Task::(_debug) Task 1b87215e-b204-45de-a131-b8140bd477f0: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910510::DEBUG::2013-11-19 21:14:54,589::task::495::TaskManager.Task::(_debug) Task 1b87215e-b204-45de-a131-b8140bd477f0: moving from state preparing -> state finished >Thread-15910510::DEBUG::2013-11-19 21:14:54,589::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910510::DEBUG::2013-11-19 21:14:54,590::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910510::DEBUG::2013-11-19 21:14:54,590::task::495::TaskManager.Task::(_debug) Task 1b87215e-b204-45de-a131-b8140bd477f0: ref 0 aborting False >Thread-15910510::INFO::2013-11-19 21:14:54,591::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910511::INFO::2013-11-19 21:14:55,219::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910511::DEBUG::2013-11-19 21:14:55,222::task::495::TaskManager.Task::(_debug) Task 83cb05c0-5d22-449c-801c-f89c983b9794: moving from state init -> state preparing >Thread-15910511::DEBUG::2013-11-19 21:14:55,224::task::495::TaskManager.Task::(_debug) Task 83cb05c0-5d22-449c-801c-f89c983b9794: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000824928283691', 'lastCheck': 1384895686.858032, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00170588493347', 'lastCheck': 1384895688.7861741, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895693.498703, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0021698474884', 'lastCheck': 1384895692.501792, 'valid': True, 'code': 0}} >Thread-15910511::DEBUG::2013-11-19 21:14:55,225::task::495::TaskManager.Task::(_debug) Task 83cb05c0-5d22-449c-801c-f89c983b9794: moving from state preparing -> state finished >Thread-15910511::DEBUG::2013-11-19 21:14:55,226::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910511::DEBUG::2013-11-19 21:14:55,227::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910511::DEBUG::2013-11-19 21:14:55,227::task::495::TaskManager.Task::(_debug) Task 83cb05c0-5d22-449c-801c-f89c983b9794: ref 0 aborting False >Thread-15910511::INFO::2013-11-19 21:14:55,228::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000824928283691', 'lastCheck': 1384895686.858032, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00170588493347', 'lastCheck': 1384895688.7861741, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895693.498703, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0021698474884', 'lastCheck': 1384895692.501792, 'valid': True, 'code': 0}} >Thread-15910513::DEBUG::2013-11-19 21:14:55,598::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910513::INFO::2013-11-19 21:14:55,599::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910513::DEBUG::2013-11-19 21:14:55,600::task::495::TaskManager.Task::(_debug) Task 6559708c-0be9-4fcd-a750-ed411969e554: moving from state init -> state preparing >Thread-15910513::DEBUG::2013-11-19 21:14:55,600::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910513::DEBUG::2013-11-19 21:14:55,601::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910513::DEBUG::2013-11-19 21:14:55,601::task::495::TaskManager.Task::(_debug) Task 6559708c-0be9-4fcd-a750-ed411969e554: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910513::DEBUG::2013-11-19 21:14:55,602::task::495::TaskManager.Task::(_debug) Task 6559708c-0be9-4fcd-a750-ed411969e554: moving from state preparing -> state finished >Thread-15910513::DEBUG::2013-11-19 21:14:55,602::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910513::DEBUG::2013-11-19 21:14:55,603::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910513::DEBUG::2013-11-19 21:14:55,603::task::495::TaskManager.Task::(_debug) Task 6559708c-0be9-4fcd-a750-ed411969e554: ref 0 aborting False >Thread-15910513::INFO::2013-11-19 21:14:55,604::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910514::DEBUG::2013-11-19 21:14:56,610::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910514::INFO::2013-11-19 21:14:56,611::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910514::DEBUG::2013-11-19 21:14:56,611::task::495::TaskManager.Task::(_debug) Task 54bb086e-0d6b-4584-9107-7631194df15f: moving from state init -> state preparing >Thread-15910514::DEBUG::2013-11-19 21:14:56,612::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910514::DEBUG::2013-11-19 21:14:56,612::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910514::DEBUG::2013-11-19 21:14:56,613::task::495::TaskManager.Task::(_debug) Task 54bb086e-0d6b-4584-9107-7631194df15f: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910514::DEBUG::2013-11-19 21:14:56,613::task::495::TaskManager.Task::(_debug) Task 54bb086e-0d6b-4584-9107-7631194df15f: moving from state preparing -> state finished >Thread-15910514::DEBUG::2013-11-19 21:14:56,614::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910514::DEBUG::2013-11-19 21:14:56,614::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910514::DEBUG::2013-11-19 21:14:56,615::task::495::TaskManager.Task::(_debug) Task 54bb086e-0d6b-4584-9107-7631194df15f: ref 0 aborting False >Thread-15910514::INFO::2013-11-19 21:14:56,616::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910516::DEBUG::2013-11-19 21:14:57,621::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910516::INFO::2013-11-19 21:14:57,622::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910516::DEBUG::2013-11-19 21:14:57,623::task::495::TaskManager.Task::(_debug) Task 02038fd4-0929-46bd-8979-dc485ee17628: moving from state init -> state preparing >Thread-15910516::DEBUG::2013-11-19 21:14:57,623::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910516::DEBUG::2013-11-19 21:14:57,624::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910516::DEBUG::2013-11-19 21:14:57,625::task::495::TaskManager.Task::(_debug) Task 02038fd4-0929-46bd-8979-dc485ee17628: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910516::DEBUG::2013-11-19 21:14:57,625::task::495::TaskManager.Task::(_debug) Task 02038fd4-0929-46bd-8979-dc485ee17628: moving from state preparing -> state finished >Thread-15910516::DEBUG::2013-11-19 21:14:57,626::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910516::DEBUG::2013-11-19 21:14:57,626::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910516::DEBUG::2013-11-19 21:14:57,627::task::495::TaskManager.Task::(_debug) Task 02038fd4-0929-46bd-8979-dc485ee17628: ref 0 aborting False >Thread-15910516::INFO::2013-11-19 21:14:57,627::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910517::DEBUG::2013-11-19 21:14:58,633::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910517::INFO::2013-11-19 21:14:58,634::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910517::DEBUG::2013-11-19 21:14:58,643::task::495::TaskManager.Task::(_debug) Task 22ca50db-aed2-478c-8366-b471cd13a14d: moving from state init -> state preparing >Thread-15910517::DEBUG::2013-11-19 21:14:58,644::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910517::DEBUG::2013-11-19 21:14:58,644::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910517::DEBUG::2013-11-19 21:14:58,645::task::495::TaskManager.Task::(_debug) Task 22ca50db-aed2-478c-8366-b471cd13a14d: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910517::DEBUG::2013-11-19 21:14:58,646::task::495::TaskManager.Task::(_debug) Task 22ca50db-aed2-478c-8366-b471cd13a14d: moving from state preparing -> state finished >Thread-15910517::DEBUG::2013-11-19 21:14:58,646::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910517::DEBUG::2013-11-19 21:14:58,647::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910517::DEBUG::2013-11-19 21:14:58,647::task::495::TaskManager.Task::(_debug) Task 22ca50db-aed2-478c-8366-b471cd13a14d: ref 0 aborting False >Thread-15910517::INFO::2013-11-19 21:14:58,648::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910519::DEBUG::2013-11-19 21:14:59,654::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910519::INFO::2013-11-19 21:14:59,654::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910519::DEBUG::2013-11-19 21:14:59,655::task::495::TaskManager.Task::(_debug) Task 2faa9699-8774-43da-97a9-c4e9e81c1d5a: moving from state init -> state preparing >Thread-15910519::DEBUG::2013-11-19 21:14:59,656::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910519::DEBUG::2013-11-19 21:14:59,656::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910519::DEBUG::2013-11-19 21:14:59,657::task::495::TaskManager.Task::(_debug) Task 2faa9699-8774-43da-97a9-c4e9e81c1d5a: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910519::DEBUG::2013-11-19 21:14:59,657::task::495::TaskManager.Task::(_debug) Task 2faa9699-8774-43da-97a9-c4e9e81c1d5a: moving from state preparing -> state finished >Thread-15910519::DEBUG::2013-11-19 21:14:59,658::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910519::DEBUG::2013-11-19 21:14:59,658::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910519::DEBUG::2013-11-19 21:14:59,659::task::495::TaskManager.Task::(_debug) Task 2faa9699-8774-43da-97a9-c4e9e81c1d5a: ref 0 aborting False >Thread-15910519::INFO::2013-11-19 21:14:59,659::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910520::DEBUG::2013-11-19 21:15:00,665::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910520::INFO::2013-11-19 21:15:00,666::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910520::DEBUG::2013-11-19 21:15:00,667::task::495::TaskManager.Task::(_debug) Task dc6bd7a0-b338-4e44-939b-50b173ddda18: moving from state init -> state preparing >Thread-15910520::DEBUG::2013-11-19 21:15:00,667::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910520::DEBUG::2013-11-19 21:15:00,668::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910520::DEBUG::2013-11-19 21:15:00,669::task::495::TaskManager.Task::(_debug) Task dc6bd7a0-b338-4e44-939b-50b173ddda18: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910520::DEBUG::2013-11-19 21:15:00,669::task::495::TaskManager.Task::(_debug) Task dc6bd7a0-b338-4e44-939b-50b173ddda18: moving from state preparing -> state finished >Thread-15910520::DEBUG::2013-11-19 21:15:00,670::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910520::DEBUG::2013-11-19 21:15:00,670::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910520::DEBUG::2013-11-19 21:15:00,671::task::495::TaskManager.Task::(_debug) Task dc6bd7a0-b338-4e44-939b-50b173ddda18: ref 0 aborting False >Thread-15910520::INFO::2013-11-19 21:15:00,671::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910522::DEBUG::2013-11-19 21:15:01,677::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910522::INFO::2013-11-19 21:15:01,678::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910522::DEBUG::2013-11-19 21:15:01,679::task::495::TaskManager.Task::(_debug) Task b55ea658-4986-420e-914e-2442013aa4f8: moving from state init -> state preparing >Thread-15910522::DEBUG::2013-11-19 21:15:01,679::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910522::DEBUG::2013-11-19 21:15:01,680::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910522::DEBUG::2013-11-19 21:15:01,680::task::495::TaskManager.Task::(_debug) Task b55ea658-4986-420e-914e-2442013aa4f8: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910522::DEBUG::2013-11-19 21:15:01,681::task::495::TaskManager.Task::(_debug) Task b55ea658-4986-420e-914e-2442013aa4f8: moving from state preparing -> state finished >Thread-15910522::DEBUG::2013-11-19 21:15:01,681::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910522::DEBUG::2013-11-19 21:15:01,682::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910522::DEBUG::2013-11-19 21:15:01,682::task::495::TaskManager.Task::(_debug) Task b55ea658-4986-420e-914e-2442013aa4f8: ref 0 aborting False >Thread-15910522::INFO::2013-11-19 21:15:01,683::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910523::DEBUG::2013-11-19 21:15:02,689::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910523::INFO::2013-11-19 21:15:02,690::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910523::DEBUG::2013-11-19 21:15:02,690::task::495::TaskManager.Task::(_debug) Task f3d7ad6c-7ab1-489e-bcb8-33eb51c577da: moving from state init -> state preparing >Thread-15910523::DEBUG::2013-11-19 21:15:02,691::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910523::DEBUG::2013-11-19 21:15:02,691::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910523::DEBUG::2013-11-19 21:15:02,692::task::495::TaskManager.Task::(_debug) Task f3d7ad6c-7ab1-489e-bcb8-33eb51c577da: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910523::DEBUG::2013-11-19 21:15:02,692::task::495::TaskManager.Task::(_debug) Task f3d7ad6c-7ab1-489e-bcb8-33eb51c577da: moving from state preparing -> state finished >Thread-15910523::DEBUG::2013-11-19 21:15:02,693::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910523::DEBUG::2013-11-19 21:15:02,694::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910523::DEBUG::2013-11-19 21:15:02,694::task::495::TaskManager.Task::(_debug) Task f3d7ad6c-7ab1-489e-bcb8-33eb51c577da: ref 0 aborting False >Thread-15910523::INFO::2013-11-19 21:15:02,695::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-1024767::ERROR::2013-11-19 21:15:03,511::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910525::DEBUG::2013-11-19 21:15:03,700::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910525::INFO::2013-11-19 21:15:03,701::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910525::DEBUG::2013-11-19 21:15:03,713::task::495::TaskManager.Task::(_debug) Task be899098-1413-45b3-988f-8eec5f1747d6: moving from state init -> state preparing >Thread-15910525::DEBUG::2013-11-19 21:15:03,713::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910525::DEBUG::2013-11-19 21:15:03,714::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910525::DEBUG::2013-11-19 21:15:03,714::task::495::TaskManager.Task::(_debug) Task be899098-1413-45b3-988f-8eec5f1747d6: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910525::DEBUG::2013-11-19 21:15:03,715::task::495::TaskManager.Task::(_debug) Task be899098-1413-45b3-988f-8eec5f1747d6: moving from state preparing -> state finished >Thread-15910525::DEBUG::2013-11-19 21:15:03,716::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910525::DEBUG::2013-11-19 21:15:03,716::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910525::DEBUG::2013-11-19 21:15:03,717::task::495::TaskManager.Task::(_debug) Task be899098-1413-45b3-988f-8eec5f1747d6: ref 0 aborting False >Thread-15910525::INFO::2013-11-19 21:15:03,717::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910526::DEBUG::2013-11-19 21:15:04,723::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910526::INFO::2013-11-19 21:15:04,724::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910526::DEBUG::2013-11-19 21:15:04,724::task::495::TaskManager.Task::(_debug) Task e9800a1f-1535-4037-a9b0-03203dc22042: moving from state init -> state preparing >Thread-15910526::DEBUG::2013-11-19 21:15:04,725::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910526::DEBUG::2013-11-19 21:15:04,725::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910526::DEBUG::2013-11-19 21:15:04,726::task::495::TaskManager.Task::(_debug) Task e9800a1f-1535-4037-a9b0-03203dc22042: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910526::DEBUG::2013-11-19 21:15:04,726::task::495::TaskManager.Task::(_debug) Task e9800a1f-1535-4037-a9b0-03203dc22042: moving from state preparing -> state finished >Thread-15910526::DEBUG::2013-11-19 21:15:04,727::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910526::DEBUG::2013-11-19 21:15:04,727::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910526::DEBUG::2013-11-19 21:15:04,728::task::495::TaskManager.Task::(_debug) Task e9800a1f-1535-4037-a9b0-03203dc22042: ref 0 aborting False >Thread-15910526::INFO::2013-11-19 21:15:04,729::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910527::INFO::2013-11-19 21:15:05,353::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910527::DEBUG::2013-11-19 21:15:05,354::task::495::TaskManager.Task::(_debug) Task b6530d88-7669-4dfe-afd9-4d0aa325f1a8: moving from state init -> state preparing >Thread-15910527::DEBUG::2013-11-19 21:15:05,354::task::495::TaskManager.Task::(_debug) Task b6530d88-7669-4dfe-afd9-4d0aa325f1a8: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000344038009644', 'lastCheck': 1384895696.870894, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00145101547241', 'lastCheck': 1384895698.8019259, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895703.512305, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000396966934204', 'lastCheck': 1384895702.5128241, 'valid': True, 'code': 0}} >Thread-15910527::DEBUG::2013-11-19 21:15:05,355::task::495::TaskManager.Task::(_debug) Task b6530d88-7669-4dfe-afd9-4d0aa325f1a8: moving from state preparing -> state finished >Thread-15910527::DEBUG::2013-11-19 21:15:05,356::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910527::DEBUG::2013-11-19 21:15:05,356::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910527::DEBUG::2013-11-19 21:15:05,357::task::495::TaskManager.Task::(_debug) Task b6530d88-7669-4dfe-afd9-4d0aa325f1a8: ref 0 aborting False >Thread-15910527::INFO::2013-11-19 21:15:05,357::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000344038009644', 'lastCheck': 1384895696.870894, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00145101547241', 'lastCheck': 1384895698.8019259, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895703.512305, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000396966934204', 'lastCheck': 1384895702.5128241, 'valid': True, 'code': 0}} >Thread-15910529::DEBUG::2013-11-19 21:15:05,734::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910529::INFO::2013-11-19 21:15:05,735::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910529::DEBUG::2013-11-19 21:15:05,736::task::495::TaskManager.Task::(_debug) Task 75bdbfec-4c24-4d97-89df-8c16f7bc0fae: moving from state init -> state preparing >Thread-15910529::DEBUG::2013-11-19 21:15:05,736::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910529::DEBUG::2013-11-19 21:15:05,737::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910529::DEBUG::2013-11-19 21:15:05,737::task::495::TaskManager.Task::(_debug) Task 75bdbfec-4c24-4d97-89df-8c16f7bc0fae: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910529::DEBUG::2013-11-19 21:15:05,738::task::495::TaskManager.Task::(_debug) Task 75bdbfec-4c24-4d97-89df-8c16f7bc0fae: moving from state preparing -> state finished >Thread-15910529::DEBUG::2013-11-19 21:15:05,738::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910529::DEBUG::2013-11-19 21:15:05,739::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910529::DEBUG::2013-11-19 21:15:05,739::task::495::TaskManager.Task::(_debug) Task 75bdbfec-4c24-4d97-89df-8c16f7bc0fae: ref 0 aborting False >Thread-15910529::INFO::2013-11-19 21:15:05,740::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910530::DEBUG::2013-11-19 21:15:06,746::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910530::INFO::2013-11-19 21:15:06,747::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910530::DEBUG::2013-11-19 21:15:06,747::task::495::TaskManager.Task::(_debug) Task bb3a4a71-7b30-475c-a53a-ac8f17a22b8d: moving from state init -> state preparing >Thread-15910530::DEBUG::2013-11-19 21:15:06,748::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910530::DEBUG::2013-11-19 21:15:06,748::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910530::DEBUG::2013-11-19 21:15:06,749::task::495::TaskManager.Task::(_debug) Task bb3a4a71-7b30-475c-a53a-ac8f17a22b8d: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910530::DEBUG::2013-11-19 21:15:06,749::task::495::TaskManager.Task::(_debug) Task bb3a4a71-7b30-475c-a53a-ac8f17a22b8d: moving from state preparing -> state finished >Thread-15910530::DEBUG::2013-11-19 21:15:06,750::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910530::DEBUG::2013-11-19 21:15:06,750::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910530::DEBUG::2013-11-19 21:15:06,751::task::495::TaskManager.Task::(_debug) Task bb3a4a71-7b30-475c-a53a-ac8f17a22b8d: ref 0 aborting False >Thread-15910530::INFO::2013-11-19 21:15:06,752::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910532::DEBUG::2013-11-19 21:15:07,757::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910532::INFO::2013-11-19 21:15:07,758::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910532::DEBUG::2013-11-19 21:15:07,759::task::495::TaskManager.Task::(_debug) Task 0b05d2fc-e8ba-4364-93b4-91c26fbc96db: moving from state init -> state preparing >Thread-15910532::DEBUG::2013-11-19 21:15:07,759::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910532::DEBUG::2013-11-19 21:15:07,760::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910532::DEBUG::2013-11-19 21:15:07,760::task::495::TaskManager.Task::(_debug) Task 0b05d2fc-e8ba-4364-93b4-91c26fbc96db: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910532::DEBUG::2013-11-19 21:15:07,761::task::495::TaskManager.Task::(_debug) Task 0b05d2fc-e8ba-4364-93b4-91c26fbc96db: moving from state preparing -> state finished >Thread-15910532::DEBUG::2013-11-19 21:15:07,762::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910532::DEBUG::2013-11-19 21:15:07,762::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910532::DEBUG::2013-11-19 21:15:07,763::task::495::TaskManager.Task::(_debug) Task 0b05d2fc-e8ba-4364-93b4-91c26fbc96db: ref 0 aborting False >Thread-15910532::INFO::2013-11-19 21:15:07,763::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:08,396::safelease::61::Storage.Misc.excCmd::(acquire) SUCCESS: <err> = ''; <rc> = 0 >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:08,397::safelease::65::ClusterLock::(acquire) Clustered lock acquired successfully >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:08,397::spm::468::Storage.SPM::(start) spm lock acquired successfully >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:08,398::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:08,399::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:08,399::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:08,400::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:08,401::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:08,402::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15910533::DEBUG::2013-11-19 21:15:08,771::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910533::INFO::2013-11-19 21:15:08,773::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910533::DEBUG::2013-11-19 21:15:08,792::task::495::TaskManager.Task::(_debug) Task 74016fe4-21ab-4c64-9e75-68921daeabfb: moving from state init -> state preparing >Thread-15910533::DEBUG::2013-11-19 21:15:08,792::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910533::DEBUG::2013-11-19 21:15:08,793::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910533::DEBUG::2013-11-19 21:15:08,794::task::495::TaskManager.Task::(_debug) Task 74016fe4-21ab-4c64-9e75-68921daeabfb: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910533::DEBUG::2013-11-19 21:15:08,794::task::495::TaskManager.Task::(_debug) Task 74016fe4-21ab-4c64-9e75-68921daeabfb: moving from state preparing -> state finished >Thread-15910533::DEBUG::2013-11-19 21:15:08,795::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910533::DEBUG::2013-11-19 21:15:08,795::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910533::DEBUG::2013-11-19 21:15:08,796::task::495::TaskManager.Task::(_debug) Task 74016fe4-21ab-4c64-9e75-68921daeabfb: ref 0 aborting False >Thread-15910533::INFO::2013-11-19 21:15:08,797::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:08,990::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:08,993::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:08,994::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=277', u'POOL_SPM_ID=-1', u'_SHA_CKSUM=44c47fe1672a8deeb5c0f134ce9469f0ec72525b'] >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:08,995::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:08,996::persistentDict::146::Storage.PersistentDict::(transaction) Flushing changes >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:08,997::persistentDict::265::Storage.PersistentDict::(flush) about to write lines (VGTagMetadataRW)=[u'CLASS=Data', u'DESCRIPTION=OS', u'IOOPTIMEOUTSEC=10', u'LEASERETRIES=3', u'LEASETIMESEC=60', u'LOCKPOLICY=', u'LOCKRENEWALINTERVALSEC=5', u'LOGBLKSIZE=512', u'MASTER_VERSION=1', u'PHYBLKSIZE=512', u'POOL_DESCRIPTION=Default', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_ID=2', u'POOL_SPM_LVER=278', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'ROLE=Master', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'TYPE=FCP', u'VERSION=2', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', '_SHA_CKSUM=913ce85a0b55e60c3f92c982d8a15ae168c40596'] >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:08,997::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:08,998::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:08,999::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:08,999::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:09,000::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:09,001::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:09,565::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:09,573::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:09,574::blockSD::145::storage.Metadata.VGTagMetadataRW::(writelines) Updating metadata adding=MDT_POOL_SPM_ID=2, MDT_POOL_SPM_LVER=278, MDT__SHA_CKSUM=913ce85a0b55e60c3f92c982d8a15ae168c40596 removing=MDT__SHA_CKSUM=44c47fe1672a8deeb5c0f134ce9469f0ec72525b, MDT_POOL_SPM_ID=-1, MDT_POOL_SPM_LVER=277 >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:09,576::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --deltag MDT_POOL_SPM_ID=-1 --deltag MDT__SHA_CKSUM=44c47fe1672a8deeb5c0f134ce9469f0ec72525b --deltag MDT_POOL_SPM_LVER=277 --addtag MDT_POOL_SPM_LVER=278 --addtag MDT_POOL_SPM_ID=2 --addtag MDT__SHA_CKSUM=913ce85a0b55e60c3f92c982d8a15ae168c40596 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15910535::DEBUG::2013-11-19 21:15:09,807::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910535::INFO::2013-11-19 21:15:09,810::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910535::DEBUG::2013-11-19 21:15:09,811::task::495::TaskManager.Task::(_debug) Task 4ce1fcf1-d025-4027-b973-96b18f551db6: moving from state init -> state preparing >Thread-15910535::DEBUG::2013-11-19 21:15:09,812::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910535::DEBUG::2013-11-19 21:15:09,812::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910535::DEBUG::2013-11-19 21:15:09,813::task::495::TaskManager.Task::(_debug) Task 4ce1fcf1-d025-4027-b973-96b18f551db6: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910535::DEBUG::2013-11-19 21:15:09,814::task::495::TaskManager.Task::(_debug) Task 4ce1fcf1-d025-4027-b973-96b18f551db6: moving from state preparing -> state finished >Thread-15910535::DEBUG::2013-11-19 21:15:09,815::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910535::DEBUG::2013-11-19 21:15:09,816::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910535::DEBUG::2013-11-19 21:15:09,817::task::495::TaskManager.Task::(_debug) Task 4ce1fcf1-d025-4027-b973-96b18f551db6: ref 0 aborting False >Thread-15910535::INFO::2013-11-19 21:15:09,817::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,261::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,262::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,263::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,264::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,265::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`646117f5-1e62-471a-97fc-246bf947cb9f`::Request was made in '/usr/share/vdsm/storage/spm.py' line '330' at '_upgradePool' >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,266::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,266::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,267::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`646117f5-1e62-471a-97fc-246bf947cb9f`::Granted request >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::INFO::2013-11-19 21:15:10,268::spm::336::Storage.SPM::(_upgradePool) Trying to upgrade master domain `6b892067-ac7e-4039-a7be-5b7387525b16` >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,269::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b6d7fe31-1d2f-4141-85fa-2f051914b317`::Request was made in '/usr/share/vdsm/storage/spm.py' line '337' at '_upgradePool' >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,269::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'exclusive' >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,270::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'exclusive' (1 active user) >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,270::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b6d7fe31-1d2f-4141-85fa-2f051914b317`::Granted request >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,271::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,272::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,273::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,273::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,274::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,275::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15910536::DEBUG::2013-11-19 21:15:10,825::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910536::INFO::2013-11-19 21:15:10,827::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910536::DEBUG::2013-11-19 21:15:10,828::task::495::TaskManager.Task::(_debug) Task 721ed015-cfb7-4aca-b2bc-eb7c0079bc89: moving from state init -> state preparing >Thread-15910536::DEBUG::2013-11-19 21:15:10,828::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910536::DEBUG::2013-11-19 21:15:10,829::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910536::DEBUG::2013-11-19 21:15:10,830::task::495::TaskManager.Task::(_debug) Task 721ed015-cfb7-4aca-b2bc-eb7c0079bc89: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910536::DEBUG::2013-11-19 21:15:10,830::task::495::TaskManager.Task::(_debug) Task 721ed015-cfb7-4aca-b2bc-eb7c0079bc89: moving from state preparing -> state finished >Thread-15910536::DEBUG::2013-11-19 21:15:10,831::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910536::DEBUG::2013-11-19 21:15:10,831::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910536::DEBUG::2013-11-19 21:15:10,832::task::495::TaskManager.Task::(_debug) Task 721ed015-cfb7-4aca-b2bc-eb7c0079bc89: ref 0 aborting False >Thread-15910536::INFO::2013-11-19 21:15:10,832::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,921::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,924::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,925::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=278', u'POOL_SPM_ID=2', u'_SHA_CKSUM=913ce85a0b55e60c3f92c982d8a15ae168c40596'] >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,927::blockSD::625::Storage.StorageDomain::(upgrade) Trying to upgrade domain `6b892067-ac7e-4039-a7be-5b7387525b16` from version 2 to version 2 >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,927::blockSD::631::Storage.StorageDomain::(upgrade) No need to upgrade domain `6b892067-ac7e-4039-a7be-5b7387525b16`, leaving unchanged >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,928::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,928::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,929::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,929::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,930::spm::340::Storage.SPM::(_upgradePool) Marking all domains for upgrade >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,931::spm::347::Storage.SPM::(_upgradePool) Registering with state change event >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,931::spm::349::Storage.SPM::(_upgradePool) Running initial domain upgrade threads >Thread-15910537::DEBUG::2013-11-19 21:15:10,932::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 45cdff8b-89c0-4926-a8c7-80354064bfa6 >Thread-15910537::DEBUG::2013-11-19 21:15:10,934::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`6d31b91c-c6f3-4a78-8c7a-6001ea6b2467`::Request was made in '/usr/share/vdsm/storage/spm.py' line '376' at '_upgradePoolDomain' >Thread-15910538::DEBUG::2013-11-19 21:15:10,935::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 772472cb-2429-466d-96cd-00b3301ef78a >Thread-15910539::DEBUG::2013-11-19 21:15:10,936::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 5d7b5178-3790-472c-a13b-b481eb27a61e >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,936::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910537::DEBUG::2013-11-19 21:15:10,944::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' for lock type 'exclusive' >Thread-15910538::DEBUG::2013-11-19 21:15:10,948::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`2c65d13e-9f93-4f6d-9178-2c03158c43c0`::Request was made in '/usr/share/vdsm/storage/spm.py' line '376' at '_upgradePoolDomain' >Thread-15910539::ERROR::2013-11-19 21:15:10,952::spm::373::Storage.SPM::(_upgradePoolDomain) Error while preparing domain `5d7b5178-3790-472c-a13b-b481eb27a61e` upgrade >Traceback (most recent call last): > File "/usr/share/vdsm/storage/spm.py", line 369, in _upgradePoolDomain >IndexError: list index out of range >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,953::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910538::DEBUG::2013-11-19 21:15:10,958::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'exclusive' >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,960::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,961::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910537::DEBUG::2013-11-19 21:15:10,961::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' is free. Now locking as 'exclusive' (1 active user) >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:10,962::blockSD::791::Storage.Misc.excCmd::(mountMaster) '/usr/bin/sudo -n /sbin/e2fsck -p /dev/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15910537::DEBUG::2013-11-19 21:15:10,963::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`6d31b91c-c6f3-4a78-8c7a-6001ea6b2467`::Granted request >Thread-15910538::DEBUG::2013-11-19 21:15:10,964::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'exclusive' (1 active user) >Thread-15910538::DEBUG::2013-11-19 21:15:11,169::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`2c65d13e-9f93-4f6d-9178-2c03158c43c0`::Granted request >Thread-15910537::DEBUG::2013-11-19 21:15:11,173::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`4d1fc42b-09c4-4001-ad7f-613dcb89e13d`::Request was made in '/usr/share/vdsm/storage/spm.py' line '377' at '_upgradePoolDomain' >Thread-15910537::DEBUG::2013-11-19 21:15:11,180::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' for lock type 'exclusive' >Thread-15910538::DEBUG::2013-11-19 21:15:11,187::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`441cfe1f-5d54-45e8-b117-aa7c65252dae`::Request was made in '/usr/share/vdsm/storage/spm.py' line '377' at '_upgradePoolDomain' >Thread-15910537::DEBUG::2013-11-19 21:15:11,187::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' is free. Now locking as 'exclusive' (1 active user) >Thread-15910538::DEBUG::2013-11-19 21:15:11,188::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'exclusive' >Thread-15910537::DEBUG::2013-11-19 21:15:11,189::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`4d1fc42b-09c4-4001-ad7f-613dcb89e13d`::Granted request >Thread-15910538::DEBUG::2013-11-19 21:15:11,191::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'exclusive' (1 active user) >Thread-15910537::DEBUG::2013-11-19 21:15:11,192::spm::385::Storage.SPM::(_upgradePoolDomain) Domain `45cdff8b-89c0-4926-a8c7-80354064bfa6` is not a data domain it is an 2 domain, not upgrading >Thread-15910538::DEBUG::2013-11-19 21:15:11,193::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`441cfe1f-5d54-45e8-b117-aa7c65252dae`::Granted request >Thread-15910537::DEBUG::2013-11-19 21:15:11,194::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' >Thread-15910538::DEBUG::2013-11-19 21:15:11,196::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15910537::DEBUG::2013-11-19 21:15:11,197::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' (0 active users) >Thread-15910538::DEBUG::2013-11-19 21:15:11,198::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15910537::DEBUG::2013-11-19 21:15:11,199::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' is free, finding out if anyone is waiting for it. >Thread-15910538::DEBUG::2013-11-19 21:15:11,200::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15910537::DEBUG::2013-11-19 21:15:11,200::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6', Clearing records. >Thread-15910538::DEBUG::2013-11-19 21:15:11,201::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15910538::DEBUG::2013-11-19 21:15:11,203::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15910537::DEBUG::2013-11-19 21:15:11,204::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' >Thread-15910538::DEBUG::2013-11-19 21:15:11,205::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15910537::DEBUG::2013-11-19 21:15:11,206::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' (0 active users) >Thread-15910537::DEBUG::2013-11-19 21:15:11,208::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' is free, finding out if anyone is waiting for it. >Thread-15910537::DEBUG::2013-11-19 21:15:11,521::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6', Clearing records. >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:11,529::blockSD::791::Storage.Misc.excCmd::(mountMaster) SUCCESS: <err> = ''; <rc> = 0 >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:11,530::blockSD::816::Storage.Misc.excCmd::(mountMaster) '/usr/bin/sudo -n /sbin/tune2fs -j /dev/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15910541::DEBUG::2013-11-19 21:15:11,881::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910541::INFO::2013-11-19 21:15:11,883::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910541::DEBUG::2013-11-19 21:15:11,885::task::495::TaskManager.Task::(_debug) Task 60934217-81cd-49cb-8bd5-b8fa63953eb4: moving from state init -> state preparing >Thread-15910541::DEBUG::2013-11-19 21:15:11,885::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910541::DEBUG::2013-11-19 21:15:11,886::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910541::DEBUG::2013-11-19 21:15:11,887::task::495::TaskManager.Task::(_debug) Task 60934217-81cd-49cb-8bd5-b8fa63953eb4: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910541::DEBUG::2013-11-19 21:15:11,888::task::495::TaskManager.Task::(_debug) Task 60934217-81cd-49cb-8bd5-b8fa63953eb4: moving from state preparing -> state finished >Thread-15910541::DEBUG::2013-11-19 21:15:11,889::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910541::DEBUG::2013-11-19 21:15:11,890::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910541::DEBUG::2013-11-19 21:15:11,891::task::495::TaskManager.Task::(_debug) Task 60934217-81cd-49cb-8bd5-b8fa63953eb4: ref 0 aborting False >Thread-15910541::INFO::2013-11-19 21:15:11,892::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:12,075::blockSD::816::Storage.Misc.excCmd::(mountMaster) FAILED: <err> = 'The filesystem already has a journal.\n'; <rc> = 1 >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:12,078::fileUtils::83::Storage.Misc.excCmd::(mount) '/usr/bin/sudo -n /bin/mount /dev/6b892067-ac7e-4039-a7be-5b7387525b16/master /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15910538::DEBUG::2013-11-19 21:15:12,080::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15910538::DEBUG::2013-11-19 21:15:12,304::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15910538::DEBUG::2013-11-19 21:15:12,306::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'SDUUID=772472cb-2429-466d-96cd-00b3301ef78a', u'VERSION=2', u'LOCKPOLICY=', u'VGUUID=XTb294-NYv6-dVpO-bB1n-4f9P-d0GL-dhd2DI', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'LEASETIMESEC=5', u'LOCKRENEWALINTERVALSEC=5', u'ROLE=Regular', u'PHYBLKSIZE=512', u'DESCRIPTION=Database', u'IOOPTIMEOUTSEC=1', u'CLASS=Data', u'TYPE=FCP', u'PV0=pv:3600c0ff0001520c7414f605001000000,uuid:NujM33-vNiD-sBHY-mrPo-EQ2f-1qsz-HXfn0q,pestart:0,pecount:8929,mapoffset:0', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'MASTER_VERSION=0', u'_SHA_CKSUM=ef8840d9fdef24516a56fd970bed1960d2b1ae92'] >Thread-15910538::DEBUG::2013-11-19 21:15:12,311::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15910538::DEBUG::2013-11-19 21:15:12,315::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15910538::DEBUG::2013-11-19 21:15:12,315::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15910538::DEBUG::2013-11-19 21:15:12,319::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15910538::DEBUG::2013-11-19 21:15:12,321::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15910538::DEBUG::2013-11-19 21:15:12,321::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15910538::DEBUG::2013-11-19 21:15:12,325::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15910538::DEBUG::2013-11-19 21:15:12,331::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:12,499::fileUtils::83::Storage.Misc.excCmd::(mount) SUCCESS: <err> = ''; <rc> = 0 >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:12,501::blockSD::833::Storage.Misc.excCmd::(mountMaster) '/usr/bin/sudo -n /bin/chown vdsm:kvm /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:12,893::blockSD::833::Storage.Misc.excCmd::(mountMaster) SUCCESS: <err> = ''; <rc> = 0 >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::WARNING::2013-11-19 21:15:12,896::spm::490::Storage.SPM::(start) Backup domain validation failed >Traceback (most recent call last): > File "/usr/share/vdsm/storage/spm.py", line 488, in start > File "/usr/share/vdsm/storage/sp.py", line 1208, in checkBackupDomain > File "/usr/share/vdsm/storage/sd.py", line 666, in isBackup > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:12,898::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:12,899::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:12,900::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:12,900::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:12,901::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:12,901::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:12,902::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:12,902::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:12,903::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >Thread-15910542::DEBUG::2013-11-19 21:15:12,905::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:12,905::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:12,907::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >Thread-15910542::INFO::2013-11-19 21:15:12,907::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:12,908::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >Thread-15910542::DEBUG::2013-11-19 21:15:12,909::task::495::TaskManager.Task::(_debug) Task d0f39847-12cc-49ab-b3cc-b58d33b4eb7d: moving from state init -> state preparing >Thread-15910542::DEBUG::2013-11-19 21:15:12,910::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910542::DEBUG::2013-11-19 21:15:12,910::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910542::DEBUG::2013-11-19 21:15:12,911::task::495::TaskManager.Task::(_debug) Task d0f39847-12cc-49ab-b3cc-b58d33b4eb7d: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910542::DEBUG::2013-11-19 21:15:12,912::task::495::TaskManager.Task::(_debug) Task d0f39847-12cc-49ab-b3cc-b58d33b4eb7d: moving from state preparing -> state finished >Thread-15910542::DEBUG::2013-11-19 21:15:12,912::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910542::DEBUG::2013-11-19 21:15:12,913::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910542::DEBUG::2013-11-19 21:15:12,913::task::495::TaskManager.Task::(_debug) Task d0f39847-12cc-49ab-b3cc-b58d33b4eb7d: ref 0 aborting False >Thread-15910542::INFO::2013-11-19 21:15:12,914::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:12,919::threadPool::25::Misc.ThreadPool::(__init__) Enter - numThreads: 5.0, waitTimeout: 3, maxTasks: 500.0 >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:12,924::storage_mailbox::486::Storage.MailBox.SpmMailMonitor::(__init__) SPM_MailMonitor - clearing outgoing mail, command is: ['dd', 'of=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/outbox', 'oflag=direct', 'iflag=fullblock', 'conv=notrunc', 'count=1'] >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:12,925::storage_mailbox::488::Storage.Misc.excCmd::(__init__) 'dd of=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/outbox oflag=direct iflag=fullblock conv=notrunc count=1 bs=1024000' (cwd None) >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:13,371::storage_mailbox::488::Storage.Misc.excCmd::(__init__) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0406752 s, 25.2 MB/s\n'; <rc> = 0 >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:13,373::storage_mailbox::493::Storage.MailBox.SpmMailMonitor::(__init__) SPM_MailMonitor created for pool c4e5889a-348d-11e2-83fe-9f72369eab30 >Dummy-15910548::DEBUG::2013-11-19 21:15:13,374::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:13,376::spm::523::Storage.SPM::(start) ended. >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:13,378::task::495::TaskManager.Task::(_debug) Task 21b4b46d-58c8-40ca-8f03-a8063e50f1b9: moving from state running -> state finished >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:13,624::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:13,647::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:13,649::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:13,649::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:13,650::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:13,650::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:13,651::task::915::TaskManager.Task::(_runJobs) Task.run: exit - success: result >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:13,652::task::495::TaskManager.Task::(_debug) Task 21b4b46d-58c8-40ca-8f03-a8063e50f1b9: ref 0 aborting False >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:15:13,653::threadPool::47::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 >Thread-1024767::ERROR::2013-11-19 21:15:13,881::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15910548::DEBUG::2013-11-19 21:15:13,907::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0703364 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15910550::DEBUG::2013-11-19 21:15:13,929::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910550::INFO::2013-11-19 21:15:13,930::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910550::DEBUG::2013-11-19 21:15:13,931::task::495::TaskManager.Task::(_debug) Task 52511ad9-48e4-4a5b-b02b-e2b6551c98e5: moving from state init -> state preparing >Thread-15910550::DEBUG::2013-11-19 21:15:13,932::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910550::DEBUG::2013-11-19 21:15:13,933::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': '1 jobs completed successfuly', 'taskState': 'finished', 'taskResult': 'success', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'} >Thread-15910550::DEBUG::2013-11-19 21:15:13,934::task::495::TaskManager.Task::(_debug) Task 52511ad9-48e4-4a5b-b02b-e2b6551c98e5: finished: {'taskStatus': {'code': 0, 'message': '1 jobs completed successfuly', 'taskState': 'finished', 'taskResult': 'success', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910550::DEBUG::2013-11-19 21:15:13,935::task::495::TaskManager.Task::(_debug) Task 52511ad9-48e4-4a5b-b02b-e2b6551c98e5: moving from state preparing -> state finished >Thread-15910550::DEBUG::2013-11-19 21:15:13,935::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910550::DEBUG::2013-11-19 21:15:13,936::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910550::DEBUG::2013-11-19 21:15:13,937::task::495::TaskManager.Task::(_debug) Task 52511ad9-48e4-4a5b-b02b-e2b6551c98e5: ref 0 aborting False >Thread-15910550::INFO::2013-11-19 21:15:13,938::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': '1 jobs completed successfuly', 'taskState': 'finished', 'taskResult': 'success', 'taskID': '21b4b46d-58c8-40ca-8f03-a8063e50f1b9'}} >Thread-15910551::DEBUG::2013-11-19 21:15:13,942::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910551::INFO::2013-11-19 21:15:13,943::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910551::DEBUG::2013-11-19 21:15:13,943::task::495::TaskManager.Task::(_debug) Task 58ff7d91-906a-455b-88e6-1d4e83222e18: moving from state init -> state preparing >Thread-15910551::DEBUG::2013-11-19 21:15:13,944::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910551::DEBUG::2013-11-19 21:15:13,945::task::495::TaskManager.Task::(_debug) Task 58ff7d91-906a-455b-88e6-1d4e83222e18: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910551::DEBUG::2013-11-19 21:15:13,945::task::495::TaskManager.Task::(_debug) Task 58ff7d91-906a-455b-88e6-1d4e83222e18: moving from state preparing -> state finished >Thread-15910551::DEBUG::2013-11-19 21:15:13,946::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910551::DEBUG::2013-11-19 21:15:13,946::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910551::DEBUG::2013-11-19 21:15:13,947::task::495::TaskManager.Task::(_debug) Task 58ff7d91-906a-455b-88e6-1d4e83222e18: ref 0 aborting False >Thread-15910551::INFO::2013-11-19 21:15:13,947::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910552::DEBUG::2013-11-19 21:15:13,953::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910552::INFO::2013-11-19 21:15:13,954::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: clearTask, args: ( taskID=21b4b46d-58c8-40ca-8f03-a8063e50f1b9) >Thread-15910552::DEBUG::2013-11-19 21:15:13,954::task::495::TaskManager.Task::(_debug) Task da521451-7213-4dbd-8c00-e09e6cd3a39a: moving from state init -> state preparing >Thread-15910552::DEBUG::2013-11-19 21:15:13,955::taskManager::146::TaskManager::(clearTask) Entry. taskID: 21b4b46d-58c8-40ca-8f03-a8063e50f1b9 >Thread-15910552::DEBUG::2013-11-19 21:15:13,955::taskManager::151::TaskManager::(clearTask) Return. >Thread-15910552::DEBUG::2013-11-19 21:15:13,956::task::495::TaskManager.Task::(_debug) Task da521451-7213-4dbd-8c00-e09e6cd3a39a: finished: None >Thread-15910552::DEBUG::2013-11-19 21:15:13,956::task::495::TaskManager.Task::(_debug) Task da521451-7213-4dbd-8c00-e09e6cd3a39a: moving from state preparing -> state finished >Thread-15910552::DEBUG::2013-11-19 21:15:13,957::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910552::DEBUG::2013-11-19 21:15:13,957::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910552::DEBUG::2013-11-19 21:15:13,958::task::495::TaskManager.Task::(_debug) Task da521451-7213-4dbd-8c00-e09e6cd3a39a: ref 0 aborting False >Thread-15910552::INFO::2013-11-19 21:15:13,958::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: clearTask, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15910553::DEBUG::2013-11-19 21:15:13,995::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910553::INFO::2013-11-19 21:15:13,996::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910553::DEBUG::2013-11-19 21:15:13,997::task::495::TaskManager.Task::(_debug) Task 803f1de4-c8a7-420f-8294-06c3ec01c2a0: moving from state init -> state preparing >Thread-15910553::DEBUG::2013-11-19 21:15:13,998::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`079e5672-ac83-4781-96a1-6a6b44b1af78`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910553::DEBUG::2013-11-19 21:15:13,998::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910553::DEBUG::2013-11-19 21:15:13,999::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910553::DEBUG::2013-11-19 21:15:14,000::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`079e5672-ac83-4781-96a1-6a6b44b1af78`::Granted request >Thread-15910553::DEBUG::2013-11-19 21:15:14,001::task::495::TaskManager.Task::(_debug) Task 803f1de4-c8a7-420f-8294-06c3ec01c2a0: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910553::DEBUG::2013-11-19 21:15:14,001::task::495::TaskManager.Task::(_debug) Task 803f1de4-c8a7-420f-8294-06c3ec01c2a0: ref 1 aborting False >Thread-15910553::WARNING::2013-11-19 21:15:14,003::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910553::DEBUG::2013-11-19 21:15:14,004::task::495::TaskManager.Task::(_debug) Task 803f1de4-c8a7-420f-8294-06c3ec01c2a0: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910553::DEBUG::2013-11-19 21:15:14,005::task::495::TaskManager.Task::(_debug) Task 803f1de4-c8a7-420f-8294-06c3ec01c2a0: moving from state preparing -> state finished >Thread-15910553::DEBUG::2013-11-19 21:15:14,006::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910553::DEBUG::2013-11-19 21:15:14,006::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910553::DEBUG::2013-11-19 21:15:14,007::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910553::DEBUG::2013-11-19 21:15:14,008::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910553::DEBUG::2013-11-19 21:15:14,008::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910553::DEBUG::2013-11-19 21:15:14,009::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910553::DEBUG::2013-11-19 21:15:14,009::task::495::TaskManager.Task::(_debug) Task 803f1de4-c8a7-420f-8294-06c3ec01c2a0: ref 0 aborting False >Thread-15910553::INFO::2013-11-19 21:15:14,010::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910554::DEBUG::2013-11-19 21:15:14,126::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910554::INFO::2013-11-19 21:15:14,127::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksInfo, args: () >Thread-15910554::DEBUG::2013-11-19 21:15:14,128::task::495::TaskManager.Task::(_debug) Task c33acd6c-e73a-4738-bb2d-67f8fe29448d: moving from state init -> state preparing >Thread-15910554::DEBUG::2013-11-19 21:15:14,128::taskManager::168::TaskManager::(getAllTasksInfo) Entry. >Thread-15910554::DEBUG::2013-11-19 21:15:14,129::taskManager::173::TaskManager::(getAllTasksInfo) Return. Response: {} >Thread-15910554::DEBUG::2013-11-19 21:15:14,129::task::495::TaskManager.Task::(_debug) Task c33acd6c-e73a-4738-bb2d-67f8fe29448d: finished: {'allTasksInfo': {}} >Thread-15910554::DEBUG::2013-11-19 21:15:14,130::task::495::TaskManager.Task::(_debug) Task c33acd6c-e73a-4738-bb2d-67f8fe29448d: moving from state preparing -> state finished >Thread-15910554::DEBUG::2013-11-19 21:15:14,131::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910554::DEBUG::2013-11-19 21:15:14,131::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910554::DEBUG::2013-11-19 21:15:14,132::task::495::TaskManager.Task::(_debug) Task c33acd6c-e73a-4738-bb2d-67f8fe29448d: ref 0 aborting False >Thread-15910554::INFO::2013-11-19 21:15:14,132::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'allTasksInfo': {}} >Thread-15910555::DEBUG::2013-11-19 21:15:14,139::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910555::INFO::2013-11-19 21:15:14,140::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksInfo, args: () >Thread-15910555::DEBUG::2013-11-19 21:15:14,140::task::495::TaskManager.Task::(_debug) Task fe73aa27-87d9-4467-b066-695a4fac0638: moving from state init -> state preparing >Thread-15910555::DEBUG::2013-11-19 21:15:14,141::taskManager::168::TaskManager::(getAllTasksInfo) Entry. >Thread-15910555::DEBUG::2013-11-19 21:15:14,141::taskManager::173::TaskManager::(getAllTasksInfo) Return. Response: {} >Thread-15910555::DEBUG::2013-11-19 21:15:14,142::task::495::TaskManager.Task::(_debug) Task fe73aa27-87d9-4467-b066-695a4fac0638: finished: {'allTasksInfo': {}} >Thread-15910555::DEBUG::2013-11-19 21:15:14,143::task::495::TaskManager.Task::(_debug) Task fe73aa27-87d9-4467-b066-695a4fac0638: moving from state preparing -> state finished >Thread-15910555::DEBUG::2013-11-19 21:15:14,143::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910555::DEBUG::2013-11-19 21:15:14,144::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910555::DEBUG::2013-11-19 21:15:14,144::task::495::TaskManager.Task::(_debug) Task fe73aa27-87d9-4467-b066-695a4fac0638: ref 0 aborting False >Thread-15910555::INFO::2013-11-19 21:15:14,145::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'allTasksInfo': {}} >Thread-15910556::INFO::2013-11-19 21:15:15,652::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910556::DEBUG::2013-11-19 21:15:15,653::task::495::TaskManager.Task::(_debug) Task 18cc9775-b91c-442c-a192-18e66179b787: moving from state init -> state preparing >Thread-15910556::WARNING::2013-11-19 21:15:15,653::hsm::1895::Storage.HSM::(public_repoStats) repoStats detected invalid master:6b892067-ac7e-4039-a7be-5b7387525b16 {'finish': 1384895706.8834889, 'mdasize': '134479872', 'mdathreshold': True, 'mdavalid': True, 'diskfree': '1142595518464', 'disktotal': '3595827150848', 'result': {'delay': '0.000278949737549', 'lastCheck': 1384895706.8834889, 'valid': True, 'code': 0}, 'masterValidate': {'mount': False, 'valid': True}, 'mdafree': '67205632'} >Thread-15910556::DEBUG::2013-11-19 21:15:15,654::task::495::TaskManager.Task::(_debug) Task 18cc9775-b91c-442c-a192-18e66179b787: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000278949737549', 'lastCheck': 1384895706.8834889, 'valid': True, 'code': 355}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00109696388245', 'lastCheck': 1384895708.8166831, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895713.8826809, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000510931015015', 'lastCheck': 1384895712.876039, 'valid': True, 'code': 0}} >Thread-15910556::DEBUG::2013-11-19 21:15:15,655::task::495::TaskManager.Task::(_debug) Task 18cc9775-b91c-442c-a192-18e66179b787: moving from state preparing -> state finished >Thread-15910556::DEBUG::2013-11-19 21:15:15,655::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910556::DEBUG::2013-11-19 21:15:15,656::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910556::DEBUG::2013-11-19 21:15:15,657::task::495::TaskManager.Task::(_debug) Task 18cc9775-b91c-442c-a192-18e66179b787: ref 0 aborting False >Thread-15910556::INFO::2013-11-19 21:15:15,657::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000278949737549', 'lastCheck': 1384895706.8834889, 'valid': True, 'code': 355}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00109696388245', 'lastCheck': 1384895708.8166831, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895713.8826809, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000510931015015', 'lastCheck': 1384895712.876039, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:15:15,930::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:15:16,461::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0709684 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:15:18,477::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:15:19,007::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0743424 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:15:21,022::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:15:21,569::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0713424 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:15:23,585::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:15:23,899::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15910548::DEBUG::2013-11-19 21:15:24,118::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0711656 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15910562::DEBUG::2013-11-19 21:15:24,133::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910562::INFO::2013-11-19 21:15:24,134::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910562::DEBUG::2013-11-19 21:15:24,134::task::495::TaskManager.Task::(_debug) Task 9ab733fe-d1c8-4bbf-b4d3-e606453a84fb: moving from state init -> state preparing >Thread-15910562::DEBUG::2013-11-19 21:15:24,135::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910562::DEBUG::2013-11-19 21:15:24,135::task::495::TaskManager.Task::(_debug) Task 9ab733fe-d1c8-4bbf-b4d3-e606453a84fb: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910562::DEBUG::2013-11-19 21:15:24,136::task::495::TaskManager.Task::(_debug) Task 9ab733fe-d1c8-4bbf-b4d3-e606453a84fb: moving from state preparing -> state finished >Thread-15910562::DEBUG::2013-11-19 21:15:24,136::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910562::DEBUG::2013-11-19 21:15:24,136::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910562::DEBUG::2013-11-19 21:15:24,137::task::495::TaskManager.Task::(_debug) Task 9ab733fe-d1c8-4bbf-b4d3-e606453a84fb: ref 0 aborting False >Thread-15910562::INFO::2013-11-19 21:15:24,137::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910563::DEBUG::2013-11-19 21:15:24,141::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910563::INFO::2013-11-19 21:15:24,142::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910563::DEBUG::2013-11-19 21:15:24,142::task::495::TaskManager.Task::(_debug) Task dbcfcf97-940b-42e1-8f8e-7823c170677f: moving from state init -> state preparing >Thread-15910563::DEBUG::2013-11-19 21:15:24,143::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`dacf4bd5-93e9-4054-a2ea-1dd8cb3759cc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910563::DEBUG::2013-11-19 21:15:24,143::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910563::DEBUG::2013-11-19 21:15:24,143::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910563::DEBUG::2013-11-19 21:15:24,144::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`dacf4bd5-93e9-4054-a2ea-1dd8cb3759cc`::Granted request >Thread-15910563::DEBUG::2013-11-19 21:15:24,144::task::495::TaskManager.Task::(_debug) Task dbcfcf97-940b-42e1-8f8e-7823c170677f: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910563::DEBUG::2013-11-19 21:15:24,145::task::495::TaskManager.Task::(_debug) Task dbcfcf97-940b-42e1-8f8e-7823c170677f: ref 1 aborting False >Thread-15910563::WARNING::2013-11-19 21:15:24,146::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910563::DEBUG::2013-11-19 21:15:24,147::task::495::TaskManager.Task::(_debug) Task dbcfcf97-940b-42e1-8f8e-7823c170677f: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910563::DEBUG::2013-11-19 21:15:24,147::task::495::TaskManager.Task::(_debug) Task dbcfcf97-940b-42e1-8f8e-7823c170677f: moving from state preparing -> state finished >Thread-15910563::DEBUG::2013-11-19 21:15:24,147::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910563::DEBUG::2013-11-19 21:15:24,148::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910563::DEBUG::2013-11-19 21:15:24,148::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910563::DEBUG::2013-11-19 21:15:24,149::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910563::DEBUG::2013-11-19 21:15:24,149::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910563::DEBUG::2013-11-19 21:15:24,149::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910563::DEBUG::2013-11-19 21:15:24,150::task::495::TaskManager.Task::(_debug) Task dbcfcf97-940b-42e1-8f8e-7823c170677f: ref 0 aborting False >Thread-15910563::INFO::2013-11-19 21:15:24,150::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910564::INFO::2013-11-19 21:15:25,814::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910564::DEBUG::2013-11-19 21:15:25,814::task::495::TaskManager.Task::(_debug) Task 27b8f6b4-2800-4ea4-b1aa-4bd0c0b25827: moving from state init -> state preparing >Thread-15910564::DEBUG::2013-11-19 21:15:25,815::task::495::TaskManager.Task::(_debug) Task 27b8f6b4-2800-4ea4-b1aa-4bd0c0b25827: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000325202941895', 'lastCheck': 1384895716.897404, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00138807296753', 'lastCheck': 1384895718.888042, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895723.900955, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000407934188843', 'lastCheck': 1384895722.887794, 'valid': True, 'code': 0}} >Thread-15910564::DEBUG::2013-11-19 21:15:25,816::task::495::TaskManager.Task::(_debug) Task 27b8f6b4-2800-4ea4-b1aa-4bd0c0b25827: moving from state preparing -> state finished >Thread-15910564::DEBUG::2013-11-19 21:15:25,816::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910564::DEBUG::2013-11-19 21:15:25,817::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910564::DEBUG::2013-11-19 21:15:25,817::task::495::TaskManager.Task::(_debug) Task 27b8f6b4-2800-4ea4-b1aa-4bd0c0b25827: ref 0 aborting False >Thread-15910564::INFO::2013-11-19 21:15:25,818::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000325202941895', 'lastCheck': 1384895716.897404, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00138807296753', 'lastCheck': 1384895718.888042, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895723.900955, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000407934188843', 'lastCheck': 1384895722.887794, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:15:26,134::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:15:26,642::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0707014 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:15:28,658::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:15:29,194::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0711509 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:15:31,209::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:15:31,770::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.107993 s, 9.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:15:33,785::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:15:34,035::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910570::DEBUG::2013-11-19 21:15:34,275::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910570::INFO::2013-11-19 21:15:34,277::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910570::DEBUG::2013-11-19 21:15:34,293::task::495::TaskManager.Task::(_debug) Task 45e3aadf-4441-4cc2-bca7-306d7d7c53cd: moving from state init -> state preparing >Thread-15910570::DEBUG::2013-11-19 21:15:34,295::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910570::DEBUG::2013-11-19 21:15:34,296::task::495::TaskManager.Task::(_debug) Task 45e3aadf-4441-4cc2-bca7-306d7d7c53cd: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910570::DEBUG::2013-11-19 21:15:34,298::task::495::TaskManager.Task::(_debug) Task 45e3aadf-4441-4cc2-bca7-306d7d7c53cd: moving from state preparing -> state finished >Thread-15910570::DEBUG::2013-11-19 21:15:34,299::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910570::DEBUG::2013-11-19 21:15:34,300::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910570::DEBUG::2013-11-19 21:15:34,301::task::495::TaskManager.Task::(_debug) Task 45e3aadf-4441-4cc2-bca7-306d7d7c53cd: ref 0 aborting False >Thread-15910570::INFO::2013-11-19 21:15:34,303::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910571::DEBUG::2013-11-19 21:15:34,309::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910571::INFO::2013-11-19 21:15:34,311::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910571::DEBUG::2013-11-19 21:15:34,312::task::495::TaskManager.Task::(_debug) Task 2d0c5157-346c-4281-a6bf-a66baa0a96ee: moving from state init -> state preparing >Thread-15910571::DEBUG::2013-11-19 21:15:34,314::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f9281132-3f21-474e-a3af-2d70e9119254`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910571::DEBUG::2013-11-19 21:15:34,316::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910571::DEBUG::2013-11-19 21:15:34,319::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910571::DEBUG::2013-11-19 21:15:34,320::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f9281132-3f21-474e-a3af-2d70e9119254`::Granted request >Thread-15910571::DEBUG::2013-11-19 21:15:34,322::task::495::TaskManager.Task::(_debug) Task 2d0c5157-346c-4281-a6bf-a66baa0a96ee: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910571::DEBUG::2013-11-19 21:15:34,324::task::495::TaskManager.Task::(_debug) Task 2d0c5157-346c-4281-a6bf-a66baa0a96ee: ref 1 aborting False >Thread-15910571::WARNING::2013-11-19 21:15:34,328::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910571::DEBUG::2013-11-19 21:15:34,330::task::495::TaskManager.Task::(_debug) Task 2d0c5157-346c-4281-a6bf-a66baa0a96ee: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910571::DEBUG::2013-11-19 21:15:34,332::task::495::TaskManager.Task::(_debug) Task 2d0c5157-346c-4281-a6bf-a66baa0a96ee: moving from state preparing -> state finished >Thread-15910571::DEBUG::2013-11-19 21:15:34,333::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910571::DEBUG::2013-11-19 21:15:34,335::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910571::DEBUG::2013-11-19 21:15:34,337::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910571::DEBUG::2013-11-19 21:15:34,338::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910571::DEBUG::2013-11-19 21:15:34,339::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910571::DEBUG::2013-11-19 21:15:34,340::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910571::DEBUG::2013-11-19 21:15:34,342::task::495::TaskManager.Task::(_debug) Task 2d0c5157-346c-4281-a6bf-a66baa0a96ee: ref 0 aborting False >Thread-15910571::INFO::2013-11-19 21:15:34,344::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15910548::DEBUG::2013-11-19 21:15:34,401::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0781844 s, 13.1 MB/s\n'; <rc> = 0 >Thread-15910572::INFO::2013-11-19 21:15:36,039::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910572::DEBUG::2013-11-19 21:15:36,040::task::495::TaskManager.Task::(_debug) Task b1299791-f110-4f0d-ae2e-fa2835db9abf: moving from state init -> state preparing >Thread-15910572::DEBUG::2013-11-19 21:15:36,041::task::495::TaskManager.Task::(_debug) Task b1299791-f110-4f0d-ae2e-fa2835db9abf: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000499963760376', 'lastCheck': 1384895726.9124539, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00159597396851', 'lastCheck': 1384895728.9060919, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895734.0372019, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000357151031494', 'lastCheck': 1384895732.900136, 'valid': True, 'code': 0}} >Thread-15910572::DEBUG::2013-11-19 21:15:36,042::task::495::TaskManager.Task::(_debug) Task b1299791-f110-4f0d-ae2e-fa2835db9abf: moving from state preparing -> state finished >Thread-15910572::DEBUG::2013-11-19 21:15:36,042::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910572::DEBUG::2013-11-19 21:15:36,043::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910572::DEBUG::2013-11-19 21:15:36,043::task::495::TaskManager.Task::(_debug) Task b1299791-f110-4f0d-ae2e-fa2835db9abf: ref 0 aborting False >Thread-15910572::INFO::2013-11-19 21:15:36,044::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000499963760376', 'lastCheck': 1384895726.9124539, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00159597396851', 'lastCheck': 1384895728.9060919, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895734.0372019, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000357151031494', 'lastCheck': 1384895732.900136, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:15:36,425::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:15:36,962::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0712188 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:15:38,977::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:15:39,496::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0715435 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:15:41,510::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:15:42,003::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0679397 s, 15.1 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:15:44,018::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:15:44,247::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910578::DEBUG::2013-11-19 21:15:44,465::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910578::INFO::2013-11-19 21:15:44,467::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910578::DEBUG::2013-11-19 21:15:44,468::task::495::TaskManager.Task::(_debug) Task 4f73bb23-75fa-48e7-ac58-ca86145246fe: moving from state init -> state preparing >Thread-15910578::DEBUG::2013-11-19 21:15:44,469::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910578::DEBUG::2013-11-19 21:15:44,470::task::495::TaskManager.Task::(_debug) Task 4f73bb23-75fa-48e7-ac58-ca86145246fe: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910578::DEBUG::2013-11-19 21:15:44,471::task::495::TaskManager.Task::(_debug) Task 4f73bb23-75fa-48e7-ac58-ca86145246fe: moving from state preparing -> state finished >Thread-15910578::DEBUG::2013-11-19 21:15:44,472::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910578::DEBUG::2013-11-19 21:15:44,472::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910578::DEBUG::2013-11-19 21:15:44,473::task::495::TaskManager.Task::(_debug) Task 4f73bb23-75fa-48e7-ac58-ca86145246fe: ref 0 aborting False >Thread-15910578::INFO::2013-11-19 21:15:44,474::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910579::DEBUG::2013-11-19 21:15:44,479::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910579::INFO::2013-11-19 21:15:44,481::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910579::DEBUG::2013-11-19 21:15:44,481::task::495::TaskManager.Task::(_debug) Task a98df52e-340d-424a-ae04-f8e78d0d66d0: moving from state init -> state preparing >Thread-15910579::DEBUG::2013-11-19 21:15:44,483::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c5ebb70d-dae9-4da6-9ee9-bf4751a53c0c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910579::DEBUG::2013-11-19 21:15:44,484::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910579::DEBUG::2013-11-19 21:15:44,485::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910579::DEBUG::2013-11-19 21:15:44,486::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c5ebb70d-dae9-4da6-9ee9-bf4751a53c0c`::Granted request >Thread-15910579::DEBUG::2013-11-19 21:15:44,488::task::495::TaskManager.Task::(_debug) Task a98df52e-340d-424a-ae04-f8e78d0d66d0: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910579::DEBUG::2013-11-19 21:15:44,489::task::495::TaskManager.Task::(_debug) Task a98df52e-340d-424a-ae04-f8e78d0d66d0: ref 1 aborting False >Thread-15910579::WARNING::2013-11-19 21:15:44,492::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910579::DEBUG::2013-11-19 21:15:44,493::task::495::TaskManager.Task::(_debug) Task a98df52e-340d-424a-ae04-f8e78d0d66d0: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910579::DEBUG::2013-11-19 21:15:44,494::task::495::TaskManager.Task::(_debug) Task a98df52e-340d-424a-ae04-f8e78d0d66d0: moving from state preparing -> state finished >Thread-15910579::DEBUG::2013-11-19 21:15:44,495::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910579::DEBUG::2013-11-19 21:15:44,496::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910579::DEBUG::2013-11-19 21:15:44,497::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910579::DEBUG::2013-11-19 21:15:44,498::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910579::DEBUG::2013-11-19 21:15:44,499::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910579::DEBUG::2013-11-19 21:15:44,500::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910579::DEBUG::2013-11-19 21:15:44,501::task::495::TaskManager.Task::(_debug) Task a98df52e-340d-424a-ae04-f8e78d0d66d0: ref 0 aborting False >Thread-15910579::INFO::2013-11-19 21:15:44,502::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15910548::DEBUG::2013-11-19 21:15:44,568::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100855 s, 10.2 MB/s\n'; <rc> = 0 >Thread-15910580::INFO::2013-11-19 21:15:46,258::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910580::DEBUG::2013-11-19 21:15:46,259::task::495::TaskManager.Task::(_debug) Task 52720f41-0b65-4395-8a24-4dd1986b8d24: moving from state init -> state preparing >Thread-15910580::DEBUG::2013-11-19 21:15:46,260::task::495::TaskManager.Task::(_debug) Task 52720f41-0b65-4395-8a24-4dd1986b8d24: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000674962997437', 'lastCheck': 1384895737.020926, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00257802009583', 'lastCheck': 1384895739.192733, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895744.25155, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000842809677124', 'lastCheck': 1384895743.1882861, 'valid': True, 'code': 0}} >Thread-15910580::DEBUG::2013-11-19 21:15:46,261::task::495::TaskManager.Task::(_debug) Task 52720f41-0b65-4395-8a24-4dd1986b8d24: moving from state preparing -> state finished >Thread-15910580::DEBUG::2013-11-19 21:15:46,261::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910580::DEBUG::2013-11-19 21:15:46,262::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910580::DEBUG::2013-11-19 21:15:46,262::task::495::TaskManager.Task::(_debug) Task 52720f41-0b65-4395-8a24-4dd1986b8d24: ref 0 aborting False >Thread-15910580::INFO::2013-11-19 21:15:46,263::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000674962997437', 'lastCheck': 1384895737.020926, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00257802009583', 'lastCheck': 1384895739.192733, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895744.25155, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000842809677124', 'lastCheck': 1384895743.1882861, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:15:46,583::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:15:47,120::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0717487 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:15:49,134::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:15:49,648::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0699884 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:15:51,663::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:15:52,196::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0706343 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:15:54,212::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:15:54,427::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910586::DEBUG::2013-11-19 21:15:54,615::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910586::INFO::2013-11-19 21:15:54,616::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910586::DEBUG::2013-11-19 21:15:54,617::task::495::TaskManager.Task::(_debug) Task 75876aa4-6f2b-4e46-9af3-60a33ab18928: moving from state init -> state preparing >Thread-15910586::DEBUG::2013-11-19 21:15:54,619::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910586::DEBUG::2013-11-19 21:15:54,620::task::495::TaskManager.Task::(_debug) Task 75876aa4-6f2b-4e46-9af3-60a33ab18928: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910586::DEBUG::2013-11-19 21:15:54,620::task::495::TaskManager.Task::(_debug) Task 75876aa4-6f2b-4e46-9af3-60a33ab18928: moving from state preparing -> state finished >Thread-15910586::DEBUG::2013-11-19 21:15:54,621::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910586::DEBUG::2013-11-19 21:15:54,622::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910586::DEBUG::2013-11-19 21:15:54,623::task::495::TaskManager.Task::(_debug) Task 75876aa4-6f2b-4e46-9af3-60a33ab18928: ref 0 aborting False >Thread-15910586::INFO::2013-11-19 21:15:54,624::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910587::DEBUG::2013-11-19 21:15:54,630::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910587::INFO::2013-11-19 21:15:54,631::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910587::DEBUG::2013-11-19 21:15:54,632::task::495::TaskManager.Task::(_debug) Task 6141c5c2-28fd-47f6-9ae8-3bedee3185bd: moving from state init -> state preparing >Thread-15910587::DEBUG::2013-11-19 21:15:54,634::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b94eed69-1342-4c35-bdb1-7c3b216f94a0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910587::DEBUG::2013-11-19 21:15:54,635::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910587::DEBUG::2013-11-19 21:15:54,636::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910587::DEBUG::2013-11-19 21:15:54,637::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b94eed69-1342-4c35-bdb1-7c3b216f94a0`::Granted request >Thread-15910587::DEBUG::2013-11-19 21:15:54,639::task::495::TaskManager.Task::(_debug) Task 6141c5c2-28fd-47f6-9ae8-3bedee3185bd: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910587::DEBUG::2013-11-19 21:15:54,640::task::495::TaskManager.Task::(_debug) Task 6141c5c2-28fd-47f6-9ae8-3bedee3185bd: ref 1 aborting False >Thread-15910587::WARNING::2013-11-19 21:15:54,643::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910587::DEBUG::2013-11-19 21:15:54,644::task::495::TaskManager.Task::(_debug) Task 6141c5c2-28fd-47f6-9ae8-3bedee3185bd: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910587::DEBUG::2013-11-19 21:15:54,645::task::495::TaskManager.Task::(_debug) Task 6141c5c2-28fd-47f6-9ae8-3bedee3185bd: moving from state preparing -> state finished >Thread-15910587::DEBUG::2013-11-19 21:15:54,646::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910587::DEBUG::2013-11-19 21:15:54,647::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910587::DEBUG::2013-11-19 21:15:54,648::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910587::DEBUG::2013-11-19 21:15:54,649::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910587::DEBUG::2013-11-19 21:15:54,650::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910587::DEBUG::2013-11-19 21:15:54,651::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910587::DEBUG::2013-11-19 21:15:54,652::task::495::TaskManager.Task::(_debug) Task 6141c5c2-28fd-47f6-9ae8-3bedee3185bd: ref 0 aborting False >Thread-15910587::INFO::2013-11-19 21:15:54,653::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15910548::DEBUG::2013-11-19 21:15:54,765::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100751 s, 10.2 MB/s\n'; <rc> = 0 >Thread-15910588::INFO::2013-11-19 21:15:56,440::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910588::DEBUG::2013-11-19 21:15:56,441::task::495::TaskManager.Task::(_debug) Task d3d0bebf-bff1-418c-b645-1aee647bcec7: moving from state init -> state preparing >Thread-15910588::DEBUG::2013-11-19 21:15:56,441::task::495::TaskManager.Task::(_debug) Task d3d0bebf-bff1-418c-b645-1aee647bcec7: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000349998474121', 'lastCheck': 1384895747.2468829, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00129795074463', 'lastCheck': 1384895749.3710151, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895754.431953, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000501871109009', 'lastCheck': 1384895753.355706, 'valid': True, 'code': 0}} >Thread-15910588::DEBUG::2013-11-19 21:15:56,442::task::495::TaskManager.Task::(_debug) Task d3d0bebf-bff1-418c-b645-1aee647bcec7: moving from state preparing -> state finished >Thread-15910588::DEBUG::2013-11-19 21:15:56,443::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910588::DEBUG::2013-11-19 21:15:56,443::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910588::DEBUG::2013-11-19 21:15:56,444::task::495::TaskManager.Task::(_debug) Task d3d0bebf-bff1-418c-b645-1aee647bcec7: ref 0 aborting False >Thread-15910588::INFO::2013-11-19 21:15:56,444::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000349998474121', 'lastCheck': 1384895747.2468829, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00129795074463', 'lastCheck': 1384895749.3710151, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895754.431953, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000501871109009', 'lastCheck': 1384895753.355706, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:15:56,780::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:15:57,314::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0708345 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:15:59,329::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:15:59,854::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0711629 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:16:01,868::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:16:02,390::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0705285 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:16:04,405::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:16:04,659::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910594::DEBUG::2013-11-19 21:16:04,762::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910594::INFO::2013-11-19 21:16:04,764::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910594::DEBUG::2013-11-19 21:16:04,764::task::495::TaskManager.Task::(_debug) Task 54bd5431-26a4-4ea4-9318-e318f0e839ac: moving from state init -> state preparing >Thread-15910594::DEBUG::2013-11-19 21:16:04,765::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910594::DEBUG::2013-11-19 21:16:04,766::task::495::TaskManager.Task::(_debug) Task 54bd5431-26a4-4ea4-9318-e318f0e839ac: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910594::DEBUG::2013-11-19 21:16:04,766::task::495::TaskManager.Task::(_debug) Task 54bd5431-26a4-4ea4-9318-e318f0e839ac: moving from state preparing -> state finished >Thread-15910594::DEBUG::2013-11-19 21:16:04,767::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910594::DEBUG::2013-11-19 21:16:04,768::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910594::DEBUG::2013-11-19 21:16:04,768::task::495::TaskManager.Task::(_debug) Task 54bd5431-26a4-4ea4-9318-e318f0e839ac: ref 0 aborting False >Thread-15910594::INFO::2013-11-19 21:16:04,769::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910595::DEBUG::2013-11-19 21:16:04,774::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910595::INFO::2013-11-19 21:16:04,775::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910595::DEBUG::2013-11-19 21:16:04,775::task::495::TaskManager.Task::(_debug) Task a184a53e-c62f-4ce9-8ef5-772eabf31f99: moving from state init -> state preparing >Thread-15910595::DEBUG::2013-11-19 21:16:04,776::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`71595538-d15d-4687-bf96-82895f02bee2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910595::DEBUG::2013-11-19 21:16:04,777::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910595::DEBUG::2013-11-19 21:16:04,778::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910595::DEBUG::2013-11-19 21:16:04,780::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`71595538-d15d-4687-bf96-82895f02bee2`::Granted request >Thread-15910595::DEBUG::2013-11-19 21:16:04,781::task::495::TaskManager.Task::(_debug) Task a184a53e-c62f-4ce9-8ef5-772eabf31f99: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910595::DEBUG::2013-11-19 21:16:04,782::task::495::TaskManager.Task::(_debug) Task a184a53e-c62f-4ce9-8ef5-772eabf31f99: ref 1 aborting False >Thread-15910595::WARNING::2013-11-19 21:16:04,784::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910595::DEBUG::2013-11-19 21:16:04,786::task::495::TaskManager.Task::(_debug) Task a184a53e-c62f-4ce9-8ef5-772eabf31f99: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910595::DEBUG::2013-11-19 21:16:04,787::task::495::TaskManager.Task::(_debug) Task a184a53e-c62f-4ce9-8ef5-772eabf31f99: moving from state preparing -> state finished >Thread-15910595::DEBUG::2013-11-19 21:16:04,788::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910595::DEBUG::2013-11-19 21:16:04,789::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910595::DEBUG::2013-11-19 21:16:04,790::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910595::DEBUG::2013-11-19 21:16:04,792::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910595::DEBUG::2013-11-19 21:16:04,792::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910595::DEBUG::2013-11-19 21:16:04,793::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910595::DEBUG::2013-11-19 21:16:04,795::task::495::TaskManager.Task::(_debug) Task a184a53e-c62f-4ce9-8ef5-772eabf31f99: ref 0 aborting False >Thread-15910595::INFO::2013-11-19 21:16:04,796::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15910548::DEBUG::2013-11-19 21:16:04,936::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.106169 s, 9.6 MB/s\n'; <rc> = 0 >Thread-15910596::INFO::2013-11-19 21:16:06,671::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910596::DEBUG::2013-11-19 21:16:06,672::task::495::TaskManager.Task::(_debug) Task bf3ecb62-1d20-442e-ba8d-24633a23789b: moving from state init -> state preparing >Thread-15910596::DEBUG::2013-11-19 21:16:06,672::task::495::TaskManager.Task::(_debug) Task bf3ecb62-1d20-442e-ba8d-24633a23789b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000333070755005', 'lastCheck': 1384895757.429872, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00161910057068', 'lastCheck': 1384895759.5464809, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895764.663202, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00043511390686', 'lastCheck': 1384895763.544986, 'valid': True, 'code': 0}} >Thread-15910596::DEBUG::2013-11-19 21:16:06,673::task::495::TaskManager.Task::(_debug) Task bf3ecb62-1d20-442e-ba8d-24633a23789b: moving from state preparing -> state finished >Thread-15910596::DEBUG::2013-11-19 21:16:06,674::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910596::DEBUG::2013-11-19 21:16:06,674::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910596::DEBUG::2013-11-19 21:16:06,675::task::495::TaskManager.Task::(_debug) Task bf3ecb62-1d20-442e-ba8d-24633a23789b: ref 0 aborting False >Thread-15910596::INFO::2013-11-19 21:16:06,676::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000333070755005', 'lastCheck': 1384895757.429872, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00161910057068', 'lastCheck': 1384895759.5464809, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895764.663202, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00043511390686', 'lastCheck': 1384895763.544986, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:16:06,952::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:16:07,495::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0750174 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:16:09,509::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:16:10,015::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0707134 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:16:12,031::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:16:12,564::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0709703 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:16:14,580::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:16:14,798::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910602::DEBUG::2013-11-19 21:16:14,920::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910602::INFO::2013-11-19 21:16:14,922::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910602::DEBUG::2013-11-19 21:16:14,923::task::495::TaskManager.Task::(_debug) Task 8584ffaa-9d60-4a55-9edb-fdbf5953856a: moving from state init -> state preparing >Thread-15910602::DEBUG::2013-11-19 21:16:14,924::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910602::DEBUG::2013-11-19 21:16:14,924::task::495::TaskManager.Task::(_debug) Task 8584ffaa-9d60-4a55-9edb-fdbf5953856a: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910602::DEBUG::2013-11-19 21:16:14,925::task::495::TaskManager.Task::(_debug) Task 8584ffaa-9d60-4a55-9edb-fdbf5953856a: moving from state preparing -> state finished >Thread-15910602::DEBUG::2013-11-19 21:16:14,926::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910602::DEBUG::2013-11-19 21:16:14,926::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910602::DEBUG::2013-11-19 21:16:14,927::task::495::TaskManager.Task::(_debug) Task 8584ffaa-9d60-4a55-9edb-fdbf5953856a: ref 0 aborting False >Thread-15910602::INFO::2013-11-19 21:16:14,927::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910603::DEBUG::2013-11-19 21:16:14,932::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910603::INFO::2013-11-19 21:16:14,933::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910603::DEBUG::2013-11-19 21:16:14,934::task::495::TaskManager.Task::(_debug) Task 828e2416-f6af-40dd-ab36-9d27d77323fe: moving from state init -> state preparing >Thread-15910603::DEBUG::2013-11-19 21:16:14,936::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2a7b7918-da39-4c02-9913-f87819b3de26`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910603::DEBUG::2013-11-19 21:16:14,936::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910603::DEBUG::2013-11-19 21:16:14,937::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910603::DEBUG::2013-11-19 21:16:14,938::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2a7b7918-da39-4c02-9913-f87819b3de26`::Granted request >Thread-15910603::DEBUG::2013-11-19 21:16:14,939::task::495::TaskManager.Task::(_debug) Task 828e2416-f6af-40dd-ab36-9d27d77323fe: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910603::DEBUG::2013-11-19 21:16:14,939::task::495::TaskManager.Task::(_debug) Task 828e2416-f6af-40dd-ab36-9d27d77323fe: ref 1 aborting False >Thread-15910603::WARNING::2013-11-19 21:16:14,942::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910603::DEBUG::2013-11-19 21:16:14,943::task::495::TaskManager.Task::(_debug) Task 828e2416-f6af-40dd-ab36-9d27d77323fe: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910603::DEBUG::2013-11-19 21:16:14,944::task::495::TaskManager.Task::(_debug) Task 828e2416-f6af-40dd-ab36-9d27d77323fe: moving from state preparing -> state finished >Thread-15910603::DEBUG::2013-11-19 21:16:14,944::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910603::DEBUG::2013-11-19 21:16:14,945::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910603::DEBUG::2013-11-19 21:16:14,946::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910603::DEBUG::2013-11-19 21:16:14,946::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910603::DEBUG::2013-11-19 21:16:14,947::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910603::DEBUG::2013-11-19 21:16:14,947::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910603::DEBUG::2013-11-19 21:16:14,948::task::495::TaskManager.Task::(_debug) Task 828e2416-f6af-40dd-ab36-9d27d77323fe: ref 0 aborting False >Thread-15910603::INFO::2013-11-19 21:16:14,949::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15910548::DEBUG::2013-11-19 21:16:15,106::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0713685 s, 14.3 MB/s\n'; <rc> = 0 >Thread-15910604::INFO::2013-11-19 21:16:16,810::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910604::DEBUG::2013-11-19 21:16:16,811::task::495::TaskManager.Task::(_debug) Task 9ff9cddb-23d7-4c08-a9fc-480471740f7e: moving from state init -> state preparing >Thread-15910604::DEBUG::2013-11-19 21:16:16,811::task::495::TaskManager.Task::(_debug) Task 9ff9cddb-23d7-4c08-a9fc-480471740f7e: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00046706199646', 'lastCheck': 1384895767.6651831, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00163722038269', 'lastCheck': 1384895769.7116101, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895774.802418, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000565052032471', 'lastCheck': 1384895773.709693, 'valid': True, 'code': 0}} >Thread-15910604::DEBUG::2013-11-19 21:16:16,813::task::495::TaskManager.Task::(_debug) Task 9ff9cddb-23d7-4c08-a9fc-480471740f7e: moving from state preparing -> state finished >Thread-15910604::DEBUG::2013-11-19 21:16:16,813::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910604::DEBUG::2013-11-19 21:16:16,814::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910604::DEBUG::2013-11-19 21:16:16,815::task::495::TaskManager.Task::(_debug) Task 9ff9cddb-23d7-4c08-a9fc-480471740f7e: ref 0 aborting False >Thread-15910604::INFO::2013-11-19 21:16:16,815::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00046706199646', 'lastCheck': 1384895767.6651831, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00163722038269', 'lastCheck': 1384895769.7116101, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895774.802418, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000565052032471', 'lastCheck': 1384895773.709693, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:16:17,121::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:16:17,634::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.070076 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:16:19,649::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:16:20,210::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0717554 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:16:22,226::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:16:22,763::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0723858 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:16:24,778::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:16:25,017::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910610::DEBUG::2013-11-19 21:16:25,060::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910610::INFO::2013-11-19 21:16:25,061::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910610::DEBUG::2013-11-19 21:16:25,062::task::495::TaskManager.Task::(_debug) Task df23ec8a-4061-438f-9c23-4465902f2873: moving from state init -> state preparing >Thread-15910610::DEBUG::2013-11-19 21:16:25,063::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910610::DEBUG::2013-11-19 21:16:25,064::task::495::TaskManager.Task::(_debug) Task df23ec8a-4061-438f-9c23-4465902f2873: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910610::DEBUG::2013-11-19 21:16:25,064::task::495::TaskManager.Task::(_debug) Task df23ec8a-4061-438f-9c23-4465902f2873: moving from state preparing -> state finished >Thread-15910610::DEBUG::2013-11-19 21:16:25,065::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910610::DEBUG::2013-11-19 21:16:25,066::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910610::DEBUG::2013-11-19 21:16:25,066::task::495::TaskManager.Task::(_debug) Task df23ec8a-4061-438f-9c23-4465902f2873: ref 0 aborting False >Thread-15910610::INFO::2013-11-19 21:16:25,067::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910611::DEBUG::2013-11-19 21:16:25,072::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910611::INFO::2013-11-19 21:16:25,073::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910611::DEBUG::2013-11-19 21:16:25,073::task::495::TaskManager.Task::(_debug) Task 443cadef-28a0-4905-bf5d-e7c8692abb9a: moving from state init -> state preparing >Thread-15910611::DEBUG::2013-11-19 21:16:25,075::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`26e8e958-c795-4d1c-9ec6-c0ecc427f3f7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910611::DEBUG::2013-11-19 21:16:25,075::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910611::DEBUG::2013-11-19 21:16:25,076::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910611::DEBUG::2013-11-19 21:16:25,077::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`26e8e958-c795-4d1c-9ec6-c0ecc427f3f7`::Granted request >Thread-15910611::DEBUG::2013-11-19 21:16:25,078::task::495::TaskManager.Task::(_debug) Task 443cadef-28a0-4905-bf5d-e7c8692abb9a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910611::DEBUG::2013-11-19 21:16:25,079::task::495::TaskManager.Task::(_debug) Task 443cadef-28a0-4905-bf5d-e7c8692abb9a: ref 1 aborting False >Thread-15910611::WARNING::2013-11-19 21:16:25,081::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910611::DEBUG::2013-11-19 21:16:25,082::task::495::TaskManager.Task::(_debug) Task 443cadef-28a0-4905-bf5d-e7c8692abb9a: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910611::DEBUG::2013-11-19 21:16:25,083::task::495::TaskManager.Task::(_debug) Task 443cadef-28a0-4905-bf5d-e7c8692abb9a: moving from state preparing -> state finished >Thread-15910611::DEBUG::2013-11-19 21:16:25,083::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910611::DEBUG::2013-11-19 21:16:25,084::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910611::DEBUG::2013-11-19 21:16:25,085::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910611::DEBUG::2013-11-19 21:16:25,085::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910611::DEBUG::2013-11-19 21:16:25,086::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910611::DEBUG::2013-11-19 21:16:25,086::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910611::DEBUG::2013-11-19 21:16:25,087::task::495::TaskManager.Task::(_debug) Task 443cadef-28a0-4905-bf5d-e7c8692abb9a: ref 0 aborting False >Thread-15910611::INFO::2013-11-19 21:16:25,088::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15910548::DEBUG::2013-11-19 21:16:25,326::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0714091 s, 14.3 MB/s\n'; <rc> = 0 >Thread-15910612::INFO::2013-11-19 21:16:27,027::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910612::DEBUG::2013-11-19 21:16:27,028::task::495::TaskManager.Task::(_debug) Task cafbae51-4acf-4c81-a21a-3bd298753610: moving from state init -> state preparing >Thread-15910612::DEBUG::2013-11-19 21:16:27,028::task::495::TaskManager.Task::(_debug) Task cafbae51-4acf-4c81-a21a-3bd298753610: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000403165817261', 'lastCheck': 1384895777.7982261, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00130200386047', 'lastCheck': 1384895779.9537661, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895785.0196431, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000472068786621', 'lastCheck': 1384895783.897423, 'valid': True, 'code': 0}} >Thread-15910612::DEBUG::2013-11-19 21:16:27,029::task::495::TaskManager.Task::(_debug) Task cafbae51-4acf-4c81-a21a-3bd298753610: moving from state preparing -> state finished >Thread-15910612::DEBUG::2013-11-19 21:16:27,030::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910612::DEBUG::2013-11-19 21:16:27,030::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910612::DEBUG::2013-11-19 21:16:27,031::task::495::TaskManager.Task::(_debug) Task cafbae51-4acf-4c81-a21a-3bd298753610: ref 0 aborting False >Thread-15910612::INFO::2013-11-19 21:16:27,032::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000403165817261', 'lastCheck': 1384895777.7982261, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00130200386047', 'lastCheck': 1384895779.9537661, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895785.0196431, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000472068786621', 'lastCheck': 1384895783.897423, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:16:27,342::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:16:27,873::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.070541 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:16:29,888::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:16:30,409::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0701698 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:16:32,424::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:16:32,960::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0708102 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:16:34,975::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:16:35,193::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15910548::DEBUG::2013-11-19 21:16:35,501::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0707617 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15910618::DEBUG::2013-11-19 21:16:35,536::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910618::INFO::2013-11-19 21:16:35,537::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910618::DEBUG::2013-11-19 21:16:35,538::task::495::TaskManager.Task::(_debug) Task 015fec1f-4fa0-4c87-ac4d-85c985b78c0a: moving from state init -> state preparing >Thread-15910618::DEBUG::2013-11-19 21:16:35,539::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910618::DEBUG::2013-11-19 21:16:35,539::task::495::TaskManager.Task::(_debug) Task 015fec1f-4fa0-4c87-ac4d-85c985b78c0a: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910618::DEBUG::2013-11-19 21:16:35,540::task::495::TaskManager.Task::(_debug) Task 015fec1f-4fa0-4c87-ac4d-85c985b78c0a: moving from state preparing -> state finished >Thread-15910618::DEBUG::2013-11-19 21:16:35,541::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910618::DEBUG::2013-11-19 21:16:35,541::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910618::DEBUG::2013-11-19 21:16:35,542::task::495::TaskManager.Task::(_debug) Task 015fec1f-4fa0-4c87-ac4d-85c985b78c0a: ref 0 aborting False >Thread-15910618::INFO::2013-11-19 21:16:35,542::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910619::DEBUG::2013-11-19 21:16:35,547::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910619::INFO::2013-11-19 21:16:35,548::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910619::DEBUG::2013-11-19 21:16:35,549::task::495::TaskManager.Task::(_debug) Task d5f62c79-a02f-4a2a-a55a-c28b67db1fe2: moving from state init -> state preparing >Thread-15910619::DEBUG::2013-11-19 21:16:35,563::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b0149772-b837-4b82-969c-ffcbba40661e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910619::DEBUG::2013-11-19 21:16:35,564::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910619::DEBUG::2013-11-19 21:16:35,564::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910619::DEBUG::2013-11-19 21:16:35,565::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b0149772-b837-4b82-969c-ffcbba40661e`::Granted request >Thread-15910619::DEBUG::2013-11-19 21:16:35,566::task::495::TaskManager.Task::(_debug) Task d5f62c79-a02f-4a2a-a55a-c28b67db1fe2: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910619::DEBUG::2013-11-19 21:16:35,567::task::495::TaskManager.Task::(_debug) Task d5f62c79-a02f-4a2a-a55a-c28b67db1fe2: ref 1 aborting False >Thread-15910619::WARNING::2013-11-19 21:16:35,569::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910619::DEBUG::2013-11-19 21:16:35,569::task::495::TaskManager.Task::(_debug) Task d5f62c79-a02f-4a2a-a55a-c28b67db1fe2: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910619::DEBUG::2013-11-19 21:16:35,570::task::495::TaskManager.Task::(_debug) Task d5f62c79-a02f-4a2a-a55a-c28b67db1fe2: moving from state preparing -> state finished >Thread-15910619::DEBUG::2013-11-19 21:16:35,571::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910619::DEBUG::2013-11-19 21:16:35,571::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910619::DEBUG::2013-11-19 21:16:35,572::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910619::DEBUG::2013-11-19 21:16:35,573::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910619::DEBUG::2013-11-19 21:16:35,573::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910619::DEBUG::2013-11-19 21:16:35,574::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910619::DEBUG::2013-11-19 21:16:35,574::task::495::TaskManager.Task::(_debug) Task d5f62c79-a02f-4a2a-a55a-c28b67db1fe2: ref 0 aborting False >Thread-15910619::INFO::2013-11-19 21:16:35,575::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910620::INFO::2013-11-19 21:16:37,205::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910620::DEBUG::2013-11-19 21:16:37,206::task::495::TaskManager.Task::(_debug) Task 673882be-4045-4f77-ac8a-6bf5bec06d38: moving from state init -> state preparing >Thread-15910620::DEBUG::2013-11-19 21:16:37,206::task::495::TaskManager.Task::(_debug) Task 673882be-4045-4f77-ac8a-6bf5bec06d38: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000420808792114', 'lastCheck': 1384895788.0150459, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0015881061554', 'lastCheck': 1384895790.1179559, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895795.1972849, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000478029251099', 'lastCheck': 1384895794.1134479, 'valid': True, 'code': 0}} >Thread-15910620::DEBUG::2013-11-19 21:16:37,207::task::495::TaskManager.Task::(_debug) Task 673882be-4045-4f77-ac8a-6bf5bec06d38: moving from state preparing -> state finished >Thread-15910620::DEBUG::2013-11-19 21:16:37,208::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910620::DEBUG::2013-11-19 21:16:37,208::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910620::DEBUG::2013-11-19 21:16:37,209::task::495::TaskManager.Task::(_debug) Task 673882be-4045-4f77-ac8a-6bf5bec06d38: ref 0 aborting False >Thread-15910620::INFO::2013-11-19 21:16:37,209::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000420808792114', 'lastCheck': 1384895788.0150459, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0015881061554', 'lastCheck': 1384895790.1179559, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895795.1972849, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000478029251099', 'lastCheck': 1384895794.1134479, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:16:37,515::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:16:38,083::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.108334 s, 9.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:16:40,099::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:16:40,627::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.071435 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:16:42,642::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:16:43,162::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0699412 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:16:45,177::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:16:45,404::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15910548::DEBUG::2013-11-19 21:16:45,710::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0715822 s, 14.3 MB/s\n'; <rc> = 0 >Thread-15910626::DEBUG::2013-11-19 21:16:46,058::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910626::INFO::2013-11-19 21:16:46,060::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910626::DEBUG::2013-11-19 21:16:46,060::task::495::TaskManager.Task::(_debug) Task a2c429d9-bbaa-4793-b32b-9ea7340a57d5: moving from state init -> state preparing >Thread-15910626::DEBUG::2013-11-19 21:16:46,061::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910626::DEBUG::2013-11-19 21:16:46,062::task::495::TaskManager.Task::(_debug) Task a2c429d9-bbaa-4793-b32b-9ea7340a57d5: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910626::DEBUG::2013-11-19 21:16:46,062::task::495::TaskManager.Task::(_debug) Task a2c429d9-bbaa-4793-b32b-9ea7340a57d5: moving from state preparing -> state finished >Thread-15910626::DEBUG::2013-11-19 21:16:46,063::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910626::DEBUG::2013-11-19 21:16:46,063::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910626::DEBUG::2013-11-19 21:16:46,064::task::495::TaskManager.Task::(_debug) Task a2c429d9-bbaa-4793-b32b-9ea7340a57d5: ref 0 aborting False >Thread-15910626::INFO::2013-11-19 21:16:46,065::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910627::DEBUG::2013-11-19 21:16:46,069::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910627::INFO::2013-11-19 21:16:46,070::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910627::DEBUG::2013-11-19 21:16:46,071::task::495::TaskManager.Task::(_debug) Task 35f17e52-73ef-4be7-aa04-e9b4479df747: moving from state init -> state preparing >Thread-15910627::DEBUG::2013-11-19 21:16:46,072::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`09a913e6-1961-4c59-bbd0-fb24793819f9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910627::DEBUG::2013-11-19 21:16:46,073::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910627::DEBUG::2013-11-19 21:16:46,073::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910627::DEBUG::2013-11-19 21:16:46,074::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`09a913e6-1961-4c59-bbd0-fb24793819f9`::Granted request >Thread-15910627::DEBUG::2013-11-19 21:16:46,075::task::495::TaskManager.Task::(_debug) Task 35f17e52-73ef-4be7-aa04-e9b4479df747: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910627::DEBUG::2013-11-19 21:16:46,076::task::495::TaskManager.Task::(_debug) Task 35f17e52-73ef-4be7-aa04-e9b4479df747: ref 1 aborting False >Thread-15910627::WARNING::2013-11-19 21:16:46,078::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910627::DEBUG::2013-11-19 21:16:46,079::task::495::TaskManager.Task::(_debug) Task 35f17e52-73ef-4be7-aa04-e9b4479df747: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910627::DEBUG::2013-11-19 21:16:46,079::task::495::TaskManager.Task::(_debug) Task 35f17e52-73ef-4be7-aa04-e9b4479df747: moving from state preparing -> state finished >Thread-15910627::DEBUG::2013-11-19 21:16:46,080::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910627::DEBUG::2013-11-19 21:16:46,080::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910627::DEBUG::2013-11-19 21:16:46,081::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910627::DEBUG::2013-11-19 21:16:46,082::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910627::DEBUG::2013-11-19 21:16:46,082::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910627::DEBUG::2013-11-19 21:16:46,083::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910627::DEBUG::2013-11-19 21:16:46,083::task::495::TaskManager.Task::(_debug) Task 35f17e52-73ef-4be7-aa04-e9b4479df747: ref 0 aborting False >Thread-15910627::INFO::2013-11-19 21:16:46,084::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910628::INFO::2013-11-19 21:16:47,414::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910628::DEBUG::2013-11-19 21:16:47,414::task::495::TaskManager.Task::(_debug) Task afec0397-2d43-45c3-bcaf-1c01c6afc652: moving from state init -> state preparing >Thread-15910628::DEBUG::2013-11-19 21:16:47,415::task::495::TaskManager.Task::(_debug) Task afec0397-2d43-45c3-bcaf-1c01c6afc652: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000419139862061', 'lastCheck': 1384895798.18977, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00225496292114', 'lastCheck': 1384895800.338099, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895805.4073839, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00051212310791', 'lastCheck': 1384895804.3251081, 'valid': True, 'code': 0}} >Thread-15910628::DEBUG::2013-11-19 21:16:47,416::task::495::TaskManager.Task::(_debug) Task afec0397-2d43-45c3-bcaf-1c01c6afc652: moving from state preparing -> state finished >Thread-15910628::DEBUG::2013-11-19 21:16:47,416::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910628::DEBUG::2013-11-19 21:16:47,417::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910628::DEBUG::2013-11-19 21:16:47,418::task::495::TaskManager.Task::(_debug) Task afec0397-2d43-45c3-bcaf-1c01c6afc652: ref 0 aborting False >Thread-15910628::INFO::2013-11-19 21:16:47,418::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000419139862061', 'lastCheck': 1384895798.18977, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00225496292114', 'lastCheck': 1384895800.338099, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895805.4073839, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00051212310791', 'lastCheck': 1384895804.3251081, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:16:47,725::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:16:48,251::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.070392 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:16:50,266::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:16:50,786::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0703785 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:16:52,802::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:16:53,353::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.071789 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:16:55,368::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:16:55,602::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15910548::DEBUG::2013-11-19 21:16:55,890::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0707244 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15910634::DEBUG::2013-11-19 21:16:56,190::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910634::INFO::2013-11-19 21:16:56,191::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910634::DEBUG::2013-11-19 21:16:56,192::task::495::TaskManager.Task::(_debug) Task 4b052b01-732f-4fda-a0e1-33b65449a4ba: moving from state init -> state preparing >Thread-15910634::DEBUG::2013-11-19 21:16:56,194::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910634::DEBUG::2013-11-19 21:16:56,195::task::495::TaskManager.Task::(_debug) Task 4b052b01-732f-4fda-a0e1-33b65449a4ba: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910634::DEBUG::2013-11-19 21:16:56,196::task::495::TaskManager.Task::(_debug) Task 4b052b01-732f-4fda-a0e1-33b65449a4ba: moving from state preparing -> state finished >Thread-15910634::DEBUG::2013-11-19 21:16:56,197::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910634::DEBUG::2013-11-19 21:16:56,197::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910634::DEBUG::2013-11-19 21:16:56,198::task::495::TaskManager.Task::(_debug) Task 4b052b01-732f-4fda-a0e1-33b65449a4ba: ref 0 aborting False >Thread-15910634::INFO::2013-11-19 21:16:56,198::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910635::DEBUG::2013-11-19 21:16:56,203::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910635::INFO::2013-11-19 21:16:56,204::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910635::DEBUG::2013-11-19 21:16:56,205::task::495::TaskManager.Task::(_debug) Task 50ac24df-7d97-4689-b865-62ea27e93e91: moving from state init -> state preparing >Thread-15910635::DEBUG::2013-11-19 21:16:56,206::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d1d2f717-9fdf-4e2b-a6ce-797e40fd931c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910635::DEBUG::2013-11-19 21:16:56,206::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910635::DEBUG::2013-11-19 21:16:56,207::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910635::DEBUG::2013-11-19 21:16:56,208::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d1d2f717-9fdf-4e2b-a6ce-797e40fd931c`::Granted request >Thread-15910635::DEBUG::2013-11-19 21:16:56,209::task::495::TaskManager.Task::(_debug) Task 50ac24df-7d97-4689-b865-62ea27e93e91: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910635::DEBUG::2013-11-19 21:16:56,209::task::495::TaskManager.Task::(_debug) Task 50ac24df-7d97-4689-b865-62ea27e93e91: ref 1 aborting False >Thread-15910635::WARNING::2013-11-19 21:16:56,211::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910635::DEBUG::2013-11-19 21:16:56,212::task::495::TaskManager.Task::(_debug) Task 50ac24df-7d97-4689-b865-62ea27e93e91: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910635::DEBUG::2013-11-19 21:16:56,213::task::495::TaskManager.Task::(_debug) Task 50ac24df-7d97-4689-b865-62ea27e93e91: moving from state preparing -> state finished >Thread-15910635::DEBUG::2013-11-19 21:16:56,214::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910635::DEBUG::2013-11-19 21:16:56,214::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910635::DEBUG::2013-11-19 21:16:56,215::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910635::DEBUG::2013-11-19 21:16:56,216::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910635::DEBUG::2013-11-19 21:16:56,216::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910635::DEBUG::2013-11-19 21:16:56,217::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910635::DEBUG::2013-11-19 21:16:56,217::task::495::TaskManager.Task::(_debug) Task 50ac24df-7d97-4689-b865-62ea27e93e91: ref 0 aborting False >Thread-15910635::INFO::2013-11-19 21:16:56,218::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910636::INFO::2013-11-19 21:16:57,613::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910636::DEBUG::2013-11-19 21:16:57,614::task::495::TaskManager.Task::(_debug) Task 4f29e1a0-a441-43fe-8f1f-b82cce5cc745: moving from state init -> state preparing >Thread-15910636::DEBUG::2013-11-19 21:16:57,614::task::495::TaskManager.Task::(_debug) Task 4f29e1a0-a441-43fe-8f1f-b82cce5cc745: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000356912612915', 'lastCheck': 1384895808.401027, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00161600112915', 'lastCheck': 1384895810.4942491, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895815.6064429, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000406980514526', 'lastCheck': 1384895814.491873, 'valid': True, 'code': 0}} >Thread-15910636::DEBUG::2013-11-19 21:16:57,615::task::495::TaskManager.Task::(_debug) Task 4f29e1a0-a441-43fe-8f1f-b82cce5cc745: moving from state preparing -> state finished >Thread-15910636::DEBUG::2013-11-19 21:16:57,616::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910636::DEBUG::2013-11-19 21:16:57,616::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910636::DEBUG::2013-11-19 21:16:57,617::task::495::TaskManager.Task::(_debug) Task 4f29e1a0-a441-43fe-8f1f-b82cce5cc745: ref 0 aborting False >Thread-15910636::INFO::2013-11-19 21:16:57,617::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000356912612915', 'lastCheck': 1384895808.401027, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00161600112915', 'lastCheck': 1384895810.4942491, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895815.6064429, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000406980514526', 'lastCheck': 1384895814.491873, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:16:57,906::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:16:58,453::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0713689 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:17:00,468::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:17:00,985::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0680891 s, 15.0 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:17:03,000::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:17:03,517::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0705446 s, 14.5 MB/s\n'; <rc> = 0 >Thread-42840::DEBUG::2013-11-19 21:17:04,679::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 21:17:05,366::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:17:05,533::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:17:05,765::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15910548::DEBUG::2013-11-19 21:17:06,081::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0714527 s, 14.3 MB/s\n'; <rc> = 0 >Thread-15910642::DEBUG::2013-11-19 21:17:06,339::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910642::INFO::2013-11-19 21:17:06,340::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910642::DEBUG::2013-11-19 21:17:06,341::task::495::TaskManager.Task::(_debug) Task c83301f2-645b-4044-aaf6-aaae537cb59a: moving from state init -> state preparing >Thread-15910642::DEBUG::2013-11-19 21:17:06,342::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910642::DEBUG::2013-11-19 21:17:06,342::task::495::TaskManager.Task::(_debug) Task c83301f2-645b-4044-aaf6-aaae537cb59a: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910642::DEBUG::2013-11-19 21:17:06,343::task::495::TaskManager.Task::(_debug) Task c83301f2-645b-4044-aaf6-aaae537cb59a: moving from state preparing -> state finished >Thread-15910642::DEBUG::2013-11-19 21:17:06,343::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910642::DEBUG::2013-11-19 21:17:06,344::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910642::DEBUG::2013-11-19 21:17:06,344::task::495::TaskManager.Task::(_debug) Task c83301f2-645b-4044-aaf6-aaae537cb59a: ref 0 aborting False >Thread-15910642::INFO::2013-11-19 21:17:06,345::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910643::DEBUG::2013-11-19 21:17:06,350::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910643::INFO::2013-11-19 21:17:06,351::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910643::DEBUG::2013-11-19 21:17:06,352::task::495::TaskManager.Task::(_debug) Task cc57e2a2-d62a-4125-a63b-e55356b9b47b: moving from state init -> state preparing >Thread-15910643::DEBUG::2013-11-19 21:17:06,353::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`312bbdf6-0e0d-445d-a2a8-4a4ba538b42a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910643::DEBUG::2013-11-19 21:17:06,353::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910643::DEBUG::2013-11-19 21:17:06,354::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910643::DEBUG::2013-11-19 21:17:06,355::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`312bbdf6-0e0d-445d-a2a8-4a4ba538b42a`::Granted request >Thread-15910643::DEBUG::2013-11-19 21:17:06,356::task::495::TaskManager.Task::(_debug) Task cc57e2a2-d62a-4125-a63b-e55356b9b47b: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910643::DEBUG::2013-11-19 21:17:06,356::task::495::TaskManager.Task::(_debug) Task cc57e2a2-d62a-4125-a63b-e55356b9b47b: ref 1 aborting False >Thread-15910643::WARNING::2013-11-19 21:17:06,358::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910643::DEBUG::2013-11-19 21:17:06,359::task::495::TaskManager.Task::(_debug) Task cc57e2a2-d62a-4125-a63b-e55356b9b47b: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910643::DEBUG::2013-11-19 21:17:06,360::task::495::TaskManager.Task::(_debug) Task cc57e2a2-d62a-4125-a63b-e55356b9b47b: moving from state preparing -> state finished >Thread-15910643::DEBUG::2013-11-19 21:17:06,361::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910643::DEBUG::2013-11-19 21:17:06,361::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910643::DEBUG::2013-11-19 21:17:06,362::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910643::DEBUG::2013-11-19 21:17:06,362::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910643::DEBUG::2013-11-19 21:17:06,363::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910643::DEBUG::2013-11-19 21:17:06,364::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910643::DEBUG::2013-11-19 21:17:06,364::task::495::TaskManager.Task::(_debug) Task cc57e2a2-d62a-4125-a63b-e55356b9b47b: ref 0 aborting False >Thread-15910643::INFO::2013-11-19 21:17:06,365::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910644::INFO::2013-11-19 21:17:07,801::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910644::DEBUG::2013-11-19 21:17:07,802::task::495::TaskManager.Task::(_debug) Task f4a3f23b-c005-4622-99bd-521fef818601: moving from state init -> state preparing >Thread-15910644::DEBUG::2013-11-19 21:17:07,803::task::495::TaskManager.Task::(_debug) Task f4a3f23b-c005-4622-99bd-521fef818601: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00232195854187', 'lastCheck': 1384895818.5982511, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00161409378052', 'lastCheck': 1384895820.683538, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895825.767473, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000328063964844', 'lastCheck': 1384895825.3684521, 'valid': True, 'code': 0}} >Thread-15910644::DEBUG::2013-11-19 21:17:07,804::task::495::TaskManager.Task::(_debug) Task f4a3f23b-c005-4622-99bd-521fef818601: moving from state preparing -> state finished >Thread-15910644::DEBUG::2013-11-19 21:17:07,804::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910644::DEBUG::2013-11-19 21:17:07,805::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910644::DEBUG::2013-11-19 21:17:07,805::task::495::TaskManager.Task::(_debug) Task f4a3f23b-c005-4622-99bd-521fef818601: ref 0 aborting False >Thread-15910644::INFO::2013-11-19 21:17:07,806::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00232195854187', 'lastCheck': 1384895818.5982511, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00161409378052', 'lastCheck': 1384895820.683538, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895825.767473, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000328063964844', 'lastCheck': 1384895825.3684521, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:17:08,096::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:17:08,630::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0707999 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:17:10,645::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:17:11,215::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0932177 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:17:13,230::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:17:13,760::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0704161 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:17:15,777::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:17:16,014::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15910548::DEBUG::2013-11-19 21:17:16,322::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.071351 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15910650::DEBUG::2013-11-19 21:17:16,480::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910650::INFO::2013-11-19 21:17:16,482::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910650::DEBUG::2013-11-19 21:17:16,482::task::495::TaskManager.Task::(_debug) Task 14cb84da-7b59-4851-a88e-89a5aa76e7db: moving from state init -> state preparing >Thread-15910650::DEBUG::2013-11-19 21:17:16,483::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910650::DEBUG::2013-11-19 21:17:16,484::task::495::TaskManager.Task::(_debug) Task 14cb84da-7b59-4851-a88e-89a5aa76e7db: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910650::DEBUG::2013-11-19 21:17:16,484::task::495::TaskManager.Task::(_debug) Task 14cb84da-7b59-4851-a88e-89a5aa76e7db: moving from state preparing -> state finished >Thread-15910650::DEBUG::2013-11-19 21:17:16,485::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910650::DEBUG::2013-11-19 21:17:16,485::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910650::DEBUG::2013-11-19 21:17:16,486::task::495::TaskManager.Task::(_debug) Task 14cb84da-7b59-4851-a88e-89a5aa76e7db: ref 0 aborting False >Thread-15910650::INFO::2013-11-19 21:17:16,487::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910651::DEBUG::2013-11-19 21:17:16,492::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910651::INFO::2013-11-19 21:17:16,493::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910651::DEBUG::2013-11-19 21:17:16,493::task::495::TaskManager.Task::(_debug) Task 038c4f13-e1a7-4962-8455-cf339f1441bf: moving from state init -> state preparing >Thread-15910651::DEBUG::2013-11-19 21:17:16,495::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3f3a50fd-7c17-4e40-80d9-f0379448f2ef`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910651::DEBUG::2013-11-19 21:17:16,495::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910651::DEBUG::2013-11-19 21:17:16,496::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910651::DEBUG::2013-11-19 21:17:16,497::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3f3a50fd-7c17-4e40-80d9-f0379448f2ef`::Granted request >Thread-15910651::DEBUG::2013-11-19 21:17:16,498::task::495::TaskManager.Task::(_debug) Task 038c4f13-e1a7-4962-8455-cf339f1441bf: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910651::DEBUG::2013-11-19 21:17:16,498::task::495::TaskManager.Task::(_debug) Task 038c4f13-e1a7-4962-8455-cf339f1441bf: ref 1 aborting False >Thread-15910651::WARNING::2013-11-19 21:17:16,500::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910651::DEBUG::2013-11-19 21:17:16,501::task::495::TaskManager.Task::(_debug) Task 038c4f13-e1a7-4962-8455-cf339f1441bf: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910651::DEBUG::2013-11-19 21:17:16,502::task::495::TaskManager.Task::(_debug) Task 038c4f13-e1a7-4962-8455-cf339f1441bf: moving from state preparing -> state finished >Thread-15910651::DEBUG::2013-11-19 21:17:16,502::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910651::DEBUG::2013-11-19 21:17:16,503::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910651::DEBUG::2013-11-19 21:17:16,504::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910651::DEBUG::2013-11-19 21:17:16,505::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910651::DEBUG::2013-11-19 21:17:16,505::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910651::DEBUG::2013-11-19 21:17:16,506::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910651::DEBUG::2013-11-19 21:17:16,506::task::495::TaskManager.Task::(_debug) Task 038c4f13-e1a7-4962-8455-cf339f1441bf: ref 0 aborting False >Thread-15910651::INFO::2013-11-19 21:17:16,507::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910652::INFO::2013-11-19 21:17:18,023::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910652::DEBUG::2013-11-19 21:17:18,024::task::495::TaskManager.Task::(_debug) Task b8a8f579-8822-4783-98e5-63ce3e9bcc58: moving from state init -> state preparing >Thread-15910652::DEBUG::2013-11-19 21:17:18,024::task::495::TaskManager.Task::(_debug) Task b8a8f579-8822-4783-98e5-63ce3e9bcc58: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000414133071899', 'lastCheck': 1384895828.7669859, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00159597396851', 'lastCheck': 1384895830.9146409, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895836.016191, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000412940979004', 'lastCheck': 1384895835.44748, 'valid': True, 'code': 0}} >Thread-15910652::DEBUG::2013-11-19 21:17:18,025::task::495::TaskManager.Task::(_debug) Task b8a8f579-8822-4783-98e5-63ce3e9bcc58: moving from state preparing -> state finished >Thread-15910652::DEBUG::2013-11-19 21:17:18,026::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910652::DEBUG::2013-11-19 21:17:18,026::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910652::DEBUG::2013-11-19 21:17:18,027::task::495::TaskManager.Task::(_debug) Task b8a8f579-8822-4783-98e5-63ce3e9bcc58: ref 0 aborting False >Thread-15910652::INFO::2013-11-19 21:17:18,027::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000414133071899', 'lastCheck': 1384895828.7669859, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00159597396851', 'lastCheck': 1384895830.9146409, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895836.016191, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000412940979004', 'lastCheck': 1384895835.44748, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:17:18,337::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:17:18,864::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.071148 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:17:20,880::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:17:21,391::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0705971 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:17:23,406::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:17:23,952::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0716661 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:17:25,967::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:17:26,238::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15910548::DEBUG::2013-11-19 21:17:26,499::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.071093 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15910658::DEBUG::2013-11-19 21:17:26,620::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910658::INFO::2013-11-19 21:17:26,621::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910658::DEBUG::2013-11-19 21:17:26,622::task::495::TaskManager.Task::(_debug) Task 6664076c-33fb-4e03-939a-ea5340a974a8: moving from state init -> state preparing >Thread-15910658::DEBUG::2013-11-19 21:17:26,623::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910658::DEBUG::2013-11-19 21:17:26,623::task::495::TaskManager.Task::(_debug) Task 6664076c-33fb-4e03-939a-ea5340a974a8: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910658::DEBUG::2013-11-19 21:17:26,624::task::495::TaskManager.Task::(_debug) Task 6664076c-33fb-4e03-939a-ea5340a974a8: moving from state preparing -> state finished >Thread-15910658::DEBUG::2013-11-19 21:17:26,624::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910658::DEBUG::2013-11-19 21:17:26,625::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910658::DEBUG::2013-11-19 21:17:26,626::task::495::TaskManager.Task::(_debug) Task 6664076c-33fb-4e03-939a-ea5340a974a8: ref 0 aborting False >Thread-15910658::INFO::2013-11-19 21:17:26,626::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910659::DEBUG::2013-11-19 21:17:26,631::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910659::INFO::2013-11-19 21:17:26,632::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910659::DEBUG::2013-11-19 21:17:26,633::task::495::TaskManager.Task::(_debug) Task 91419e3c-3b5c-4fe8-b1a4-1968e95829e6: moving from state init -> state preparing >Thread-15910659::DEBUG::2013-11-19 21:17:26,634::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f68ce179-2830-49e2-aefc-1268b3977925`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910659::DEBUG::2013-11-19 21:17:26,635::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910659::DEBUG::2013-11-19 21:17:26,635::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910659::DEBUG::2013-11-19 21:17:26,636::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f68ce179-2830-49e2-aefc-1268b3977925`::Granted request >Thread-15910659::DEBUG::2013-11-19 21:17:26,637::task::495::TaskManager.Task::(_debug) Task 91419e3c-3b5c-4fe8-b1a4-1968e95829e6: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910659::DEBUG::2013-11-19 21:17:26,638::task::495::TaskManager.Task::(_debug) Task 91419e3c-3b5c-4fe8-b1a4-1968e95829e6: ref 1 aborting False >Thread-15910659::WARNING::2013-11-19 21:17:26,640::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910659::DEBUG::2013-11-19 21:17:26,641::task::495::TaskManager.Task::(_debug) Task 91419e3c-3b5c-4fe8-b1a4-1968e95829e6: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910659::DEBUG::2013-11-19 21:17:26,641::task::495::TaskManager.Task::(_debug) Task 91419e3c-3b5c-4fe8-b1a4-1968e95829e6: moving from state preparing -> state finished >Thread-15910659::DEBUG::2013-11-19 21:17:26,642::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910659::DEBUG::2013-11-19 21:17:26,643::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910659::DEBUG::2013-11-19 21:17:26,643::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910659::DEBUG::2013-11-19 21:17:26,644::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910659::DEBUG::2013-11-19 21:17:26,644::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910659::DEBUG::2013-11-19 21:17:26,645::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910659::DEBUG::2013-11-19 21:17:26,646::task::495::TaskManager.Task::(_debug) Task 91419e3c-3b5c-4fe8-b1a4-1968e95829e6: ref 0 aborting False >Thread-15910659::INFO::2013-11-19 21:17:26,646::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910660::INFO::2013-11-19 21:17:28,249::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910660::DEBUG::2013-11-19 21:17:28,249::task::495::TaskManager.Task::(_debug) Task 72d1de26-d6da-44e8-a5d7-768435acd2c6: moving from state init -> state preparing >Thread-15910660::DEBUG::2013-11-19 21:17:28,250::task::495::TaskManager.Task::(_debug) Task 72d1de26-d6da-44e8-a5d7-768435acd2c6: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000483989715576', 'lastCheck': 1384895839.010541, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00164389610291', 'lastCheck': 1384895841.0841999, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895846.242121, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0003981590271', 'lastCheck': 1384895845.6361229, 'valid': True, 'code': 0}} >Thread-15910660::DEBUG::2013-11-19 21:17:28,251::task::495::TaskManager.Task::(_debug) Task 72d1de26-d6da-44e8-a5d7-768435acd2c6: moving from state preparing -> state finished >Thread-15910660::DEBUG::2013-11-19 21:17:28,251::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910660::DEBUG::2013-11-19 21:17:28,252::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910660::DEBUG::2013-11-19 21:17:28,252::task::495::TaskManager.Task::(_debug) Task 72d1de26-d6da-44e8-a5d7-768435acd2c6: ref 0 aborting False >Thread-15910660::INFO::2013-11-19 21:17:28,253::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000483989715576', 'lastCheck': 1384895839.010541, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00164389610291', 'lastCheck': 1384895841.0841999, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895846.242121, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0003981590271', 'lastCheck': 1384895845.6361229, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:17:28,514::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:17:29,041::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0707649 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:17:31,057::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:17:31,591::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0711923 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:17:33,607::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:17:34,123::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0711648 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:17:36,138::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:17:36,368::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15910548::DEBUG::2013-11-19 21:17:36,667::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0703491 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15910666::DEBUG::2013-11-19 21:17:36,769::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910666::INFO::2013-11-19 21:17:36,770::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910666::DEBUG::2013-11-19 21:17:36,771::task::495::TaskManager.Task::(_debug) Task 1c5429f7-bed2-44ee-bcb6-031022411ec4: moving from state init -> state preparing >Thread-15910666::DEBUG::2013-11-19 21:17:36,772::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910666::DEBUG::2013-11-19 21:17:36,772::task::495::TaskManager.Task::(_debug) Task 1c5429f7-bed2-44ee-bcb6-031022411ec4: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910666::DEBUG::2013-11-19 21:17:36,773::task::495::TaskManager.Task::(_debug) Task 1c5429f7-bed2-44ee-bcb6-031022411ec4: moving from state preparing -> state finished >Thread-15910666::DEBUG::2013-11-19 21:17:36,774::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910666::DEBUG::2013-11-19 21:17:36,774::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910666::DEBUG::2013-11-19 21:17:36,775::task::495::TaskManager.Task::(_debug) Task 1c5429f7-bed2-44ee-bcb6-031022411ec4: ref 0 aborting False >Thread-15910666::INFO::2013-11-19 21:17:36,775::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910667::DEBUG::2013-11-19 21:17:36,780::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910667::INFO::2013-11-19 21:17:36,781::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910667::DEBUG::2013-11-19 21:17:36,782::task::495::TaskManager.Task::(_debug) Task aa47e312-4ffa-4e0f-8c75-703a9f557fdc: moving from state init -> state preparing >Thread-15910667::DEBUG::2013-11-19 21:17:36,784::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`086902f1-f9cf-41db-8df9-0520be01c4bc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910667::DEBUG::2013-11-19 21:17:36,785::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910667::DEBUG::2013-11-19 21:17:36,786::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910667::DEBUG::2013-11-19 21:17:36,787::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`086902f1-f9cf-41db-8df9-0520be01c4bc`::Granted request >Thread-15910667::DEBUG::2013-11-19 21:17:36,797::task::495::TaskManager.Task::(_debug) Task aa47e312-4ffa-4e0f-8c75-703a9f557fdc: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910667::DEBUG::2013-11-19 21:17:36,798::task::495::TaskManager.Task::(_debug) Task aa47e312-4ffa-4e0f-8c75-703a9f557fdc: ref 1 aborting False >Thread-15910667::WARNING::2013-11-19 21:17:36,801::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910667::DEBUG::2013-11-19 21:17:36,802::task::495::TaskManager.Task::(_debug) Task aa47e312-4ffa-4e0f-8c75-703a9f557fdc: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910667::DEBUG::2013-11-19 21:17:36,803::task::495::TaskManager.Task::(_debug) Task aa47e312-4ffa-4e0f-8c75-703a9f557fdc: moving from state preparing -> state finished >Thread-15910667::DEBUG::2013-11-19 21:17:36,804::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910667::DEBUG::2013-11-19 21:17:36,804::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910667::DEBUG::2013-11-19 21:17:36,806::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910667::DEBUG::2013-11-19 21:17:36,806::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910667::DEBUG::2013-11-19 21:17:36,807::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910667::DEBUG::2013-11-19 21:17:36,808::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910667::DEBUG::2013-11-19 21:17:36,809::task::495::TaskManager.Task::(_debug) Task aa47e312-4ffa-4e0f-8c75-703a9f557fdc: ref 0 aborting False >Thread-15910667::INFO::2013-11-19 21:17:36,810::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910668::INFO::2013-11-19 21:17:38,380::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910668::DEBUG::2013-11-19 21:17:38,381::task::495::TaskManager.Task::(_debug) Task 939593e5-bfef-48ea-8833-4aa4f808e03a: moving from state init -> state preparing >Thread-15910668::DEBUG::2013-11-19 21:17:38,382::task::495::TaskManager.Task::(_debug) Task 939593e5-bfef-48ea-8833-4aa4f808e03a: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00218486785889', 'lastCheck': 1384895849.2401719, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00158596038818', 'lastCheck': 1384895851.2864039, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895856.3723309, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000452995300293', 'lastCheck': 1384895855.812453, 'valid': True, 'code': 0}} >Thread-15910668::DEBUG::2013-11-19 21:17:38,382::task::495::TaskManager.Task::(_debug) Task 939593e5-bfef-48ea-8833-4aa4f808e03a: moving from state preparing -> state finished >Thread-15910668::DEBUG::2013-11-19 21:17:38,383::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910668::DEBUG::2013-11-19 21:17:38,384::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910668::DEBUG::2013-11-19 21:17:38,384::task::495::TaskManager.Task::(_debug) Task 939593e5-bfef-48ea-8833-4aa4f808e03a: ref 0 aborting False >Thread-15910668::INFO::2013-11-19 21:17:38,385::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00218486785889', 'lastCheck': 1384895849.2401719, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00158596038818', 'lastCheck': 1384895851.2864039, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895856.3723309, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000452995300293', 'lastCheck': 1384895855.812453, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:17:38,682::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:17:39,216::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0712462 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:17:41,231::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:17:41,760::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0705107 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:17:43,776::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:17:44,340::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0940987 s, 10.9 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:17:46,355::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:17:46,586::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15910548::DEBUG::2013-11-19 21:17:46,887::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0775026 s, 13.2 MB/s\n'; <rc> = 0 >Thread-15910674::DEBUG::2013-11-19 21:17:46,942::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910674::INFO::2013-11-19 21:17:46,943::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910674::DEBUG::2013-11-19 21:17:46,944::task::495::TaskManager.Task::(_debug) Task 0b827f33-9dd6-4334-ad80-8c5326f36efe: moving from state init -> state preparing >Thread-15910674::DEBUG::2013-11-19 21:17:46,945::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910674::DEBUG::2013-11-19 21:17:46,945::task::495::TaskManager.Task::(_debug) Task 0b827f33-9dd6-4334-ad80-8c5326f36efe: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910674::DEBUG::2013-11-19 21:17:46,946::task::495::TaskManager.Task::(_debug) Task 0b827f33-9dd6-4334-ad80-8c5326f36efe: moving from state preparing -> state finished >Thread-15910674::DEBUG::2013-11-19 21:17:46,947::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910674::DEBUG::2013-11-19 21:17:46,947::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910674::DEBUG::2013-11-19 21:17:46,948::task::495::TaskManager.Task::(_debug) Task 0b827f33-9dd6-4334-ad80-8c5326f36efe: ref 0 aborting False >Thread-15910674::INFO::2013-11-19 21:17:46,948::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910675::DEBUG::2013-11-19 21:17:46,953::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910675::INFO::2013-11-19 21:17:46,954::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910675::DEBUG::2013-11-19 21:17:46,955::task::495::TaskManager.Task::(_debug) Task 709fe604-d374-4163-8491-3270198aa6ba: moving from state init -> state preparing >Thread-15910675::DEBUG::2013-11-19 21:17:46,956::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c4b4ac7c-7feb-4713-b494-12d9772f2c22`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910675::DEBUG::2013-11-19 21:17:46,956::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910675::DEBUG::2013-11-19 21:17:46,957::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910675::DEBUG::2013-11-19 21:17:46,958::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c4b4ac7c-7feb-4713-b494-12d9772f2c22`::Granted request >Thread-15910675::DEBUG::2013-11-19 21:17:46,959::task::495::TaskManager.Task::(_debug) Task 709fe604-d374-4163-8491-3270198aa6ba: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910675::DEBUG::2013-11-19 21:17:46,959::task::495::TaskManager.Task::(_debug) Task 709fe604-d374-4163-8491-3270198aa6ba: ref 1 aborting False >Thread-15910675::WARNING::2013-11-19 21:17:46,961::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910675::DEBUG::2013-11-19 21:17:46,962::task::495::TaskManager.Task::(_debug) Task 709fe604-d374-4163-8491-3270198aa6ba: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910675::DEBUG::2013-11-19 21:17:46,963::task::495::TaskManager.Task::(_debug) Task 709fe604-d374-4163-8491-3270198aa6ba: moving from state preparing -> state finished >Thread-15910675::DEBUG::2013-11-19 21:17:46,964::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910675::DEBUG::2013-11-19 21:17:46,964::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910675::DEBUG::2013-11-19 21:17:46,965::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910675::DEBUG::2013-11-19 21:17:46,966::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910675::DEBUG::2013-11-19 21:17:46,966::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910675::DEBUG::2013-11-19 21:17:46,967::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910675::DEBUG::2013-11-19 21:17:46,967::task::495::TaskManager.Task::(_debug) Task 709fe604-d374-4163-8491-3270198aa6ba: ref 0 aborting False >Thread-15910675::INFO::2013-11-19 21:17:46,968::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910676::INFO::2013-11-19 21:17:48,595::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910676::DEBUG::2013-11-19 21:17:48,596::task::495::TaskManager.Task::(_debug) Task 83574dfb-3557-463a-8353-590835da7f57: moving from state init -> state preparing >Thread-15910676::DEBUG::2013-11-19 21:17:48,596::task::495::TaskManager.Task::(_debug) Task 83574dfb-3557-463a-8353-590835da7f57: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000460863113403', 'lastCheck': 1384895859.3681841, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00215697288513', 'lastCheck': 1384895861.485774, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895866.5877221, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000427007675171', 'lastCheck': 1384895866.014514, 'valid': True, 'code': 0}} >Thread-15910676::DEBUG::2013-11-19 21:17:48,597::task::495::TaskManager.Task::(_debug) Task 83574dfb-3557-463a-8353-590835da7f57: moving from state preparing -> state finished >Thread-15910676::DEBUG::2013-11-19 21:17:48,598::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910676::DEBUG::2013-11-19 21:17:48,598::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910676::DEBUG::2013-11-19 21:17:48,599::task::495::TaskManager.Task::(_debug) Task 83574dfb-3557-463a-8353-590835da7f57: ref 0 aborting False >Thread-15910676::INFO::2013-11-19 21:17:48,599::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000460863113403', 'lastCheck': 1384895859.3681841, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00215697288513', 'lastCheck': 1384895861.485774, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895866.5877221, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000427007675171', 'lastCheck': 1384895866.014514, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:17:48,902::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:17:49,443::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0713667 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:17:51,459::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:17:51,981::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0707233 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:17:53,996::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:17:54,506::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0703108 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:17:56,521::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:17:56,757::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15910548::DEBUG::2013-11-19 21:17:57,061::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0713639 s, 14.3 MB/s\n'; <rc> = 0 >Thread-15910682::DEBUG::2013-11-19 21:17:57,109::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910682::INFO::2013-11-19 21:17:57,110::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910682::DEBUG::2013-11-19 21:17:57,111::task::495::TaskManager.Task::(_debug) Task 42079000-a12b-41fb-8ddc-7d8f10d37098: moving from state init -> state preparing >Thread-15910682::DEBUG::2013-11-19 21:17:57,111::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910682::DEBUG::2013-11-19 21:17:57,112::task::495::TaskManager.Task::(_debug) Task 42079000-a12b-41fb-8ddc-7d8f10d37098: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910682::DEBUG::2013-11-19 21:17:57,113::task::495::TaskManager.Task::(_debug) Task 42079000-a12b-41fb-8ddc-7d8f10d37098: moving from state preparing -> state finished >Thread-15910682::DEBUG::2013-11-19 21:17:57,113::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910682::DEBUG::2013-11-19 21:17:57,114::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910682::DEBUG::2013-11-19 21:17:57,114::task::495::TaskManager.Task::(_debug) Task 42079000-a12b-41fb-8ddc-7d8f10d37098: ref 0 aborting False >Thread-15910682::INFO::2013-11-19 21:17:57,115::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910683::DEBUG::2013-11-19 21:17:57,120::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910683::INFO::2013-11-19 21:17:57,121::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910683::DEBUG::2013-11-19 21:17:57,121::task::495::TaskManager.Task::(_debug) Task 614a71fb-db5c-4010-b4f3-9e53f4887dd6: moving from state init -> state preparing >Thread-15910683::DEBUG::2013-11-19 21:17:57,123::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bdd6267a-26c9-4d0e-a91d-ed90d54ffe32`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910683::DEBUG::2013-11-19 21:17:57,123::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910683::DEBUG::2013-11-19 21:17:57,124::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910683::DEBUG::2013-11-19 21:17:57,125::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bdd6267a-26c9-4d0e-a91d-ed90d54ffe32`::Granted request >Thread-15910683::DEBUG::2013-11-19 21:17:57,126::task::495::TaskManager.Task::(_debug) Task 614a71fb-db5c-4010-b4f3-9e53f4887dd6: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910683::DEBUG::2013-11-19 21:17:57,126::task::495::TaskManager.Task::(_debug) Task 614a71fb-db5c-4010-b4f3-9e53f4887dd6: ref 1 aborting False >Thread-15910683::WARNING::2013-11-19 21:17:57,128::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910683::DEBUG::2013-11-19 21:17:57,129::task::495::TaskManager.Task::(_debug) Task 614a71fb-db5c-4010-b4f3-9e53f4887dd6: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910683::DEBUG::2013-11-19 21:17:57,130::task::495::TaskManager.Task::(_debug) Task 614a71fb-db5c-4010-b4f3-9e53f4887dd6: moving from state preparing -> state finished >Thread-15910683::DEBUG::2013-11-19 21:17:57,131::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910683::DEBUG::2013-11-19 21:17:57,131::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910683::DEBUG::2013-11-19 21:17:57,132::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910683::DEBUG::2013-11-19 21:17:57,133::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910683::DEBUG::2013-11-19 21:17:57,133::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910683::DEBUG::2013-11-19 21:17:57,134::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910683::DEBUG::2013-11-19 21:17:57,134::task::495::TaskManager.Task::(_debug) Task 614a71fb-db5c-4010-b4f3-9e53f4887dd6: ref 0 aborting False >Thread-15910683::INFO::2013-11-19 21:17:57,135::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910684::INFO::2013-11-19 21:17:58,771::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910684::DEBUG::2013-11-19 21:17:58,772::task::495::TaskManager.Task::(_debug) Task 3663cc8c-83c4-4e10-84bc-618744192e74: moving from state init -> state preparing >Thread-15910684::DEBUG::2013-11-19 21:17:58,772::task::495::TaskManager.Task::(_debug) Task 3663cc8c-83c4-4e10-84bc-618744192e74: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000458002090454', 'lastCheck': 1384895869.588304, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00163602828979', 'lastCheck': 1384895871.6756339, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895876.7621729, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000884056091309', 'lastCheck': 1384895876.197823, 'valid': True, 'code': 0}} >Thread-15910684::DEBUG::2013-11-19 21:17:58,773::task::495::TaskManager.Task::(_debug) Task 3663cc8c-83c4-4e10-84bc-618744192e74: moving from state preparing -> state finished >Thread-15910684::DEBUG::2013-11-19 21:17:58,774::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910684::DEBUG::2013-11-19 21:17:58,774::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910684::DEBUG::2013-11-19 21:17:58,775::task::495::TaskManager.Task::(_debug) Task 3663cc8c-83c4-4e10-84bc-618744192e74: ref 0 aborting False >Thread-15910684::INFO::2013-11-19 21:17:58,775::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000458002090454', 'lastCheck': 1384895869.588304, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00163602828979', 'lastCheck': 1384895871.6756339, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895876.7621729, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000884056091309', 'lastCheck': 1384895876.197823, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:17:59,076::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:17:59,609::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0679936 s, 15.1 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:18:01,625::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:18:02,153::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0709541 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:18:04,168::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:18:04,707::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0730574 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:18:06,722::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:18:06,935::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15910548::DEBUG::2013-11-19 21:18:07,241::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0706304 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15910690::DEBUG::2013-11-19 21:18:07,257::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910690::INFO::2013-11-19 21:18:07,258::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910690::DEBUG::2013-11-19 21:18:07,258::task::495::TaskManager.Task::(_debug) Task e781fdf5-a84b-4839-9694-ee541c43e071: moving from state init -> state preparing >Thread-15910690::DEBUG::2013-11-19 21:18:07,259::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910690::DEBUG::2013-11-19 21:18:07,260::task::495::TaskManager.Task::(_debug) Task e781fdf5-a84b-4839-9694-ee541c43e071: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910690::DEBUG::2013-11-19 21:18:07,260::task::495::TaskManager.Task::(_debug) Task e781fdf5-a84b-4839-9694-ee541c43e071: moving from state preparing -> state finished >Thread-15910690::DEBUG::2013-11-19 21:18:07,261::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910690::DEBUG::2013-11-19 21:18:07,262::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910690::DEBUG::2013-11-19 21:18:07,262::task::495::TaskManager.Task::(_debug) Task e781fdf5-a84b-4839-9694-ee541c43e071: ref 0 aborting False >Thread-15910690::INFO::2013-11-19 21:18:07,263::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910691::DEBUG::2013-11-19 21:18:07,268::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910691::INFO::2013-11-19 21:18:07,269::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910691::DEBUG::2013-11-19 21:18:07,269::task::495::TaskManager.Task::(_debug) Task 7796564e-4e24-4b13-9332-7c81d37d383b: moving from state init -> state preparing >Thread-15910691::DEBUG::2013-11-19 21:18:07,270::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3dd5f24e-ad2e-4112-a3d8-38b23aa91224`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910691::DEBUG::2013-11-19 21:18:07,271::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910691::DEBUG::2013-11-19 21:18:07,272::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910691::DEBUG::2013-11-19 21:18:07,272::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3dd5f24e-ad2e-4112-a3d8-38b23aa91224`::Granted request >Thread-15910691::DEBUG::2013-11-19 21:18:07,273::task::495::TaskManager.Task::(_debug) Task 7796564e-4e24-4b13-9332-7c81d37d383b: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910691::DEBUG::2013-11-19 21:18:07,274::task::495::TaskManager.Task::(_debug) Task 7796564e-4e24-4b13-9332-7c81d37d383b: ref 1 aborting False >Thread-15910691::WARNING::2013-11-19 21:18:07,276::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910691::DEBUG::2013-11-19 21:18:07,277::task::495::TaskManager.Task::(_debug) Task 7796564e-4e24-4b13-9332-7c81d37d383b: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910691::DEBUG::2013-11-19 21:18:07,278::task::495::TaskManager.Task::(_debug) Task 7796564e-4e24-4b13-9332-7c81d37d383b: moving from state preparing -> state finished >Thread-15910691::DEBUG::2013-11-19 21:18:07,278::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910691::DEBUG::2013-11-19 21:18:07,279::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910691::DEBUG::2013-11-19 21:18:07,279::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910691::DEBUG::2013-11-19 21:18:07,280::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910691::DEBUG::2013-11-19 21:18:07,281::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910691::DEBUG::2013-11-19 21:18:07,281::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910691::DEBUG::2013-11-19 21:18:07,282::task::495::TaskManager.Task::(_debug) Task 7796564e-4e24-4b13-9332-7c81d37d383b: ref 0 aborting False >Thread-15910691::INFO::2013-11-19 21:18:07,282::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910692::INFO::2013-11-19 21:18:08,948::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910692::DEBUG::2013-11-19 21:18:08,949::task::495::TaskManager.Task::(_debug) Task bd7f6801-8e0e-4216-a220-732019880dde: moving from state init -> state preparing >Thread-15910692::DEBUG::2013-11-19 21:18:08,949::task::495::TaskManager.Task::(_debug) Task bd7f6801-8e0e-4216-a220-732019880dde: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00040602684021', 'lastCheck': 1384895879.7578211, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00130486488342', 'lastCheck': 1384895881.8657351, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895886.9401901, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000515937805176', 'lastCheck': 1384895886.396148, 'valid': True, 'code': 0}} >Thread-15910692::DEBUG::2013-11-19 21:18:08,950::task::495::TaskManager.Task::(_debug) Task bd7f6801-8e0e-4216-a220-732019880dde: moving from state preparing -> state finished >Thread-15910692::DEBUG::2013-11-19 21:18:08,951::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910692::DEBUG::2013-11-19 21:18:08,951::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910692::DEBUG::2013-11-19 21:18:08,952::task::495::TaskManager.Task::(_debug) Task bd7f6801-8e0e-4216-a220-732019880dde: ref 0 aborting False >Thread-15910692::INFO::2013-11-19 21:18:08,952::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00040602684021', 'lastCheck': 1384895879.7578211, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00130486488342', 'lastCheck': 1384895881.8657351, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895886.9401901, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000515937805176', 'lastCheck': 1384895886.396148, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:18:09,257::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:18:09,764::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0705474 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:18:11,780::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:18:12,323::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0712999 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:18:14,338::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:18:14,825::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0450488 s, 22.7 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:18:16,836::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:18:17,090::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15910548::DEBUG::2013-11-19 21:18:17,399::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0974499 s, 10.5 MB/s\n'; <rc> = 0 >Thread-15910698::DEBUG::2013-11-19 21:18:17,416::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910698::INFO::2013-11-19 21:18:17,417::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910698::DEBUG::2013-11-19 21:18:17,418::task::495::TaskManager.Task::(_debug) Task 07ee5071-88f4-4f64-9a52-6a9be414a7b8: moving from state init -> state preparing >Thread-15910698::DEBUG::2013-11-19 21:18:17,419::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910698::DEBUG::2013-11-19 21:18:17,419::task::495::TaskManager.Task::(_debug) Task 07ee5071-88f4-4f64-9a52-6a9be414a7b8: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910698::DEBUG::2013-11-19 21:18:17,420::task::495::TaskManager.Task::(_debug) Task 07ee5071-88f4-4f64-9a52-6a9be414a7b8: moving from state preparing -> state finished >Thread-15910698::DEBUG::2013-11-19 21:18:17,420::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910698::DEBUG::2013-11-19 21:18:17,421::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910698::DEBUG::2013-11-19 21:18:17,421::task::495::TaskManager.Task::(_debug) Task 07ee5071-88f4-4f64-9a52-6a9be414a7b8: ref 0 aborting False >Thread-15910698::INFO::2013-11-19 21:18:17,422::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910699::DEBUG::2013-11-19 21:18:17,427::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910699::INFO::2013-11-19 21:18:17,428::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910699::DEBUG::2013-11-19 21:18:17,428::task::495::TaskManager.Task::(_debug) Task 0e0894ec-3222-4ed0-b9ef-913ba2a482a9: moving from state init -> state preparing >Thread-15910699::DEBUG::2013-11-19 21:18:17,429::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`84745321-fbe7-40a7-af5d-cac3eb3a1257`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910699::DEBUG::2013-11-19 21:18:17,430::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910699::DEBUG::2013-11-19 21:18:17,431::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910699::DEBUG::2013-11-19 21:18:17,431::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`84745321-fbe7-40a7-af5d-cac3eb3a1257`::Granted request >Thread-15910699::DEBUG::2013-11-19 21:18:17,432::task::495::TaskManager.Task::(_debug) Task 0e0894ec-3222-4ed0-b9ef-913ba2a482a9: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910699::DEBUG::2013-11-19 21:18:17,433::task::495::TaskManager.Task::(_debug) Task 0e0894ec-3222-4ed0-b9ef-913ba2a482a9: ref 1 aborting False >Thread-15910699::WARNING::2013-11-19 21:18:17,435::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910699::DEBUG::2013-11-19 21:18:17,436::task::495::TaskManager.Task::(_debug) Task 0e0894ec-3222-4ed0-b9ef-913ba2a482a9: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910699::DEBUG::2013-11-19 21:18:17,437::task::495::TaskManager.Task::(_debug) Task 0e0894ec-3222-4ed0-b9ef-913ba2a482a9: moving from state preparing -> state finished >Thread-15910699::DEBUG::2013-11-19 21:18:17,437::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910699::DEBUG::2013-11-19 21:18:17,438::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910699::DEBUG::2013-11-19 21:18:17,438::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910699::DEBUG::2013-11-19 21:18:17,439::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910699::DEBUG::2013-11-19 21:18:17,440::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910699::DEBUG::2013-11-19 21:18:17,440::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910699::DEBUG::2013-11-19 21:18:17,441::task::495::TaskManager.Task::(_debug) Task 0e0894ec-3222-4ed0-b9ef-913ba2a482a9: ref 0 aborting False >Thread-15910699::INFO::2013-11-19 21:18:17,441::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910700::INFO::2013-11-19 21:18:19,102::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910700::DEBUG::2013-11-19 21:18:19,103::task::495::TaskManager.Task::(_debug) Task 2c053d05-60ca-4904-9a2f-af77bc04b288: moving from state init -> state preparing >Thread-15910700::DEBUG::2013-11-19 21:18:19,103::task::495::TaskManager.Task::(_debug) Task 2c053d05-60ca-4904-9a2f-af77bc04b288: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000356197357178', 'lastCheck': 1384895889.9362769, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0015389919281', 'lastCheck': 1384895892.0160811, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895897.094624, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000440120697021', 'lastCheck': 1384895896.556711, 'valid': True, 'code': 0}} >Thread-15910700::DEBUG::2013-11-19 21:18:19,104::task::495::TaskManager.Task::(_debug) Task 2c053d05-60ca-4904-9a2f-af77bc04b288: moving from state preparing -> state finished >Thread-15910700::DEBUG::2013-11-19 21:18:19,105::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910700::DEBUG::2013-11-19 21:18:19,105::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910700::DEBUG::2013-11-19 21:18:19,106::task::495::TaskManager.Task::(_debug) Task 2c053d05-60ca-4904-9a2f-af77bc04b288: ref 0 aborting False >Thread-15910700::INFO::2013-11-19 21:18:19,106::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000356197357178', 'lastCheck': 1384895889.9362769, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0015389919281', 'lastCheck': 1384895892.0160811, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895897.094624, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000440120697021', 'lastCheck': 1384895896.556711, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:18:19,416::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:18:19,958::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0712801 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:18:21,973::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:18:22,455::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0439418 s, 23.3 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:18:24,466::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:18:24,986::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0711741 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:18:27,001::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:18:27,202::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15910548::DEBUG::2013-11-19 21:18:27,507::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0706307 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15910706::DEBUG::2013-11-19 21:18:27,554::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910706::INFO::2013-11-19 21:18:27,556::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910706::DEBUG::2013-11-19 21:18:27,556::task::495::TaskManager.Task::(_debug) Task a313c662-be96-46a0-acff-b7142d79bbcc: moving from state init -> state preparing >Thread-15910706::DEBUG::2013-11-19 21:18:27,557::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910706::DEBUG::2013-11-19 21:18:27,558::task::495::TaskManager.Task::(_debug) Task a313c662-be96-46a0-acff-b7142d79bbcc: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910706::DEBUG::2013-11-19 21:18:27,558::task::495::TaskManager.Task::(_debug) Task a313c662-be96-46a0-acff-b7142d79bbcc: moving from state preparing -> state finished >Thread-15910706::DEBUG::2013-11-19 21:18:27,559::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910706::DEBUG::2013-11-19 21:18:27,559::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910706::DEBUG::2013-11-19 21:18:27,560::task::495::TaskManager.Task::(_debug) Task a313c662-be96-46a0-acff-b7142d79bbcc: ref 0 aborting False >Thread-15910706::INFO::2013-11-19 21:18:27,560::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910707::DEBUG::2013-11-19 21:18:27,565::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910707::INFO::2013-11-19 21:18:27,566::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910707::DEBUG::2013-11-19 21:18:27,567::task::495::TaskManager.Task::(_debug) Task a5d79a68-a879-478b-962e-32a55cae3345: moving from state init -> state preparing >Thread-15910707::DEBUG::2013-11-19 21:18:27,568::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`309996ca-db33-4a1c-b31d-391ed3835c12`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910707::DEBUG::2013-11-19 21:18:27,569::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910707::DEBUG::2013-11-19 21:18:27,570::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910707::DEBUG::2013-11-19 21:18:27,570::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`309996ca-db33-4a1c-b31d-391ed3835c12`::Granted request >Thread-15910707::DEBUG::2013-11-19 21:18:27,571::task::495::TaskManager.Task::(_debug) Task a5d79a68-a879-478b-962e-32a55cae3345: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910707::DEBUG::2013-11-19 21:18:27,572::task::495::TaskManager.Task::(_debug) Task a5d79a68-a879-478b-962e-32a55cae3345: ref 1 aborting False >Thread-15910707::WARNING::2013-11-19 21:18:27,574::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910707::DEBUG::2013-11-19 21:18:27,575::task::495::TaskManager.Task::(_debug) Task a5d79a68-a879-478b-962e-32a55cae3345: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910707::DEBUG::2013-11-19 21:18:27,575::task::495::TaskManager.Task::(_debug) Task a5d79a68-a879-478b-962e-32a55cae3345: moving from state preparing -> state finished >Thread-15910707::DEBUG::2013-11-19 21:18:27,576::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910707::DEBUG::2013-11-19 21:18:27,576::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910707::DEBUG::2013-11-19 21:18:27,577::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910707::DEBUG::2013-11-19 21:18:27,578::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910707::DEBUG::2013-11-19 21:18:27,578::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910707::DEBUG::2013-11-19 21:18:27,579::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910707::DEBUG::2013-11-19 21:18:27,579::task::495::TaskManager.Task::(_debug) Task a5d79a68-a879-478b-962e-32a55cae3345: ref 0 aborting False >Thread-15910707::INFO::2013-11-19 21:18:27,580::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910708::INFO::2013-11-19 21:18:29,253::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910708::DEBUG::2013-11-19 21:18:29,254::task::495::TaskManager.Task::(_debug) Task 95c9af97-f729-43e6-a9c0-c55ab4861ba3: moving from state init -> state preparing >Thread-15910708::DEBUG::2013-11-19 21:18:29,255::task::495::TaskManager.Task::(_debug) Task 95c9af97-f729-43e6-a9c0-c55ab4861ba3: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00062108039856', 'lastCheck': 1384895900.0783701, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00151419639587', 'lastCheck': 1384895902.1980031, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895907.2046001, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000526189804077', 'lastCheck': 1384895906.6760631, 'valid': True, 'code': 0}} >Thread-15910708::DEBUG::2013-11-19 21:18:29,255::task::495::TaskManager.Task::(_debug) Task 95c9af97-f729-43e6-a9c0-c55ab4861ba3: moving from state preparing -> state finished >Thread-15910708::DEBUG::2013-11-19 21:18:29,256::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910708::DEBUG::2013-11-19 21:18:29,256::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910708::DEBUG::2013-11-19 21:18:29,257::task::495::TaskManager.Task::(_debug) Task 95c9af97-f729-43e6-a9c0-c55ab4861ba3: ref 0 aborting False >Thread-15910708::INFO::2013-11-19 21:18:29,258::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00062108039856', 'lastCheck': 1384895900.0783701, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00151419639587', 'lastCheck': 1384895902.1980031, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895907.2046001, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000526189804077', 'lastCheck': 1384895906.6760631, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:18:29,523::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:18:30,050::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0705775 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:18:32,066::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:18:32,615::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0711785 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:18:34,630::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:18:35,144::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0743011 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:18:37,159::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:18:37,390::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910714::DEBUG::2013-11-19 21:18:37,695::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910714::INFO::2013-11-19 21:18:37,697::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910714::DEBUG::2013-11-19 21:18:37,704::task::495::TaskManager.Task::(_debug) Task a4a01bb9-dbc7-450e-b07e-0d2ae8e8931d: moving from state init -> state preparing >Thread-15910714::DEBUG::2013-11-19 21:18:37,705::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910714::DEBUG::2013-11-19 21:18:37,706::task::495::TaskManager.Task::(_debug) Task a4a01bb9-dbc7-450e-b07e-0d2ae8e8931d: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910714::DEBUG::2013-11-19 21:18:37,707::task::495::TaskManager.Task::(_debug) Task a4a01bb9-dbc7-450e-b07e-0d2ae8e8931d: moving from state preparing -> state finished >Thread-15910714::DEBUG::2013-11-19 21:18:37,708::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910714::DEBUG::2013-11-19 21:18:37,709::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910714::DEBUG::2013-11-19 21:18:37,710::task::495::TaskManager.Task::(_debug) Task a4a01bb9-dbc7-450e-b07e-0d2ae8e8931d: ref 0 aborting False >Thread-15910714::INFO::2013-11-19 21:18:37,711::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910715::DEBUG::2013-11-19 21:18:37,717::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910715::INFO::2013-11-19 21:18:37,718::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910715::DEBUG::2013-11-19 21:18:37,719::task::495::TaskManager.Task::(_debug) Task a36f92f0-29a9-4063-a1dd-09c328d79a92: moving from state init -> state preparing >Thread-15910715::DEBUG::2013-11-19 21:18:37,720::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`04df802a-e0d6-4957-846f-47722aeace92`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910715::DEBUG::2013-11-19 21:18:37,721::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910715::DEBUG::2013-11-19 21:18:37,723::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910715::DEBUG::2013-11-19 21:18:37,723::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`04df802a-e0d6-4957-846f-47722aeace92`::Granted request >Thread-15910715::DEBUG::2013-11-19 21:18:37,725::task::495::TaskManager.Task::(_debug) Task a36f92f0-29a9-4063-a1dd-09c328d79a92: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910715::DEBUG::2013-11-19 21:18:37,726::task::495::TaskManager.Task::(_debug) Task a36f92f0-29a9-4063-a1dd-09c328d79a92: ref 1 aborting False >Thread-15910715::WARNING::2013-11-19 21:18:37,729::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910715::DEBUG::2013-11-19 21:18:37,730::task::495::TaskManager.Task::(_debug) Task a36f92f0-29a9-4063-a1dd-09c328d79a92: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910715::DEBUG::2013-11-19 21:18:37,732::task::495::TaskManager.Task::(_debug) Task a36f92f0-29a9-4063-a1dd-09c328d79a92: moving from state preparing -> state finished >Thread-15910715::DEBUG::2013-11-19 21:18:37,733::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910715::DEBUG::2013-11-19 21:18:37,733::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910715::DEBUG::2013-11-19 21:18:37,734::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Dummy-15910548::DEBUG::2013-11-19 21:18:37,736::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0713171 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15910715::DEBUG::2013-11-19 21:18:37,737::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910715::DEBUG::2013-11-19 21:18:37,738::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910715::DEBUG::2013-11-19 21:18:37,753::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910715::DEBUG::2013-11-19 21:18:37,754::task::495::TaskManager.Task::(_debug) Task a36f92f0-29a9-4063-a1dd-09c328d79a92: ref 0 aborting False >Thread-15910715::INFO::2013-11-19 21:18:37,754::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910716::INFO::2013-11-19 21:18:39,408::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910716::DEBUG::2013-11-19 21:18:39,409::task::495::TaskManager.Task::(_debug) Task e2850f1b-695b-4267-a4b9-e9e5efe519fb: moving from state init -> state preparing >Thread-15910716::DEBUG::2013-11-19 21:18:39,410::task::495::TaskManager.Task::(_debug) Task e2850f1b-695b-4267-a4b9-e9e5efe519fb: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000373840332031', 'lastCheck': 1384895910.205627, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0014910697937', 'lastCheck': 1384895912.3529811, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895917.391942, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000524997711182', 'lastCheck': 1384895916.839061, 'valid': True, 'code': 0}} >Thread-15910716::DEBUG::2013-11-19 21:18:39,410::task::495::TaskManager.Task::(_debug) Task e2850f1b-695b-4267-a4b9-e9e5efe519fb: moving from state preparing -> state finished >Thread-15910716::DEBUG::2013-11-19 21:18:39,411::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910716::DEBUG::2013-11-19 21:18:39,411::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910716::DEBUG::2013-11-19 21:18:39,412::task::495::TaskManager.Task::(_debug) Task e2850f1b-695b-4267-a4b9-e9e5efe519fb: ref 0 aborting False >Thread-15910716::INFO::2013-11-19 21:18:39,413::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000373840332031', 'lastCheck': 1384895910.205627, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0014910697937', 'lastCheck': 1384895912.3529811, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895917.391942, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000524997711182', 'lastCheck': 1384895916.839061, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:18:39,755::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:18:40,264::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0702919 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:18:42,279::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:18:42,820::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.071019 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:18:44,835::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:18:45,357::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0706501 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:18:47,373::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:18:47,578::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910722::DEBUG::2013-11-19 21:18:47,898::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910722::INFO::2013-11-19 21:18:47,900::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910722::DEBUG::2013-11-19 21:18:47,901::task::495::TaskManager.Task::(_debug) Task 67ae2a64-1fcd-44f1-80b8-398b8e5e622e: moving from state init -> state preparing >Thread-15910722::DEBUG::2013-11-19 21:18:47,903::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910722::DEBUG::2013-11-19 21:18:47,903::task::495::TaskManager.Task::(_debug) Task 67ae2a64-1fcd-44f1-80b8-398b8e5e622e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910722::DEBUG::2013-11-19 21:18:47,905::task::495::TaskManager.Task::(_debug) Task 67ae2a64-1fcd-44f1-80b8-398b8e5e622e: moving from state preparing -> state finished >Thread-15910722::DEBUG::2013-11-19 21:18:47,906::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910722::DEBUG::2013-11-19 21:18:47,907::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910722::DEBUG::2013-11-19 21:18:47,908::task::495::TaskManager.Task::(_debug) Task 67ae2a64-1fcd-44f1-80b8-398b8e5e622e: ref 0 aborting False >Thread-15910722::INFO::2013-11-19 21:18:47,910::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910723::DEBUG::2013-11-19 21:18:47,917::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910723::INFO::2013-11-19 21:18:47,918::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910723::DEBUG::2013-11-19 21:18:47,919::task::495::TaskManager.Task::(_debug) Task 6ed31f6e-e688-4224-b6fa-9bc0e2010f32: moving from state init -> state preparing >Thread-15910723::DEBUG::2013-11-19 21:18:47,921::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9425fe80-20d0-408a-a58b-577aaab43dad`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910723::DEBUG::2013-11-19 21:18:47,922::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910723::DEBUG::2013-11-19 21:18:47,924::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910723::DEBUG::2013-11-19 21:18:47,925::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9425fe80-20d0-408a-a58b-577aaab43dad`::Granted request >Thread-15910723::DEBUG::2013-11-19 21:18:47,926::task::495::TaskManager.Task::(_debug) Task 6ed31f6e-e688-4224-b6fa-9bc0e2010f32: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910723::DEBUG::2013-11-19 21:18:47,927::task::495::TaskManager.Task::(_debug) Task 6ed31f6e-e688-4224-b6fa-9bc0e2010f32: ref 1 aborting False >Thread-15910723::WARNING::2013-11-19 21:18:47,930::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910723::DEBUG::2013-11-19 21:18:47,931::task::495::TaskManager.Task::(_debug) Task 6ed31f6e-e688-4224-b6fa-9bc0e2010f32: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910723::DEBUG::2013-11-19 21:18:47,932::task::495::TaskManager.Task::(_debug) Task 6ed31f6e-e688-4224-b6fa-9bc0e2010f32: moving from state preparing -> state finished >Thread-15910723::DEBUG::2013-11-19 21:18:47,933::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910723::DEBUG::2013-11-19 21:18:47,934::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910723::DEBUG::2013-11-19 21:18:47,935::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910723::DEBUG::2013-11-19 21:18:47,936::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910723::DEBUG::2013-11-19 21:18:47,937::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910723::DEBUG::2013-11-19 21:18:47,938::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910723::DEBUG::2013-11-19 21:18:47,939::task::495::TaskManager.Task::(_debug) Task 6ed31f6e-e688-4224-b6fa-9bc0e2010f32: ref 0 aborting False >Thread-15910723::INFO::2013-11-19 21:18:47,940::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15910548::DEBUG::2013-11-19 21:18:47,945::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0933358 s, 11.0 MB/s\n'; <rc> = 0 >Thread-15910724::INFO::2013-11-19 21:18:49,590::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910724::DEBUG::2013-11-19 21:18:49,591::task::495::TaskManager.Task::(_debug) Task 1554ab68-db5c-4120-a6cf-91c905e7e359: moving from state init -> state preparing >Thread-15910724::DEBUG::2013-11-19 21:18:49,592::task::495::TaskManager.Task::(_debug) Task 1554ab68-db5c-4120-a6cf-91c905e7e359: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000409126281738', 'lastCheck': 1384895920.3937111, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00245785713196', 'lastCheck': 1384895922.515836, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895927.5822909, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000391006469727', 'lastCheck': 1384895927.046561, 'valid': True, 'code': 0}} >Thread-15910724::DEBUG::2013-11-19 21:18:49,592::task::495::TaskManager.Task::(_debug) Task 1554ab68-db5c-4120-a6cf-91c905e7e359: moving from state preparing -> state finished >Thread-15910724::DEBUG::2013-11-19 21:18:49,593::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910724::DEBUG::2013-11-19 21:18:49,593::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910724::DEBUG::2013-11-19 21:18:49,594::task::495::TaskManager.Task::(_debug) Task 1554ab68-db5c-4120-a6cf-91c905e7e359: ref 0 aborting False >Thread-15910724::INFO::2013-11-19 21:18:49,595::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000409126281738', 'lastCheck': 1384895920.3937111, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00245785713196', 'lastCheck': 1384895922.515836, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895927.5822909, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000391006469727', 'lastCheck': 1384895927.046561, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:18:49,959::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:18:50,490::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0712066 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:18:52,505::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:18:53,016::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0712308 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:18:55,031::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:18:55,564::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0712034 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:18:57,579::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:18:57,794::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15910548::DEBUG::2013-11-19 21:18:58,103::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0709988 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15910730::DEBUG::2013-11-19 21:18:58,160::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910730::INFO::2013-11-19 21:18:58,162::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910730::DEBUG::2013-11-19 21:18:58,163::task::495::TaskManager.Task::(_debug) Task d4c3d198-e642-4045-b16f-2f2bf40f6901: moving from state init -> state preparing >Thread-15910730::DEBUG::2013-11-19 21:18:58,163::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910730::DEBUG::2013-11-19 21:18:58,164::task::495::TaskManager.Task::(_debug) Task d4c3d198-e642-4045-b16f-2f2bf40f6901: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910730::DEBUG::2013-11-19 21:18:58,165::task::495::TaskManager.Task::(_debug) Task d4c3d198-e642-4045-b16f-2f2bf40f6901: moving from state preparing -> state finished >Thread-15910730::DEBUG::2013-11-19 21:18:58,165::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910730::DEBUG::2013-11-19 21:18:58,166::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910730::DEBUG::2013-11-19 21:18:58,166::task::495::TaskManager.Task::(_debug) Task d4c3d198-e642-4045-b16f-2f2bf40f6901: ref 0 aborting False >Thread-15910730::INFO::2013-11-19 21:18:58,167::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910731::DEBUG::2013-11-19 21:18:58,172::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910731::INFO::2013-11-19 21:18:58,173::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910731::DEBUG::2013-11-19 21:18:58,173::task::495::TaskManager.Task::(_debug) Task c20cb585-742a-49b4-9c59-176f2d61c983: moving from state init -> state preparing >Thread-15910731::DEBUG::2013-11-19 21:18:58,175::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fb91a55c-40ea-4632-b71c-c04586ddaa32`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910731::DEBUG::2013-11-19 21:18:58,175::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910731::DEBUG::2013-11-19 21:18:58,176::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910731::DEBUG::2013-11-19 21:18:58,177::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fb91a55c-40ea-4632-b71c-c04586ddaa32`::Granted request >Thread-15910731::DEBUG::2013-11-19 21:18:58,178::task::495::TaskManager.Task::(_debug) Task c20cb585-742a-49b4-9c59-176f2d61c983: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910731::DEBUG::2013-11-19 21:18:58,178::task::495::TaskManager.Task::(_debug) Task c20cb585-742a-49b4-9c59-176f2d61c983: ref 1 aborting False >Thread-15910731::WARNING::2013-11-19 21:18:58,180::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910731::DEBUG::2013-11-19 21:18:58,181::task::495::TaskManager.Task::(_debug) Task c20cb585-742a-49b4-9c59-176f2d61c983: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910731::DEBUG::2013-11-19 21:18:58,182::task::495::TaskManager.Task::(_debug) Task c20cb585-742a-49b4-9c59-176f2d61c983: moving from state preparing -> state finished >Thread-15910731::DEBUG::2013-11-19 21:18:58,182::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910731::DEBUG::2013-11-19 21:18:58,183::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910731::DEBUG::2013-11-19 21:18:58,184::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910731::DEBUG::2013-11-19 21:18:58,184::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910731::DEBUG::2013-11-19 21:18:58,185::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910731::DEBUG::2013-11-19 21:18:58,185::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910731::DEBUG::2013-11-19 21:18:58,186::task::495::TaskManager.Task::(_debug) Task c20cb585-742a-49b4-9c59-176f2d61c983: ref 0 aborting False >Thread-15910731::INFO::2013-11-19 21:18:58,187::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910732::INFO::2013-11-19 21:18:59,806::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910732::DEBUG::2013-11-19 21:18:59,807::task::495::TaskManager.Task::(_debug) Task dea2a88f-9b9e-4649-99f4-39feda83c740: moving from state init -> state preparing >Thread-15910732::DEBUG::2013-11-19 21:18:59,808::task::495::TaskManager.Task::(_debug) Task dea2a88f-9b9e-4649-99f4-39feda83c740: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00049090385437', 'lastCheck': 1384895930.580384, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00163197517395', 'lastCheck': 1384895932.709038, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895937.795506, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000383853912354', 'lastCheck': 1384895937.254693, 'valid': True, 'code': 0}} >Thread-15910732::DEBUG::2013-11-19 21:18:59,808::task::495::TaskManager.Task::(_debug) Task dea2a88f-9b9e-4649-99f4-39feda83c740: moving from state preparing -> state finished >Thread-15910732::DEBUG::2013-11-19 21:18:59,809::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910732::DEBUG::2013-11-19 21:18:59,809::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910732::DEBUG::2013-11-19 21:18:59,810::task::495::TaskManager.Task::(_debug) Task dea2a88f-9b9e-4649-99f4-39feda83c740: ref 0 aborting False >Thread-15910732::INFO::2013-11-19 21:18:59,811::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00049090385437', 'lastCheck': 1384895930.580384, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00163197517395', 'lastCheck': 1384895932.709038, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895937.795506, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000383853912354', 'lastCheck': 1384895937.254693, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:19:00,119::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:19:00,634::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0700107 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:19:02,649::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:19:03,169::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0703466 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:19:05,184::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:19:05,725::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0710145 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:19:07,740::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:19:07,971::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15910548::DEBUG::2013-11-19 21:19:08,264::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0740591 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15910738::DEBUG::2013-11-19 21:19:08,318::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910738::INFO::2013-11-19 21:19:08,319::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910738::DEBUG::2013-11-19 21:19:08,320::task::495::TaskManager.Task::(_debug) Task 02e9448b-65e3-4ebb-ac5e-fdbc1013cf04: moving from state init -> state preparing >Thread-15910738::DEBUG::2013-11-19 21:19:08,321::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910738::DEBUG::2013-11-19 21:19:08,321::task::495::TaskManager.Task::(_debug) Task 02e9448b-65e3-4ebb-ac5e-fdbc1013cf04: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910738::DEBUG::2013-11-19 21:19:08,322::task::495::TaskManager.Task::(_debug) Task 02e9448b-65e3-4ebb-ac5e-fdbc1013cf04: moving from state preparing -> state finished >Thread-15910738::DEBUG::2013-11-19 21:19:08,322::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910738::DEBUG::2013-11-19 21:19:08,323::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910738::DEBUG::2013-11-19 21:19:08,324::task::495::TaskManager.Task::(_debug) Task 02e9448b-65e3-4ebb-ac5e-fdbc1013cf04: ref 0 aborting False >Thread-15910738::INFO::2013-11-19 21:19:08,324::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910739::DEBUG::2013-11-19 21:19:08,329::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910739::INFO::2013-11-19 21:19:08,330::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910739::DEBUG::2013-11-19 21:19:08,330::task::495::TaskManager.Task::(_debug) Task a3d74178-47b0-4a37-8644-b88b7dc47ed3: moving from state init -> state preparing >Thread-15910739::DEBUG::2013-11-19 21:19:08,332::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6fb20b9c-6572-4438-93c3-5bed9237aecd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910739::DEBUG::2013-11-19 21:19:08,332::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910739::DEBUG::2013-11-19 21:19:08,333::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910739::DEBUG::2013-11-19 21:19:08,333::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6fb20b9c-6572-4438-93c3-5bed9237aecd`::Granted request >Thread-15910739::DEBUG::2013-11-19 21:19:08,334::task::495::TaskManager.Task::(_debug) Task a3d74178-47b0-4a37-8644-b88b7dc47ed3: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910739::DEBUG::2013-11-19 21:19:08,335::task::495::TaskManager.Task::(_debug) Task a3d74178-47b0-4a37-8644-b88b7dc47ed3: ref 1 aborting False >Thread-15910739::WARNING::2013-11-19 21:19:08,337::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910739::DEBUG::2013-11-19 21:19:08,338::task::495::TaskManager.Task::(_debug) Task a3d74178-47b0-4a37-8644-b88b7dc47ed3: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910739::DEBUG::2013-11-19 21:19:08,339::task::495::TaskManager.Task::(_debug) Task a3d74178-47b0-4a37-8644-b88b7dc47ed3: moving from state preparing -> state finished >Thread-15910739::DEBUG::2013-11-19 21:19:08,339::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910739::DEBUG::2013-11-19 21:19:08,340::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910739::DEBUG::2013-11-19 21:19:08,341::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910739::DEBUG::2013-11-19 21:19:08,341::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910739::DEBUG::2013-11-19 21:19:08,342::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910739::DEBUG::2013-11-19 21:19:08,342::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910739::DEBUG::2013-11-19 21:19:08,343::task::495::TaskManager.Task::(_debug) Task a3d74178-47b0-4a37-8644-b88b7dc47ed3: ref 0 aborting False >Thread-15910739::INFO::2013-11-19 21:19:08,343::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910740::INFO::2013-11-19 21:19:09,983::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910740::DEBUG::2013-11-19 21:19:09,984::task::495::TaskManager.Task::(_debug) Task 3c618fc0-171b-4bce-8641-0a02949f98b1: moving from state init -> state preparing >Thread-15910740::DEBUG::2013-11-19 21:19:09,984::task::495::TaskManager.Task::(_debug) Task 3c618fc0-171b-4bce-8641-0a02949f98b1: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000504016876221', 'lastCheck': 1384895940.7969739, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00129890441895', 'lastCheck': 1384895942.9170351, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895947.975529, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000461101531982', 'lastCheck': 1384895947.4119179, 'valid': True, 'code': 0}} >Thread-15910740::DEBUG::2013-11-19 21:19:09,985::task::495::TaskManager.Task::(_debug) Task 3c618fc0-171b-4bce-8641-0a02949f98b1: moving from state preparing -> state finished >Thread-15910740::DEBUG::2013-11-19 21:19:09,986::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910740::DEBUG::2013-11-19 21:19:09,986::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910740::DEBUG::2013-11-19 21:19:09,987::task::495::TaskManager.Task::(_debug) Task 3c618fc0-171b-4bce-8641-0a02949f98b1: ref 0 aborting False >Thread-15910740::INFO::2013-11-19 21:19:09,988::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000504016876221', 'lastCheck': 1384895940.7969739, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00129890441895', 'lastCheck': 1384895942.9170351, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895947.975529, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000461101531982', 'lastCheck': 1384895947.4119179, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:19:10,279::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:19:10,819::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0709502 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:19:12,834::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:19:13,355::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0705378 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:19:15,370::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:19:15,880::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0700522 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:19:17,894::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:19:18,158::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15910548::DEBUG::2013-11-19 21:19:18,418::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0701327 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15910746::DEBUG::2013-11-19 21:19:18,507::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910746::INFO::2013-11-19 21:19:18,508::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910746::DEBUG::2013-11-19 21:19:18,508::task::495::TaskManager.Task::(_debug) Task d246b964-66af-4aba-8e00-224d546835f6: moving from state init -> state preparing >Thread-15910746::DEBUG::2013-11-19 21:19:18,509::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910746::DEBUG::2013-11-19 21:19:18,510::task::495::TaskManager.Task::(_debug) Task d246b964-66af-4aba-8e00-224d546835f6: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910746::DEBUG::2013-11-19 21:19:18,511::task::495::TaskManager.Task::(_debug) Task d246b964-66af-4aba-8e00-224d546835f6: moving from state preparing -> state finished >Thread-15910746::DEBUG::2013-11-19 21:19:18,511::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910746::DEBUG::2013-11-19 21:19:18,512::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910746::DEBUG::2013-11-19 21:19:18,512::task::495::TaskManager.Task::(_debug) Task d246b964-66af-4aba-8e00-224d546835f6: ref 0 aborting False >Thread-15910746::INFO::2013-11-19 21:19:18,513::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910747::DEBUG::2013-11-19 21:19:18,518::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910747::INFO::2013-11-19 21:19:18,519::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910747::DEBUG::2013-11-19 21:19:18,519::task::495::TaskManager.Task::(_debug) Task 5170d1f2-e27c-4f39-ad28-381dad3185ac: moving from state init -> state preparing >Thread-15910747::DEBUG::2013-11-19 21:19:18,520::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bf3ac7cd-6b60-4e87-84b5-193f5f116c65`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910747::DEBUG::2013-11-19 21:19:18,521::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910747::DEBUG::2013-11-19 21:19:18,522::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910747::DEBUG::2013-11-19 21:19:18,522::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bf3ac7cd-6b60-4e87-84b5-193f5f116c65`::Granted request >Thread-15910747::DEBUG::2013-11-19 21:19:18,523::task::495::TaskManager.Task::(_debug) Task 5170d1f2-e27c-4f39-ad28-381dad3185ac: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910747::DEBUG::2013-11-19 21:19:18,524::task::495::TaskManager.Task::(_debug) Task 5170d1f2-e27c-4f39-ad28-381dad3185ac: ref 1 aborting False >Thread-15910747::WARNING::2013-11-19 21:19:18,526::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910747::DEBUG::2013-11-19 21:19:18,527::task::495::TaskManager.Task::(_debug) Task 5170d1f2-e27c-4f39-ad28-381dad3185ac: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910747::DEBUG::2013-11-19 21:19:18,527::task::495::TaskManager.Task::(_debug) Task 5170d1f2-e27c-4f39-ad28-381dad3185ac: moving from state preparing -> state finished >Thread-15910747::DEBUG::2013-11-19 21:19:18,528::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910747::DEBUG::2013-11-19 21:19:18,529::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910747::DEBUG::2013-11-19 21:19:18,530::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910747::DEBUG::2013-11-19 21:19:18,531::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910747::DEBUG::2013-11-19 21:19:18,532::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910747::DEBUG::2013-11-19 21:19:18,532::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910747::DEBUG::2013-11-19 21:19:18,533::task::495::TaskManager.Task::(_debug) Task 5170d1f2-e27c-4f39-ad28-381dad3185ac: ref 0 aborting False >Thread-15910747::INFO::2013-11-19 21:19:18,534::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910748::INFO::2013-11-19 21:19:20,169::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910748::DEBUG::2013-11-19 21:19:20,170::task::495::TaskManager.Task::(_debug) Task 6df45cec-0074-43fa-957c-6a20851e89ef: moving from state init -> state preparing >Thread-15910748::DEBUG::2013-11-19 21:19:20,171::task::495::TaskManager.Task::(_debug) Task 6df45cec-0074-43fa-957c-6a20851e89ef: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000424861907959', 'lastCheck': 1384895950.9653339, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00172710418701', 'lastCheck': 1384895953.050432, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895958.1620331, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000381946563721', 'lastCheck': 1384895957.5719521, 'valid': True, 'code': 0}} >Thread-15910748::DEBUG::2013-11-19 21:19:20,171::task::495::TaskManager.Task::(_debug) Task 6df45cec-0074-43fa-957c-6a20851e89ef: moving from state preparing -> state finished >Thread-15910748::DEBUG::2013-11-19 21:19:20,172::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910748::DEBUG::2013-11-19 21:19:20,173::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910748::DEBUG::2013-11-19 21:19:20,173::task::495::TaskManager.Task::(_debug) Task 6df45cec-0074-43fa-957c-6a20851e89ef: ref 0 aborting False >Thread-15910748::INFO::2013-11-19 21:19:20,174::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000424861907959', 'lastCheck': 1384895950.9653339, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00172710418701', 'lastCheck': 1384895953.050432, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895958.1620331, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000381946563721', 'lastCheck': 1384895957.5719521, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:19:20,434::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:19:20,982::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.077425 s, 13.2 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:19:22,997::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:19:23,521::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0807402 s, 12.7 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:19:25,537::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:19:26,082::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0842655 s, 12.2 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:19:28,098::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:19:28,310::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15910548::DEBUG::2013-11-19 21:19:28,615::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0712861 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15910754::DEBUG::2013-11-19 21:19:28,683::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910754::INFO::2013-11-19 21:19:28,685::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910754::DEBUG::2013-11-19 21:19:28,685::task::495::TaskManager.Task::(_debug) Task 2583795b-a0e7-4f8b-8931-eb96cece773c: moving from state init -> state preparing >Thread-15910754::DEBUG::2013-11-19 21:19:28,686::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910754::DEBUG::2013-11-19 21:19:28,687::task::495::TaskManager.Task::(_debug) Task 2583795b-a0e7-4f8b-8931-eb96cece773c: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910754::DEBUG::2013-11-19 21:19:28,687::task::495::TaskManager.Task::(_debug) Task 2583795b-a0e7-4f8b-8931-eb96cece773c: moving from state preparing -> state finished >Thread-15910754::DEBUG::2013-11-19 21:19:28,688::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910754::DEBUG::2013-11-19 21:19:28,688::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910754::DEBUG::2013-11-19 21:19:28,689::task::495::TaskManager.Task::(_debug) Task 2583795b-a0e7-4f8b-8931-eb96cece773c: ref 0 aborting False >Thread-15910754::INFO::2013-11-19 21:19:28,689::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910755::DEBUG::2013-11-19 21:19:28,694::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910755::INFO::2013-11-19 21:19:28,695::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910755::DEBUG::2013-11-19 21:19:28,696::task::495::TaskManager.Task::(_debug) Task e9d3655d-d8ad-41a9-b77f-554543e3d9e3: moving from state init -> state preparing >Thread-15910755::DEBUG::2013-11-19 21:19:28,697::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`0b45ba67-5968-4c36-a059-375051e8ef82`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910755::DEBUG::2013-11-19 21:19:28,698::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910755::DEBUG::2013-11-19 21:19:28,698::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910755::DEBUG::2013-11-19 21:19:28,699::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`0b45ba67-5968-4c36-a059-375051e8ef82`::Granted request >Thread-15910755::DEBUG::2013-11-19 21:19:28,700::task::495::TaskManager.Task::(_debug) Task e9d3655d-d8ad-41a9-b77f-554543e3d9e3: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910755::DEBUG::2013-11-19 21:19:28,700::task::495::TaskManager.Task::(_debug) Task e9d3655d-d8ad-41a9-b77f-554543e3d9e3: ref 1 aborting False >Thread-15910755::WARNING::2013-11-19 21:19:28,702::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910755::DEBUG::2013-11-19 21:19:28,703::task::495::TaskManager.Task::(_debug) Task e9d3655d-d8ad-41a9-b77f-554543e3d9e3: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910755::DEBUG::2013-11-19 21:19:28,704::task::495::TaskManager.Task::(_debug) Task e9d3655d-d8ad-41a9-b77f-554543e3d9e3: moving from state preparing -> state finished >Thread-15910755::DEBUG::2013-11-19 21:19:28,705::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910755::DEBUG::2013-11-19 21:19:28,705::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910755::DEBUG::2013-11-19 21:19:28,706::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910755::DEBUG::2013-11-19 21:19:28,707::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910755::DEBUG::2013-11-19 21:19:28,707::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910755::DEBUG::2013-11-19 21:19:28,708::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910755::DEBUG::2013-11-19 21:19:28,708::task::495::TaskManager.Task::(_debug) Task e9d3655d-d8ad-41a9-b77f-554543e3d9e3: ref 0 aborting False >Thread-15910755::INFO::2013-11-19 21:19:28,709::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910756::INFO::2013-11-19 21:19:30,322::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910756::DEBUG::2013-11-19 21:19:30,323::task::495::TaskManager.Task::(_debug) Task 1fe0a430-b741-49e8-af77-625c59295a88: moving from state init -> state preparing >Thread-15910756::DEBUG::2013-11-19 21:19:30,323::task::495::TaskManager.Task::(_debug) Task 1fe0a430-b741-49e8-af77-625c59295a88: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00119400024414', 'lastCheck': 1384895961.1499009, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00159883499146', 'lastCheck': 1384895963.214612, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895968.314203, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000491142272949', 'lastCheck': 1384895967.76038, 'valid': True, 'code': 0}} >Thread-15910756::DEBUG::2013-11-19 21:19:30,324::task::495::TaskManager.Task::(_debug) Task 1fe0a430-b741-49e8-af77-625c59295a88: moving from state preparing -> state finished >Thread-15910756::DEBUG::2013-11-19 21:19:30,325::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910756::DEBUG::2013-11-19 21:19:30,325::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910756::DEBUG::2013-11-19 21:19:30,326::task::495::TaskManager.Task::(_debug) Task 1fe0a430-b741-49e8-af77-625c59295a88: ref 0 aborting False >Thread-15910756::INFO::2013-11-19 21:19:30,326::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00119400024414', 'lastCheck': 1384895961.1499009, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00159883499146', 'lastCheck': 1384895963.214612, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895968.314203, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000491142272949', 'lastCheck': 1384895967.76038, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:19:30,630::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:19:31,138::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0706305 s, 14.5 MB/s\n'; <rc> = 0 >Thread-42838::DEBUG::2013-11-19 21:19:31,307::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 21:19:31,995::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:19:33,154::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:19:33,690::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0710608 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:19:35,705::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:19:36,262::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0883131 s, 11.6 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:19:38,277::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:19:38,535::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15910548::DEBUG::2013-11-19 21:19:38,829::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0711798 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15910762::DEBUG::2013-11-19 21:19:38,846::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910762::INFO::2013-11-19 21:19:38,848::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910762::DEBUG::2013-11-19 21:19:38,849::task::495::TaskManager.Task::(_debug) Task efa6b18a-79a2-4406-a58e-c4d1c9125668: moving from state init -> state preparing >Thread-15910762::DEBUG::2013-11-19 21:19:38,850::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910762::DEBUG::2013-11-19 21:19:38,850::task::495::TaskManager.Task::(_debug) Task efa6b18a-79a2-4406-a58e-c4d1c9125668: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910762::DEBUG::2013-11-19 21:19:38,851::task::495::TaskManager.Task::(_debug) Task efa6b18a-79a2-4406-a58e-c4d1c9125668: moving from state preparing -> state finished >Thread-15910762::DEBUG::2013-11-19 21:19:38,852::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910762::DEBUG::2013-11-19 21:19:38,852::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910762::DEBUG::2013-11-19 21:19:38,853::task::495::TaskManager.Task::(_debug) Task efa6b18a-79a2-4406-a58e-c4d1c9125668: ref 0 aborting False >Thread-15910762::INFO::2013-11-19 21:19:38,854::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910763::DEBUG::2013-11-19 21:19:38,860::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910763::INFO::2013-11-19 21:19:38,861::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910763::DEBUG::2013-11-19 21:19:38,861::task::495::TaskManager.Task::(_debug) Task 9ea96967-358f-43df-9ad9-a7e214b7f406: moving from state init -> state preparing >Thread-15910763::DEBUG::2013-11-19 21:19:38,862::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`032d2894-05d1-4c88-b4e2-9d4ec2524768`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910763::DEBUG::2013-11-19 21:19:38,863::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910763::DEBUG::2013-11-19 21:19:38,864::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910763::DEBUG::2013-11-19 21:19:38,864::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`032d2894-05d1-4c88-b4e2-9d4ec2524768`::Granted request >Thread-15910763::DEBUG::2013-11-19 21:19:38,865::task::495::TaskManager.Task::(_debug) Task 9ea96967-358f-43df-9ad9-a7e214b7f406: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910763::DEBUG::2013-11-19 21:19:38,866::task::495::TaskManager.Task::(_debug) Task 9ea96967-358f-43df-9ad9-a7e214b7f406: ref 1 aborting False >Thread-15910763::WARNING::2013-11-19 21:19:38,868::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910763::DEBUG::2013-11-19 21:19:38,869::task::495::TaskManager.Task::(_debug) Task 9ea96967-358f-43df-9ad9-a7e214b7f406: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910763::DEBUG::2013-11-19 21:19:38,869::task::495::TaskManager.Task::(_debug) Task 9ea96967-358f-43df-9ad9-a7e214b7f406: moving from state preparing -> state finished >Thread-15910763::DEBUG::2013-11-19 21:19:38,870::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910763::DEBUG::2013-11-19 21:19:38,870::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910763::DEBUG::2013-11-19 21:19:38,871::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910763::DEBUG::2013-11-19 21:19:38,872::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910763::DEBUG::2013-11-19 21:19:38,872::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910763::DEBUG::2013-11-19 21:19:38,873::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910763::DEBUG::2013-11-19 21:19:38,873::task::495::TaskManager.Task::(_debug) Task 9ea96967-358f-43df-9ad9-a7e214b7f406: ref 0 aborting False >Thread-15910763::INFO::2013-11-19 21:19:38,874::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910764::INFO::2013-11-19 21:19:40,539::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910764::DEBUG::2013-11-19 21:19:40,540::task::495::TaskManager.Task::(_debug) Task 135fabaa-9efd-4650-b825-23a7508179c7: moving from state init -> state preparing >Thread-15910764::DEBUG::2013-11-19 21:19:40,541::task::495::TaskManager.Task::(_debug) Task 135fabaa-9efd-4650-b825-23a7508179c7: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000313997268677', 'lastCheck': 1384895971.9998729, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00165104866028', 'lastCheck': 1384895973.3947361, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895978.5368869, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000418901443481', 'lastCheck': 1384895977.9438889, 'valid': True, 'code': 0}} >Thread-15910764::DEBUG::2013-11-19 21:19:40,541::task::495::TaskManager.Task::(_debug) Task 135fabaa-9efd-4650-b825-23a7508179c7: moving from state preparing -> state finished >Thread-15910764::DEBUG::2013-11-19 21:19:40,542::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910764::DEBUG::2013-11-19 21:19:40,542::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910764::DEBUG::2013-11-19 21:19:40,543::task::495::TaskManager.Task::(_debug) Task 135fabaa-9efd-4650-b825-23a7508179c7: ref 0 aborting False >Thread-15910764::INFO::2013-11-19 21:19:40,544::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000313997268677', 'lastCheck': 1384895971.9998729, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00165104866028', 'lastCheck': 1384895973.3947361, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895978.5368869, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000418901443481', 'lastCheck': 1384895977.9438889, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:19:40,847::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:19:41,376::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0711464 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:19:43,390::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:19:43,915::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0704753 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:19:45,931::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:19:46,459::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.071543 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:19:48,474::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:19:48,687::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15910548::DEBUG::2013-11-19 21:19:49,001::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.070849 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15910770::DEBUG::2013-11-19 21:19:49,017::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910770::INFO::2013-11-19 21:19:49,019::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910770::DEBUG::2013-11-19 21:19:49,019::task::495::TaskManager.Task::(_debug) Task d589b847-6736-4e5d-8838-3c4061d6f00b: moving from state init -> state preparing >Thread-15910770::DEBUG::2013-11-19 21:19:49,020::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910770::DEBUG::2013-11-19 21:19:49,021::task::495::TaskManager.Task::(_debug) Task d589b847-6736-4e5d-8838-3c4061d6f00b: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910770::DEBUG::2013-11-19 21:19:49,021::task::495::TaskManager.Task::(_debug) Task d589b847-6736-4e5d-8838-3c4061d6f00b: moving from state preparing -> state finished >Thread-15910770::DEBUG::2013-11-19 21:19:49,022::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910770::DEBUG::2013-11-19 21:19:49,022::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910770::DEBUG::2013-11-19 21:19:49,023::task::495::TaskManager.Task::(_debug) Task d589b847-6736-4e5d-8838-3c4061d6f00b: ref 0 aborting False >Thread-15910770::INFO::2013-11-19 21:19:49,023::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910771::DEBUG::2013-11-19 21:19:49,028::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910771::INFO::2013-11-19 21:19:49,029::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910771::DEBUG::2013-11-19 21:19:49,029::task::495::TaskManager.Task::(_debug) Task e674a020-c38a-4d3b-873e-5b106a505864: moving from state init -> state preparing >Thread-15910771::DEBUG::2013-11-19 21:19:49,031::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d0fd3e4e-56ab-4bb4-98ab-8d11bc865a31`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910771::DEBUG::2013-11-19 21:19:49,031::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910771::DEBUG::2013-11-19 21:19:49,032::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910771::DEBUG::2013-11-19 21:19:49,033::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d0fd3e4e-56ab-4bb4-98ab-8d11bc865a31`::Granted request >Thread-15910771::DEBUG::2013-11-19 21:19:49,034::task::495::TaskManager.Task::(_debug) Task e674a020-c38a-4d3b-873e-5b106a505864: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910771::DEBUG::2013-11-19 21:19:49,034::task::495::TaskManager.Task::(_debug) Task e674a020-c38a-4d3b-873e-5b106a505864: ref 1 aborting False >Thread-15910771::WARNING::2013-11-19 21:19:49,036::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910771::DEBUG::2013-11-19 21:19:49,037::task::495::TaskManager.Task::(_debug) Task e674a020-c38a-4d3b-873e-5b106a505864: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910771::DEBUG::2013-11-19 21:19:49,038::task::495::TaskManager.Task::(_debug) Task e674a020-c38a-4d3b-873e-5b106a505864: moving from state preparing -> state finished >Thread-15910771::DEBUG::2013-11-19 21:19:49,038::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910771::DEBUG::2013-11-19 21:19:49,039::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910771::DEBUG::2013-11-19 21:19:49,040::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910771::DEBUG::2013-11-19 21:19:49,040::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910771::DEBUG::2013-11-19 21:19:49,041::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910771::DEBUG::2013-11-19 21:19:49,041::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910771::DEBUG::2013-11-19 21:19:49,042::task::495::TaskManager.Task::(_debug) Task e674a020-c38a-4d3b-873e-5b106a505864: ref 0 aborting False >Thread-15910771::INFO::2013-11-19 21:19:49,043::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910772::INFO::2013-11-19 21:19:50,703::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910772::DEBUG::2013-11-19 21:19:50,705::task::495::TaskManager.Task::(_debug) Task 55fb2545-44ed-42c3-b95a-696e0294058a: moving from state init -> state preparing >Thread-15910772::DEBUG::2013-11-19 21:19:50,706::task::495::TaskManager.Task::(_debug) Task 55fb2545-44ed-42c3-b95a-696e0294058a: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000441074371338', 'lastCheck': 1384895982.065028, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00247097015381', 'lastCheck': 1384895983.6277189, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895988.691354, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000871896743774', 'lastCheck': 1384895988.1502531, 'valid': True, 'code': 0}} >Thread-15910772::DEBUG::2013-11-19 21:19:50,707::task::495::TaskManager.Task::(_debug) Task 55fb2545-44ed-42c3-b95a-696e0294058a: moving from state preparing -> state finished >Thread-15910772::DEBUG::2013-11-19 21:19:50,709::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910772::DEBUG::2013-11-19 21:19:50,710::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910772::DEBUG::2013-11-19 21:19:50,711::task::495::TaskManager.Task::(_debug) Task 55fb2545-44ed-42c3-b95a-696e0294058a: ref 0 aborting False >Thread-15910772::INFO::2013-11-19 21:19:50,712::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000441074371338', 'lastCheck': 1384895982.065028, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00247097015381', 'lastCheck': 1384895983.6277189, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895988.691354, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000871896743774', 'lastCheck': 1384895988.1502531, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:19:51,019::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:19:51,562::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0705119 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:19:53,577::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:19:54,105::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0707819 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:19:56,120::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:19:56,644::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0804743 s, 12.7 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:19:58,659::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:19:58,891::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910778::DEBUG::2013-11-19 21:19:59,193::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910778::INFO::2013-11-19 21:19:59,196::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910778::DEBUG::2013-11-19 21:19:59,197::task::495::TaskManager.Task::(_debug) Task 719f33c1-b0ed-4de8-8a82-449d4b1f0e3d: moving from state init -> state preparing >Thread-15910778::DEBUG::2013-11-19 21:19:59,198::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910778::DEBUG::2013-11-19 21:19:59,199::task::495::TaskManager.Task::(_debug) Task 719f33c1-b0ed-4de8-8a82-449d4b1f0e3d: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910778::DEBUG::2013-11-19 21:19:59,201::task::495::TaskManager.Task::(_debug) Task 719f33c1-b0ed-4de8-8a82-449d4b1f0e3d: moving from state preparing -> state finished >Thread-15910778::DEBUG::2013-11-19 21:19:59,203::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910778::DEBUG::2013-11-19 21:19:59,204::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910778::DEBUG::2013-11-19 21:19:59,205::task::495::TaskManager.Task::(_debug) Task 719f33c1-b0ed-4de8-8a82-449d4b1f0e3d: ref 0 aborting False >Thread-15910778::INFO::2013-11-19 21:19:59,206::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910779::DEBUG::2013-11-19 21:19:59,214::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910779::INFO::2013-11-19 21:19:59,216::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910779::DEBUG::2013-11-19 21:19:59,216::task::495::TaskManager.Task::(_debug) Task d64858dd-d362-4b97-a53f-c3715f509884: moving from state init -> state preparing >Thread-15910779::DEBUG::2013-11-19 21:19:59,219::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`168c7599-a437-4628-8308-fc6ad7c81915`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910779::DEBUG::2013-11-19 21:19:59,220::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910779::DEBUG::2013-11-19 21:19:59,223::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910779::DEBUG::2013-11-19 21:19:59,224::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`168c7599-a437-4628-8308-fc6ad7c81915`::Granted request >Thread-15910779::DEBUG::2013-11-19 21:19:59,242::task::495::TaskManager.Task::(_debug) Task d64858dd-d362-4b97-a53f-c3715f509884: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910779::DEBUG::2013-11-19 21:19:59,243::task::495::TaskManager.Task::(_debug) Task d64858dd-d362-4b97-a53f-c3715f509884: ref 1 aborting False >Thread-15910779::WARNING::2013-11-19 21:19:59,247::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910779::DEBUG::2013-11-19 21:19:59,249::task::495::TaskManager.Task::(_debug) Task d64858dd-d362-4b97-a53f-c3715f509884: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910779::DEBUG::2013-11-19 21:19:59,250::task::495::TaskManager.Task::(_debug) Task d64858dd-d362-4b97-a53f-c3715f509884: moving from state preparing -> state finished >Thread-15910779::DEBUG::2013-11-19 21:19:59,250::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910779::DEBUG::2013-11-19 21:19:59,252::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Dummy-15910548::DEBUG::2013-11-19 21:19:59,255::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0709386 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15910779::DEBUG::2013-11-19 21:19:59,255::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910779::DEBUG::2013-11-19 21:19:59,262::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910779::DEBUG::2013-11-19 21:19:59,272::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910779::DEBUG::2013-11-19 21:19:59,274::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910779::DEBUG::2013-11-19 21:19:59,275::task::495::TaskManager.Task::(_debug) Task d64858dd-d362-4b97-a53f-c3715f509884: ref 0 aborting False >Thread-15910779::INFO::2013-11-19 21:19:59,275::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910780::INFO::2013-11-19 21:20:00,904::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910780::DEBUG::2013-11-19 21:20:00,905::task::495::TaskManager.Task::(_debug) Task a3b42a13-3c6e-4b2e-98f8-cc0cee22cf4f: moving from state init -> state preparing >Thread-15910780::DEBUG::2013-11-19 21:20:00,905::task::495::TaskManager.Task::(_debug) Task a3b42a13-3c6e-4b2e-98f8-cc0cee22cf4f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00055193901062', 'lastCheck': 1384895992.2627511, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00130295753479', 'lastCheck': 1384895993.8322611, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895998.895349, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000479936599731', 'lastCheck': 1384895998.329006, 'valid': True, 'code': 0}} >Thread-15910780::DEBUG::2013-11-19 21:20:00,906::task::495::TaskManager.Task::(_debug) Task a3b42a13-3c6e-4b2e-98f8-cc0cee22cf4f: moving from state preparing -> state finished >Thread-15910780::DEBUG::2013-11-19 21:20:00,907::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910780::DEBUG::2013-11-19 21:20:00,907::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910780::DEBUG::2013-11-19 21:20:00,908::task::495::TaskManager.Task::(_debug) Task a3b42a13-3c6e-4b2e-98f8-cc0cee22cf4f: ref 0 aborting False >Thread-15910780::INFO::2013-11-19 21:20:00,908::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00055193901062', 'lastCheck': 1384895992.2627511, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00130295753479', 'lastCheck': 1384895993.8322611, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384895998.895349, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000479936599731', 'lastCheck': 1384895998.329006, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:20:01,276::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:20:01,818::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0672367 s, 15.2 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:20:03,834::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:20:04,356::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.070988 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:20:06,371::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:20:06,881::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0703209 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:20:08,896::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:20:09,147::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910786::DEBUG::2013-11-19 21:20:09,385::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910786::INFO::2013-11-19 21:20:09,388::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910786::DEBUG::2013-11-19 21:20:09,389::task::495::TaskManager.Task::(_debug) Task 06221832-3351-475b-a398-a2dd2328eed6: moving from state init -> state preparing >Thread-15910786::DEBUG::2013-11-19 21:20:09,391::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910786::DEBUG::2013-11-19 21:20:09,392::task::495::TaskManager.Task::(_debug) Task 06221832-3351-475b-a398-a2dd2328eed6: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910786::DEBUG::2013-11-19 21:20:09,393::task::495::TaskManager.Task::(_debug) Task 06221832-3351-475b-a398-a2dd2328eed6: moving from state preparing -> state finished >Thread-15910786::DEBUG::2013-11-19 21:20:09,394::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910786::DEBUG::2013-11-19 21:20:09,395::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910786::DEBUG::2013-11-19 21:20:09,397::task::495::TaskManager.Task::(_debug) Task 06221832-3351-475b-a398-a2dd2328eed6: ref 0 aborting False >Thread-15910786::INFO::2013-11-19 21:20:09,398::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910787::DEBUG::2013-11-19 21:20:09,406::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910787::INFO::2013-11-19 21:20:09,407::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910787::DEBUG::2013-11-19 21:20:09,408::task::495::TaskManager.Task::(_debug) Task 08abcc71-d7c3-4b03-b2a7-49bc47e6152d: moving from state init -> state preparing >Thread-15910787::DEBUG::2013-11-19 21:20:09,410::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a6a79b7c-f2d2-4a15-bbb2-0e5de5518ccd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15910787::DEBUG::2013-11-19 21:20:09,411::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15910787::DEBUG::2013-11-19 21:20:09,414::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15910787::DEBUG::2013-11-19 21:20:09,416::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a6a79b7c-f2d2-4a15-bbb2-0e5de5518ccd`::Granted request >Thread-15910787::DEBUG::2013-11-19 21:20:09,418::task::495::TaskManager.Task::(_debug) Task 08abcc71-d7c3-4b03-b2a7-49bc47e6152d: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15910787::DEBUG::2013-11-19 21:20:09,419::task::495::TaskManager.Task::(_debug) Task 08abcc71-d7c3-4b03-b2a7-49bc47e6152d: ref 1 aborting False >Thread-15910787::WARNING::2013-11-19 21:20:09,423::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910787::DEBUG::2013-11-19 21:20:09,425::task::495::TaskManager.Task::(_debug) Task 08abcc71-d7c3-4b03-b2a7-49bc47e6152d: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15910787::DEBUG::2013-11-19 21:20:09,426::task::495::TaskManager.Task::(_debug) Task 08abcc71-d7c3-4b03-b2a7-49bc47e6152d: moving from state preparing -> state finished >Thread-15910787::DEBUG::2013-11-19 21:20:09,427::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910787::DEBUG::2013-11-19 21:20:09,429::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910787::DEBUG::2013-11-19 21:20:09,431::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910787::DEBUG::2013-11-19 21:20:09,432::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910787::DEBUG::2013-11-19 21:20:09,433::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910787::DEBUG::2013-11-19 21:20:09,434::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910787::DEBUG::2013-11-19 21:20:09,436::task::495::TaskManager.Task::(_debug) Task 08abcc71-d7c3-4b03-b2a7-49bc47e6152d: ref 0 aborting False >Thread-15910787::INFO::2013-11-19 21:20:09,437::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 278}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15910548::DEBUG::2013-11-19 21:20:09,462::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0701178 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15910788::INFO::2013-11-19 21:20:11,159::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910788::DEBUG::2013-11-19 21:20:11,159::task::495::TaskManager.Task::(_debug) Task 20b7c2d8-2788-49fb-83b0-768087f2b751: moving from state init -> state preparing >Thread-15910788::DEBUG::2013-11-19 21:20:11,160::task::495::TaskManager.Task::(_debug) Task 20b7c2d8-2788-49fb-83b0-768087f2b751: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000427007675171', 'lastCheck': 1384896002.5115809, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148916244507', 'lastCheck': 1384896004.05053, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896009.151531, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000463008880615', 'lastCheck': 1384896008.5775249, 'valid': True, 'code': 0}} >Thread-15910788::DEBUG::2013-11-19 21:20:11,161::task::495::TaskManager.Task::(_debug) Task 20b7c2d8-2788-49fb-83b0-768087f2b751: moving from state preparing -> state finished >Thread-15910788::DEBUG::2013-11-19 21:20:11,161::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910788::DEBUG::2013-11-19 21:20:11,162::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910788::DEBUG::2013-11-19 21:20:11,162::task::495::TaskManager.Task::(_debug) Task 20b7c2d8-2788-49fb-83b0-768087f2b751: ref 0 aborting False >Thread-15910788::INFO::2013-11-19 21:20:11,163::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000427007675171', 'lastCheck': 1384896002.5115809, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148916244507', 'lastCheck': 1384896004.05053, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896009.151531, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000463008880615', 'lastCheck': 1384896008.5775249, 'valid': True, 'code': 0}} >Dummy-15910548::DEBUG::2013-11-19 21:20:11,477::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:20:12,007::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0704396 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15910548::DEBUG::2013-11-19 21:20:14,022::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:20:14,571::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0712785 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15910792::DEBUG::2013-11-19 21:20:16,204::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910792::INFO::2013-11-19 21:20:16,205::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksStatuses, args: () >Thread-15910792::DEBUG::2013-11-19 21:20:16,206::task::495::TaskManager.Task::(_debug) Task 00790317-3f6e-4891-b438-6a18cd85170a: moving from state init -> state preparing >Thread-15910792::DEBUG::2013-11-19 21:20:16,206::taskManager::92::TaskManager::(getAllTasksStatuses) Entry. >Thread-15910792::DEBUG::2013-11-19 21:20:16,207::taskManager::97::TaskManager::(getAllTasksStatuses) Return: {} >Thread-15910792::DEBUG::2013-11-19 21:20:16,207::task::495::TaskManager.Task::(_debug) Task 00790317-3f6e-4891-b438-6a18cd85170a: finished: {'allTasksStatus': {}} >Thread-15910792::DEBUG::2013-11-19 21:20:16,208::task::495::TaskManager.Task::(_debug) Task 00790317-3f6e-4891-b438-6a18cd85170a: moving from state preparing -> state finished >Thread-15910792::DEBUG::2013-11-19 21:20:16,209::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910792::DEBUG::2013-11-19 21:20:16,209::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910792::DEBUG::2013-11-19 21:20:16,210::task::495::TaskManager.Task::(_debug) Task 00790317-3f6e-4891-b438-6a18cd85170a: ref 0 aborting False >Thread-15910792::INFO::2013-11-19 21:20:16,210::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksStatuses, Return response: {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}} >Thread-15910793::DEBUG::2013-11-19 21:20:16,246::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910793::INFO::2013-11-19 21:20:16,247::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: deactivateStorageDomain, args: ( sdUUID=5d7b5178-3790-472c-a13b-b481eb27a61e spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 msdUUID=00000000-0000-0000-0000-000000000000 masterVersion=1) >Thread-15910793::DEBUG::2013-11-19 21:20:16,247::task::495::TaskManager.Task::(_debug) Task 5fdbc2a9-93a6-491e-afd0-c8300046a427: moving from state init -> state preparing >Thread-15910793::DEBUG::2013-11-19 21:20:16,249::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1e129772-23b3-4a80-9bc9-d8254465c6db`::Request was made in '/usr/share/vdsm/storage/spm.py' line '1112' at 'public_deactivateStorageDomain' >Thread-15910793::DEBUG::2013-11-19 21:20:16,249::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >Thread-15910793::DEBUG::2013-11-19 21:20:16,250::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >Thread-15910793::DEBUG::2013-11-19 21:20:16,251::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1e129772-23b3-4a80-9bc9-d8254465c6db`::Granted request >Thread-15910793::DEBUG::2013-11-19 21:20:16,252::task::495::TaskManager.Task::(_debug) Task 5fdbc2a9-93a6-491e-afd0-c8300046a427: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (exclusive) >Thread-15910793::DEBUG::2013-11-19 21:20:16,252::task::495::TaskManager.Task::(_debug) Task 5fdbc2a9-93a6-491e-afd0-c8300046a427: ref 1 aborting False >Thread-15910793::DEBUG::2013-11-19 21:20:16,253::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`8e68d170-b8f9-47cd-91ad-84886a61cf50`::Request was made in '/usr/share/vdsm/storage/spm.py' line '1113' at 'public_deactivateStorageDomain' >Thread-15910793::DEBUG::2013-11-19 21:20:16,254::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' for lock type 'exclusive' >Thread-15910793::DEBUG::2013-11-19 21:20:16,255::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' is free. Now locking as 'exclusive' (1 active user) >Thread-15910793::DEBUG::2013-11-19 21:20:16,255::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`8e68d170-b8f9-47cd-91ad-84886a61cf50`::Granted request >Thread-15910793::DEBUG::2013-11-19 21:20:16,256::task::495::TaskManager.Task::(_debug) Task 5fdbc2a9-93a6-491e-afd0-c8300046a427: _resourcesAcquired: Storage.5d7b5178-3790-472c-a13b-b481eb27a61e (exclusive) >Thread-15910793::DEBUG::2013-11-19 21:20:16,256::task::495::TaskManager.Task::(_debug) Task 5fdbc2a9-93a6-491e-afd0-c8300046a427: ref 1 aborting False >Thread-15910793::INFO::2013-11-19 21:20:16,257::sp::775::Storage.StoragePool::(deactivateSD) sdUUID=5d7b5178-3790-472c-a13b-b481eb27a61e spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 new_msdUUID=00000000-0000-0000-0000-000000000000 >Thread-15910793::INFO::2013-11-19 21:20:16,258::fileSD::176::Storage.StorageDomain::(validate) sdUUID=5d7b5178-3790-472c-a13b-b481eb27a61e >Thread-15910793::DEBUG::2013-11-19 21:20:16,259::persistentDict::204::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=[] >Thread-15910793::WARNING::2013-11-19 21:20:16,259::persistentDict::226::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-15910793::ERROR::2013-11-19 21:20:16,260::task::868::TaskManager.Task::(_setError) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 876, in _run > File "/usr/share/vdsm/storage/spm.py", line 115, in run > File "/usr/share/vdsm/storage/spm.py", line 1115, in public_deactivateStorageDomain > File "/usr/share/vdsm/storage/sp.py", line 798, in deactivateSD > File "/usr/share/vdsm/storage/sd.py", line 666, in isBackup > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15910793::DEBUG::2013-11-19 21:20:16,261::task::495::TaskManager.Task::(_debug) Task 5fdbc2a9-93a6-491e-afd0-c8300046a427: Task._run: 5fdbc2a9-93a6-491e-afd0-c8300046a427 ('5d7b5178-3790-472c-a13b-b481eb27a61e', 'c4e5889a-348d-11e2-83fe-9f72369eab30', '00000000-0000-0000-0000-000000000000', 1) {} failed - stopping task >Thread-15910793::DEBUG::2013-11-19 21:20:16,262::task::495::TaskManager.Task::(_debug) Task 5fdbc2a9-93a6-491e-afd0-c8300046a427: stopping in state preparing (force False) >Thread-15910793::DEBUG::2013-11-19 21:20:16,262::task::495::TaskManager.Task::(_debug) Task 5fdbc2a9-93a6-491e-afd0-c8300046a427: ref 1 aborting True >Thread-15910793::INFO::2013-11-19 21:20:16,263::task::1171::TaskManager.Task::(prepare) aborting: Task is aborted: "'CLASS'" - code 100 >Thread-15910793::DEBUG::2013-11-19 21:20:16,263::task::495::TaskManager.Task::(_debug) Task 5fdbc2a9-93a6-491e-afd0-c8300046a427: Prepare: aborted: 'CLASS' >Thread-15910793::DEBUG::2013-11-19 21:20:16,264::task::495::TaskManager.Task::(_debug) Task 5fdbc2a9-93a6-491e-afd0-c8300046a427: ref 0 aborting True >Thread-15910793::DEBUG::2013-11-19 21:20:16,264::task::495::TaskManager.Task::(_debug) Task 5fdbc2a9-93a6-491e-afd0-c8300046a427: Task._doAbort: force False >Thread-15910793::DEBUG::2013-11-19 21:20:16,265::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910793::DEBUG::2013-11-19 21:20:16,265::task::495::TaskManager.Task::(_debug) Task 5fdbc2a9-93a6-491e-afd0-c8300046a427: moving from state preparing -> state aborting >Thread-15910793::DEBUG::2013-11-19 21:20:16,266::task::495::TaskManager.Task::(_debug) Task 5fdbc2a9-93a6-491e-afd0-c8300046a427: _aborting: recover policy none >Thread-15910793::DEBUG::2013-11-19 21:20:16,266::task::495::TaskManager.Task::(_debug) Task 5fdbc2a9-93a6-491e-afd0-c8300046a427: moving from state aborting -> state failed >Thread-15910793::DEBUG::2013-11-19 21:20:16,267::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>, 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e': < ResourceRef 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e', isValid: 'True' obj: 'None'>} >Thread-15910793::DEBUG::2013-11-19 21:20:16,267::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910793::DEBUG::2013-11-19 21:20:16,268::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910793::DEBUG::2013-11-19 21:20:16,269::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910793::DEBUG::2013-11-19 21:20:16,269::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910793::DEBUG::2013-11-19 21:20:16,270::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910793::DEBUG::2013-11-19 21:20:16,270::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' >Thread-15910793::DEBUG::2013-11-19 21:20:16,271::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' (0 active users) >Thread-15910793::DEBUG::2013-11-19 21:20:16,271::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' is free, finding out if anyone is waiting for it. >Thread-15910793::DEBUG::2013-11-19 21:20:16,272::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e', Clearing records. >Thread-15910793::ERROR::2013-11-19 21:20:16,273::dispatcher::106::Storage.Dispatcher.Protect::(run) 'CLASS' >Thread-15910793::ERROR::2013-11-19 21:20:16,273::dispatcher::107::Storage.Dispatcher.Protect::(run) Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 96, in run > File "/usr/share/vdsm/storage/task.py", line 1178, in prepare >KeyError: 'CLASS' > >Thread-15910794::DEBUG::2013-11-19 21:20:16,287::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910794::INFO::2013-11-19 21:20:16,288::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910794::DEBUG::2013-11-19 21:20:16,288::task::495::TaskManager.Task::(_debug) Task fc7eb1aa-14af-4b50-9a9f-4b2943f7e58f: moving from state init -> state preparing >Thread-15910794::DEBUG::2013-11-19 21:20:16,289::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=278 spmId=2 >Thread-15910794::DEBUG::2013-11-19 21:20:16,290::task::495::TaskManager.Task::(_debug) Task fc7eb1aa-14af-4b50-9a9f-4b2943f7e58f: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910794::DEBUG::2013-11-19 21:20:16,290::task::495::TaskManager.Task::(_debug) Task fc7eb1aa-14af-4b50-9a9f-4b2943f7e58f: moving from state preparing -> state finished >Thread-15910794::DEBUG::2013-11-19 21:20:16,291::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910794::DEBUG::2013-11-19 21:20:16,291::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910794::DEBUG::2013-11-19 21:20:16,292::task::495::TaskManager.Task::(_debug) Task fc7eb1aa-14af-4b50-9a9f-4b2943f7e58f: ref 0 aborting False >Thread-15910794::INFO::2013-11-19 21:20:16,292::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 278}} >Thread-15910795::DEBUG::2013-11-19 21:20:16,299::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910795::INFO::2013-11-19 21:20:16,300::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksStatuses, args: () >Thread-15910795::DEBUG::2013-11-19 21:20:16,301::task::495::TaskManager.Task::(_debug) Task c0d88e0a-6cf4-418e-b7e6-aca6061398a9: moving from state init -> state preparing >Thread-15910795::DEBUG::2013-11-19 21:20:16,301::taskManager::92::TaskManager::(getAllTasksStatuses) Entry. >Thread-15910795::DEBUG::2013-11-19 21:20:16,302::taskManager::97::TaskManager::(getAllTasksStatuses) Return: {} >Thread-15910795::DEBUG::2013-11-19 21:20:16,302::task::495::TaskManager.Task::(_debug) Task c0d88e0a-6cf4-418e-b7e6-aca6061398a9: finished: {'allTasksStatus': {}} >Thread-15910795::DEBUG::2013-11-19 21:20:16,303::task::495::TaskManager.Task::(_debug) Task c0d88e0a-6cf4-418e-b7e6-aca6061398a9: moving from state preparing -> state finished >Thread-15910795::DEBUG::2013-11-19 21:20:16,304::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910795::DEBUG::2013-11-19 21:20:16,304::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910795::DEBUG::2013-11-19 21:20:16,305::task::495::TaskManager.Task::(_debug) Task c0d88e0a-6cf4-418e-b7e6-aca6061398a9: ref 0 aborting False >Thread-15910795::INFO::2013-11-19 21:20:16,305::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksStatuses, Return response: {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}} >Thread-15910796::DEBUG::2013-11-19 21:20:16,309::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910796::INFO::2013-11-19 21:20:16,310::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: spmStop, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910796::DEBUG::2013-11-19 21:20:16,310::task::495::TaskManager.Task::(_debug) Task f40f092b-3e0b-4e58-b570-13ecf7022426: moving from state init -> state preparing >Thread-15910796::DEBUG::2013-11-19 21:20:16,311::spm::547::Storage.SPM::(public_spmStop) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 >Thread-15910796::DEBUG::2013-11-19 21:20:16,311::taskManager::105::TaskManager::(getAllTasks) Entry. >Thread-15910796::DEBUG::2013-11-19 21:20:16,312::taskManager::110::TaskManager::(getAllTasks) Return: {} >Thread-15910796::DEBUG::2013-11-19 21:20:16,313::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`efeb1bbf-ea4e-41f2-bbd3-8d3969d320a5`::Request was made in '/usr/share/vdsm/storage/spm.py' line '573' at 'public_spmStop' >Thread-15910796::DEBUG::2013-11-19 21:20:16,313::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >Thread-15910796::DEBUG::2013-11-19 21:20:16,314::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >Thread-15910796::DEBUG::2013-11-19 21:20:16,315::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`efeb1bbf-ea4e-41f2-bbd3-8d3969d320a5`::Granted request >Thread-15910796::DEBUG::2013-11-19 21:20:16,316::task::495::TaskManager.Task::(_debug) Task f40f092b-3e0b-4e58-b570-13ecf7022426: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (exclusive) >Thread-15910796::DEBUG::2013-11-19 21:20:16,316::task::495::TaskManager.Task::(_debug) Task f40f092b-3e0b-4e58-b570-13ecf7022426: ref 1 aborting False >Thread-15910796::DEBUG::2013-11-19 21:20:16,317::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bbd5b86e-8d30-4418-b7b3-fb4db6c7aa32`::Request was made in '/usr/share/vdsm/storage/spm.py' line '584' at '_stop' >Thread-15910796::DEBUG::2013-11-19 21:20:16,318::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >Thread-15910796::DEBUG::2013-11-19 21:20:16,318::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >Thread-15910796::DEBUG::2013-11-19 21:20:16,319::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bbd5b86e-8d30-4418-b7b3-fb4db6c7aa32`::Granted request >Thread-15910796::DEBUG::2013-11-19 21:20:16,320::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`34dcfc2e-6993-4439-8809-c361f883ca4c`::Request was made in '/usr/share/vdsm/storage/spm.py' line '601' at '_stop' >Thread-15910796::DEBUG::2013-11-19 21:20:16,321::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' for lock type 'exclusive' >Thread-15910796::DEBUG::2013-11-19 21:20:16,321::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' is free. Now locking as 'exclusive' (1 active user) >Thread-15910796::DEBUG::2013-11-19 21:20:16,322::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`34dcfc2e-6993-4439-8809-c361f883ca4c`::Granted request >Thread-15910796::DEBUG::2013-11-19 21:20:16,323::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' >Thread-15910796::DEBUG::2013-11-19 21:20:16,323::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' (0 active users) >Thread-15910796::DEBUG::2013-11-19 21:20:16,324::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' is free, finding out if anyone is waiting for it. >Thread-15910796::DEBUG::2013-11-19 21:20:16,324::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e', Clearing records. >Thread-15910796::DEBUG::2013-11-19 21:20:16,325::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910796::DEBUG::2013-11-19 21:20:16,326::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910796::DEBUG::2013-11-19 21:20:16,326::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910796::DEBUG::2013-11-19 21:20:16,327::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910796::DEBUG::2013-11-19 21:20:16,327::taskManager::118::TaskManager::(unloadTasks) Entry. >Thread-15910796::DEBUG::2013-11-19 21:20:16,328::taskManager::122::TaskManager::(unloadTasks) Return >Thread-15910796::DEBUG::2013-11-19 21:20:16,330::spm::195::Storage.SPM::(__cleanupMasterMount) unmounting /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master >Thread-15910796::DEBUG::2013-11-19 21:20:16,449::fileUtils::112::Storage.Misc.excCmd::(umount) '/usr/bin/sudo -n /bin/umount -f /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:20:16,635::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15910796::DEBUG::2013-11-19 21:20:16,944::fileUtils::112::Storage.Misc.excCmd::(umount) SUCCESS: <err> = ''; <rc> = 0 >Thread-15910796::DEBUG::2013-11-19 21:20:16,948::spm::290::Storage.SPM::(__cleanupSPM) cleaning up SPM: c4e5889a-348d-11e2-83fe-9f72369eab30 >Thread-15910796::DEBUG::2013-11-19 21:20:16,949::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >Thread-15910796::DEBUG::2013-11-19 21:20:16,950::persistentDict::146::Storage.PersistentDict::(transaction) Flushing changes >Thread-15910796::DEBUG::2013-11-19 21:20:16,951::persistentDict::265::Storage.PersistentDict::(flush) about to write lines (VGTagMetadataRW)=[u'CLASS=Data', u'DESCRIPTION=OS', u'IOOPTIMEOUTSEC=10', u'LEASERETRIES=3', u'LEASETIMESEC=60', u'LOCKPOLICY=', u'LOCKRENEWALINTERVALSEC=5', u'LOGBLKSIZE=512', u'MASTER_VERSION=1', u'PHYBLKSIZE=512', u'POOL_DESCRIPTION=Default', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_ID=-1', u'POOL_SPM_LVER=278', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'ROLE=Master', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'TYPE=FCP', u'VERSION=2', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', '_SHA_CKSUM=9f9e428092a2f2a5a408c3df7cae07b957db8170'] >Thread-15910796::DEBUG::2013-11-19 21:20:16,952::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15910796::DEBUG::2013-11-19 21:20:16,953::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15910796::DEBUG::2013-11-19 21:20:16,954::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15910796::DEBUG::2013-11-19 21:20:16,954::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15910796::DEBUG::2013-11-19 21:20:16,955::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15910796::DEBUG::2013-11-19 21:20:16,957::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15910548::DEBUG::2013-11-19 21:20:17,440::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.137761 s, 7.4 MB/s\n'; <rc> = 0 >Thread-15910796::DEBUG::2013-11-19 21:20:17,763::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15910796::DEBUG::2013-11-19 21:20:17,766::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15910796::DEBUG::2013-11-19 21:20:17,768::blockSD::145::storage.Metadata.VGTagMetadataRW::(writelines) Updating metadata adding=MDT_POOL_SPM_ID=-1, MDT__SHA_CKSUM=9f9e428092a2f2a5a408c3df7cae07b957db8170 removing=MDT__SHA_CKSUM=913ce85a0b55e60c3f92c982d8a15ae168c40596, MDT_POOL_SPM_ID=2 >Thread-15910796::DEBUG::2013-11-19 21:20:17,769::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --deltag MDT__SHA_CKSUM=913ce85a0b55e60c3f92c982d8a15ae168c40596 --deltag MDT_POOL_SPM_ID=2 --addtag MDT__SHA_CKSUM=9f9e428092a2f2a5a408c3df7cae07b957db8170 --addtag MDT_POOL_SPM_ID=-1 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15910796::DEBUG::2013-11-19 21:20:18,493::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15910796::DEBUG::2013-11-19 21:20:18,494::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15910796::DEBUG::2013-11-19 21:20:18,495::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15910796::DEBUG::2013-11-19 21:20:18,496::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >Thread-15910796::INFO::2013-11-19 21:20:18,497::safelease::83::ClusterLock::(release) Releasing cluster lock for domain 6b892067-ac7e-4039-a7be-5b7387525b16 >Thread-15910796::DEBUG::2013-11-19 21:20:18,497::safelease::84::Storage.Misc.excCmd::(release) '/usr/libexec/vdsm/spmstop.sh 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd /usr/libexec/vdsm/) >Thread-1024767::ERROR::2013-11-19 21:20:19,302::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15910548::INFO::2013-11-19 21:20:19,456::storage_mailbox::673::Storage.MailBox.SpmMailMonitor::(run) SPM_MailMonitor - Incoming mail monitoring thread stopped >Thread-15910796::DEBUG::2013-11-19 21:20:19,970::safelease::84::Storage.Misc.excCmd::(release) SUCCESS: <err> = ''; <rc> = 0 >Thread-15910796::DEBUG::2013-11-19 21:20:19,971::safelease::87::ClusterLock::(release) Cluster lock released successfully >Thread-15910796::DEBUG::2013-11-19 21:20:19,972::task::495::TaskManager.Task::(_debug) Task f40f092b-3e0b-4e58-b570-13ecf7022426: finished: None >Thread-15910796::DEBUG::2013-11-19 21:20:19,972::task::495::TaskManager.Task::(_debug) Task f40f092b-3e0b-4e58-b570-13ecf7022426: moving from state preparing -> state finished >Thread-15910796::DEBUG::2013-11-19 21:20:19,973::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15910796::DEBUG::2013-11-19 21:20:19,974::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910796::DEBUG::2013-11-19 21:20:19,974::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910796::DEBUG::2013-11-19 21:20:19,975::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910796::DEBUG::2013-11-19 21:20:19,976::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910796::DEBUG::2013-11-19 21:20:19,976::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910796::DEBUG::2013-11-19 21:20:19,977::task::495::TaskManager.Task::(_debug) Task f40f092b-3e0b-4e58-b570-13ecf7022426: ref 0 aborting False >Thread-15910796::INFO::2013-11-19 21:20:19,978::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: spmStop, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15910799::INFO::2013-11-19 21:20:21,328::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910799::DEBUG::2013-11-19 21:20:21,329::task::495::TaskManager.Task::(_debug) Task 9b6481ea-707a-40fd-840d-bbbd3a096195: moving from state init -> state preparing >Thread-15910799::WARNING::2013-11-19 21:20:21,329::hsm::1895::Storage.HSM::(public_repoStats) repoStats detected invalid master:6b892067-ac7e-4039-a7be-5b7387525b16 {'finish': 1384896012.6998551, 'mdasize': '134479872', 'mdathreshold': True, 'mdavalid': True, 'diskfree': '1142595518464', 'disktotal': '3595827150848', 'result': {'delay': '0.000464916229248', 'lastCheck': 1384896012.6998551, 'valid': True, 'code': 0}, 'masterValidate': {'mount': True, 'valid': True}, 'mdafree': '67205632'} >Thread-15910799::DEBUG::2013-11-19 21:20:21,330::task::495::TaskManager.Task::(_debug) Task 9b6481ea-707a-40fd-840d-bbbd3a096195: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000464916229248', 'lastCheck': 1384896012.6998551, 'valid': True, 'code': 355}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00149393081665', 'lastCheck': 1384896014.275703, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896019.3036189, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00049901008606', 'lastCheck': 1384896019.007432, 'valid': True, 'code': 0}} >Thread-15910799::DEBUG::2013-11-19 21:20:21,331::task::495::TaskManager.Task::(_debug) Task 9b6481ea-707a-40fd-840d-bbbd3a096195: moving from state preparing -> state finished >Thread-15910799::DEBUG::2013-11-19 21:20:21,331::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910799::DEBUG::2013-11-19 21:20:21,332::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910799::DEBUG::2013-11-19 21:20:21,333::task::495::TaskManager.Task::(_debug) Task 9b6481ea-707a-40fd-840d-bbbd3a096195: ref 0 aborting False >Thread-15910799::INFO::2013-11-19 21:20:21,333::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000464916229248', 'lastCheck': 1384896012.6998551, 'valid': True, 'code': 355}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00149393081665', 'lastCheck': 1384896014.275703, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896019.3036189, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00049901008606', 'lastCheck': 1384896019.007432, 'valid': True, 'code': 0}} >Thread-42838::DEBUG::2013-11-19 21:20:23,015::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-42838::DEBUG::2013-11-19 21:20:23,016::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 21:20:23,593::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-42838::DEBUG::2013-11-19 21:20:23,596::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1024767::ERROR::2013-11-19 21:20:29,317::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910805::INFO::2013-11-19 21:20:31,433::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910805::DEBUG::2013-11-19 21:20:31,434::task::495::TaskManager.Task::(_debug) Task 34412443-9cf3-4d9a-a169-2e5f98c5e011: moving from state init -> state preparing >Thread-15910805::DEBUG::2013-11-19 21:20:31,435::task::495::TaskManager.Task::(_debug) Task 34412443-9cf3-4d9a-a169-2e5f98c5e011: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000357151031494', 'lastCheck': 1384896023.5994561, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148010253906', 'lastCheck': 1384896024.3090329, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896029.318846, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000409126281738', 'lastCheck': 1384896029.0193429, 'valid': True, 'code': 0}} >Thread-15910805::DEBUG::2013-11-19 21:20:31,435::task::495::TaskManager.Task::(_debug) Task 34412443-9cf3-4d9a-a169-2e5f98c5e011: moving from state preparing -> state finished >Thread-15910805::DEBUG::2013-11-19 21:20:31,436::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910805::DEBUG::2013-11-19 21:20:31,437::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910805::DEBUG::2013-11-19 21:20:31,437::task::495::TaskManager.Task::(_debug) Task 34412443-9cf3-4d9a-a169-2e5f98c5e011: ref 0 aborting False >Thread-15910805::INFO::2013-11-19 21:20:31,438::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000357151031494', 'lastCheck': 1384896023.5994561, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148010253906', 'lastCheck': 1384896024.3090329, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896029.318846, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000409126281738', 'lastCheck': 1384896029.0193429, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:20:39,331::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910811::INFO::2013-11-19 21:20:41,562::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910811::DEBUG::2013-11-19 21:20:41,562::task::495::TaskManager.Task::(_debug) Task 0f09536d-2115-484e-be99-4bacf636bae6: moving from state init -> state preparing >Thread-15910811::DEBUG::2013-11-19 21:20:41,563::task::495::TaskManager.Task::(_debug) Task 0f09536d-2115-484e-be99-4bacf636bae6: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000357866287231', 'lastCheck': 1384896033.6125431, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150203704834', 'lastCheck': 1384896034.3245611, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896039.3427711, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00388693809509', 'lastCheck': 1384896039.035394, 'valid': True, 'code': 0}} >Thread-15910811::DEBUG::2013-11-19 21:20:41,564::task::495::TaskManager.Task::(_debug) Task 0f09536d-2115-484e-be99-4bacf636bae6: moving from state preparing -> state finished >Thread-15910811::DEBUG::2013-11-19 21:20:41,564::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910811::DEBUG::2013-11-19 21:20:41,565::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910811::DEBUG::2013-11-19 21:20:41,565::task::495::TaskManager.Task::(_debug) Task 0f09536d-2115-484e-be99-4bacf636bae6: ref 0 aborting False >Thread-15910811::INFO::2013-11-19 21:20:41,566::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000357866287231', 'lastCheck': 1384896033.6125431, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150203704834', 'lastCheck': 1384896034.3245611, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896039.3427711, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00388693809509', 'lastCheck': 1384896039.035394, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:20:49,355::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910817::INFO::2013-11-19 21:20:51,690::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910817::DEBUG::2013-11-19 21:20:51,691::task::495::TaskManager.Task::(_debug) Task 75574db4-63ef-448c-8788-09c2a345bbea: moving from state init -> state preparing >Thread-15910817::DEBUG::2013-11-19 21:20:51,692::task::495::TaskManager.Task::(_debug) Task 75574db4-63ef-448c-8788-09c2a345bbea: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000338077545166', 'lastCheck': 1384896043.6254671, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0022280216217', 'lastCheck': 1384896044.3410809, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896049.35642, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00588321685791', 'lastCheck': 1384896049.051645, 'valid': True, 'code': 0}} >Thread-15910817::DEBUG::2013-11-19 21:20:51,693::task::495::TaskManager.Task::(_debug) Task 75574db4-63ef-448c-8788-09c2a345bbea: moving from state preparing -> state finished >Thread-15910817::DEBUG::2013-11-19 21:20:51,693::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910817::DEBUG::2013-11-19 21:20:51,694::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910817::DEBUG::2013-11-19 21:20:51,694::task::495::TaskManager.Task::(_debug) Task 75574db4-63ef-448c-8788-09c2a345bbea: ref 0 aborting False >Thread-15910817::INFO::2013-11-19 21:20:51,695::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000338077545166', 'lastCheck': 1384896043.6254671, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0022280216217', 'lastCheck': 1384896044.3410809, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896049.35642, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00588321685791', 'lastCheck': 1384896049.051645, 'valid': True, 'code': 0}} >Thread-15910821::DEBUG::2013-11-19 21:20:56,381::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910821::INFO::2013-11-19 21:20:56,382::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910821::DEBUG::2013-11-19 21:20:56,382::task::495::TaskManager.Task::(_debug) Task a2e49274-8c78-484c-99f1-5925ccab7cfd: moving from state init -> state preparing >Thread-15910821::DEBUG::2013-11-19 21:20:56,383::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=Free spmLver=278 spmId=-1 >Thread-15910821::DEBUG::2013-11-19 21:20:56,383::task::495::TaskManager.Task::(_debug) Task a2e49274-8c78-484c-99f1-5925ccab7cfd: finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 278}} >Thread-15910821::DEBUG::2013-11-19 21:20:56,384::task::495::TaskManager.Task::(_debug) Task a2e49274-8c78-484c-99f1-5925ccab7cfd: moving from state preparing -> state finished >Thread-15910821::DEBUG::2013-11-19 21:20:56,385::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910821::DEBUG::2013-11-19 21:20:56,385::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910821::DEBUG::2013-11-19 21:20:56,386::task::495::TaskManager.Task::(_debug) Task a2e49274-8c78-484c-99f1-5925ccab7cfd: ref 0 aborting False >Thread-15910821::INFO::2013-11-19 21:20:56,386::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 278}} >Thread-15910822::DEBUG::2013-11-19 21:20:56,409::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910822::INFO::2013-11-19 21:20:56,410::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: spmStart, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 prevID=-1 prevLVER=278 recoveryMode=0 scsiFencing=false maxHostID=250 domVersion=2) >Thread-15910822::DEBUG::2013-11-19 21:20:56,411::task::495::TaskManager.Task::(_debug) Task 934c4d35-aab6-4fba-9996-9cc07dd7d991: moving from state init -> state preparing >Thread-15910822::DEBUG::2013-11-19 21:20:56,412::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f2292637-8eb5-4d89-b0e9-66555f85b956`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '330' at 'public_spmStart' >Thread-15910822::DEBUG::2013-11-19 21:20:56,412::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >Thread-15910822::DEBUG::2013-11-19 21:20:56,413::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >Thread-15910822::DEBUG::2013-11-19 21:20:56,414::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f2292637-8eb5-4d89-b0e9-66555f85b956`::Granted request >Thread-15910822::DEBUG::2013-11-19 21:20:56,415::task::495::TaskManager.Task::(_debug) Task 934c4d35-aab6-4fba-9996-9cc07dd7d991: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (exclusive) >Thread-15910822::DEBUG::2013-11-19 21:20:56,415::task::495::TaskManager.Task::(_debug) Task 934c4d35-aab6-4fba-9996-9cc07dd7d991: ref 1 aborting False >Thread-15910822::DEBUG::2013-11-19 21:20:56,416::task::495::TaskManager.Task::(_debug) Task 934c4d35-aab6-4fba-9996-9cc07dd7d991: Prepare: 1 jobs exist, move to acquiring >Thread-15910822::DEBUG::2013-11-19 21:20:56,416::task::495::TaskManager.Task::(_debug) Task 934c4d35-aab6-4fba-9996-9cc07dd7d991: moving from state preparing -> state acquiring >Thread-15910822::DEBUG::2013-11-19 21:20:56,417::task::495::TaskManager.Task::(_debug) Task 934c4d35-aab6-4fba-9996-9cc07dd7d991: moving from state acquiring -> state queued >Thread-15910822::DEBUG::2013-11-19 21:20:56,417::taskManager::37::TaskManager::(_queueTask) queueing task: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910822::DEBUG::2013-11-19 21:20:56,418::taskManager::43::TaskManager::(_queueTask) task queued: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >eca29df8-c91e-46f6-8df8-b5d172c88b9f::DEBUG::2013-11-19 21:20:56,418::threadPool::47::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-15910822::DEBUG::2013-11-19 21:20:56,419::task::1183::TaskManager.Task::(prepare) returning >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:56,420::threadPool::192::Misc.ThreadPool.WorkerThread::(run) Task: 934c4d35-aab6-4fba-9996-9cc07dd7d991 running: <bound method Task.commit of <storage.task.Task instance at 0x7f487c1379e0>> with: None >Thread-15910822::DEBUG::2013-11-19 21:20:56,420::task::495::TaskManager.Task::(_debug) Task 934c4d35-aab6-4fba-9996-9cc07dd7d991: ref 0 aborting False >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:56,421::task::1195::TaskManager.Task::(commit) Commiting task: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910822::INFO::2013-11-19 21:20:56,421::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: spmStart, Return response: {'status': {'message': 'OK', 'code': 0}, 'uuid': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:56,422::task::495::TaskManager.Task::(_debug) Task 934c4d35-aab6-4fba-9996-9cc07dd7d991: moving from state queued -> state running >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:56,423::task::907::TaskManager.Task::(_runJobs) Task.run: running job 0: spmStart: <bound method SPM.start of <storage.spm.SPM instance at 0x1bf9f38>> (args: ('c4e5889a-348d-11e2-83fe-9f72369eab30', -1, '278', 0, 'false', 250, 2) kwargs: {}) >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:56,424::task::300::TaskManager.Task::(run) Job.run: running spmStart: <bound method SPM.start of <storage.spm.SPM instance at 0x1bf9f38>> (args: ('c4e5889a-348d-11e2-83fe-9f72369eab30', -1, '278', 0, 'false', 250, 2) kwargs: {}) callback None >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:56,424::spm::433::Storage.SPM::(start) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:56,425::misc::1010::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:56,425::misc::1012::SamplingMethod::(__call__) Got in to sampling method >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:56,426::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:56,426::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:56,427::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:56,428::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:56,428::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:56,429::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:57,119::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:57,122::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:57,124::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=279', u'_SHA_CKSUM=b5dee775aac07995010b8c6a5db186617df84cf4', u'POOL_SPM_ID=-1'] >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:57,125::misc::1020::SamplingMethod::(__call__) Returning last result >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:57,126::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:57,128::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:57,129::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:57,129::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:57,130::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:57,131::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15910823::DEBUG::2013-11-19 21:20:57,430::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910823::INFO::2013-11-19 21:20:57,432::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910823::DEBUG::2013-11-19 21:20:57,433::task::495::TaskManager.Task::(_debug) Task 4f9b8411-2d56-494e-b3fb-0510e0fd7c57: moving from state init -> state preparing >Thread-15910823::DEBUG::2013-11-19 21:20:57,433::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910823::DEBUG::2013-11-19 21:20:57,434::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910823::DEBUG::2013-11-19 21:20:57,435::task::495::TaskManager.Task::(_debug) Task 4f9b8411-2d56-494e-b3fb-0510e0fd7c57: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910823::DEBUG::2013-11-19 21:20:57,435::task::495::TaskManager.Task::(_debug) Task 4f9b8411-2d56-494e-b3fb-0510e0fd7c57: moving from state preparing -> state finished >Thread-15910823::DEBUG::2013-11-19 21:20:57,436::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910823::DEBUG::2013-11-19 21:20:57,436::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910823::DEBUG::2013-11-19 21:20:57,437::task::495::TaskManager.Task::(_debug) Task 4f9b8411-2d56-494e-b3fb-0510e0fd7c57: ref 0 aborting False >Thread-15910823::INFO::2013-11-19 21:20:57,438::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:57,788::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:57,791::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:57,792::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=279', u'_SHA_CKSUM=b5dee775aac07995010b8c6a5db186617df84cf4', u'POOL_SPM_ID=-1'] >934c4d35-aab6-4fba-9996-9cc07dd7d991::INFO::2013-11-19 21:20:57,793::spm::457::Storage.SPM::(start) expected previd:-1 lver:279 got request for previd:-1 lver:278 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:57,794::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:57,795::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:57,796::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:57,796::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:57,797::persistentDict::163::Storage.PersistentDict::(__init__) Created a persistant dict with LvMetadataRW backend >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:57,798::blockSD::164::Storage.Misc.excCmd::(readlines) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:58,203::blockSD::164::Storage.Misc.excCmd::(readlines) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000230349 s, 8.9 MB/s\n'; <rc> = 0 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:58,204::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000230349 s, 8.9 MB/s'], size: 2048 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:58,205::persistentDict::204::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >934c4d35-aab6-4fba-9996-9cc07dd7d991::WARNING::2013-11-19 21:20:58,205::persistentDict::226::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:58,206::persistentDict::163::Storage.PersistentDict::(__init__) Created a persistant dict with VGTagMetadataRW backend >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:58,207::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:58,207::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:58,208::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:58,209::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:58,209::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:58,210::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15910825::DEBUG::2013-11-19 21:20:58,447::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910825::INFO::2013-11-19 21:20:58,450::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910825::DEBUG::2013-11-19 21:20:58,450::task::495::TaskManager.Task::(_debug) Task 7c11ac28-8b29-419d-bfe0-68cfca23a5b9: moving from state init -> state preparing >Thread-15910825::DEBUG::2013-11-19 21:20:58,451::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910825::DEBUG::2013-11-19 21:20:58,452::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910825::DEBUG::2013-11-19 21:20:58,452::task::495::TaskManager.Task::(_debug) Task 7c11ac28-8b29-419d-bfe0-68cfca23a5b9: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910825::DEBUG::2013-11-19 21:20:58,453::task::495::TaskManager.Task::(_debug) Task 7c11ac28-8b29-419d-bfe0-68cfca23a5b9: moving from state preparing -> state finished >Thread-15910825::DEBUG::2013-11-19 21:20:58,453::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910825::DEBUG::2013-11-19 21:20:58,454::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910825::DEBUG::2013-11-19 21:20:58,455::task::495::TaskManager.Task::(_debug) Task 7c11ac28-8b29-419d-bfe0-68cfca23a5b9: ref 0 aborting False >Thread-15910825::INFO::2013-11-19 21:20:58,455::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:58,845::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:58,848::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:58,849::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=279', u'_SHA_CKSUM=b5dee775aac07995010b8c6a5db186617df84cf4', u'POOL_SPM_ID=-1'] >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:58,850::safelease::52::ClusterLock::(acquire) Acquiring cluster lock for domain 6b892067-ac7e-4039-a7be-5b7387525b16 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:20:58,851::safelease::61::Storage.Misc.excCmd::(acquire) '/usr/bin/sudo -n /usr/bin/setsid /usr/bin/ionice -c1 -n0 /bin/su vdsm -s /bin/sh -c "/usr/libexec/vdsm/spmprotect.sh start 6b892067-ac7e-4039-a7be-5b7387525b16 2 5 /dev/6b892067-ac7e-4039-a7be-5b7387525b16/leases 60000 10000 3"' (cwd /usr/libexec/vdsm/) >Thread-1024767::ERROR::2013-11-19 21:20:59,418::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910826::DEBUG::2013-11-19 21:20:59,463::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910826::INFO::2013-11-19 21:20:59,464::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910826::DEBUG::2013-11-19 21:20:59,465::task::495::TaskManager.Task::(_debug) Task f9db044d-9d2b-4481-a225-8522427dd4ad: moving from state init -> state preparing >Thread-15910826::DEBUG::2013-11-19 21:20:59,465::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910826::DEBUG::2013-11-19 21:20:59,466::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910826::DEBUG::2013-11-19 21:20:59,467::task::495::TaskManager.Task::(_debug) Task f9db044d-9d2b-4481-a225-8522427dd4ad: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910826::DEBUG::2013-11-19 21:20:59,467::task::495::TaskManager.Task::(_debug) Task f9db044d-9d2b-4481-a225-8522427dd4ad: moving from state preparing -> state finished >Thread-15910826::DEBUG::2013-11-19 21:20:59,468::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910826::DEBUG::2013-11-19 21:20:59,468::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910826::DEBUG::2013-11-19 21:20:59,469::task::495::TaskManager.Task::(_debug) Task f9db044d-9d2b-4481-a225-8522427dd4ad: ref 0 aborting False >Thread-15910826::INFO::2013-11-19 21:20:59,469::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910828::DEBUG::2013-11-19 21:21:00,475::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910828::INFO::2013-11-19 21:21:00,476::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910828::DEBUG::2013-11-19 21:21:00,477::task::495::TaskManager.Task::(_debug) Task b7645369-6ad2-47fa-a153-f9100e0b9331: moving from state init -> state preparing >Thread-15910828::DEBUG::2013-11-19 21:21:00,477::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910828::DEBUG::2013-11-19 21:21:00,478::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910828::DEBUG::2013-11-19 21:21:00,478::task::495::TaskManager.Task::(_debug) Task b7645369-6ad2-47fa-a153-f9100e0b9331: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910828::DEBUG::2013-11-19 21:21:00,479::task::495::TaskManager.Task::(_debug) Task b7645369-6ad2-47fa-a153-f9100e0b9331: moving from state preparing -> state finished >Thread-15910828::DEBUG::2013-11-19 21:21:00,479::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910828::DEBUG::2013-11-19 21:21:00,480::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910828::DEBUG::2013-11-19 21:21:00,481::task::495::TaskManager.Task::(_debug) Task b7645369-6ad2-47fa-a153-f9100e0b9331: ref 0 aborting False >Thread-15910828::INFO::2013-11-19 21:21:00,481::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910829::DEBUG::2013-11-19 21:21:01,487::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910829::INFO::2013-11-19 21:21:01,487::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910829::DEBUG::2013-11-19 21:21:01,488::task::495::TaskManager.Task::(_debug) Task ca945ac7-f9b6-4ab2-8e0c-3689483d1db1: moving from state init -> state preparing >Thread-15910829::DEBUG::2013-11-19 21:21:01,489::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910829::DEBUG::2013-11-19 21:21:01,489::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910829::DEBUG::2013-11-19 21:21:01,490::task::495::TaskManager.Task::(_debug) Task ca945ac7-f9b6-4ab2-8e0c-3689483d1db1: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910829::DEBUG::2013-11-19 21:21:01,490::task::495::TaskManager.Task::(_debug) Task ca945ac7-f9b6-4ab2-8e0c-3689483d1db1: moving from state preparing -> state finished >Thread-15910829::DEBUG::2013-11-19 21:21:01,491::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910829::DEBUG::2013-11-19 21:21:01,491::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910829::DEBUG::2013-11-19 21:21:01,492::task::495::TaskManager.Task::(_debug) Task ca945ac7-f9b6-4ab2-8e0c-3689483d1db1: ref 0 aborting False >Thread-15910829::INFO::2013-11-19 21:21:01,492::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910830::INFO::2013-11-19 21:21:01,799::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910830::DEBUG::2013-11-19 21:21:01,799::task::495::TaskManager.Task::(_debug) Task 6abd6968-5998-4fac-a45f-9c016d4518c5: moving from state init -> state preparing >Thread-15910830::DEBUG::2013-11-19 21:21:01,800::task::495::TaskManager.Task::(_debug) Task 6abd6968-5998-4fac-a45f-9c016d4518c5: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000318050384521', 'lastCheck': 1384896053.638303, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00149297714233', 'lastCheck': 1384896054.3564379, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896059.4198689, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000597953796387', 'lastCheck': 1384896059.064239, 'valid': True, 'code': 0}} >Thread-15910830::DEBUG::2013-11-19 21:21:01,801::task::495::TaskManager.Task::(_debug) Task 6abd6968-5998-4fac-a45f-9c016d4518c5: moving from state preparing -> state finished >Thread-15910830::DEBUG::2013-11-19 21:21:01,801::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910830::DEBUG::2013-11-19 21:21:01,802::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910830::DEBUG::2013-11-19 21:21:01,802::task::495::TaskManager.Task::(_debug) Task 6abd6968-5998-4fac-a45f-9c016d4518c5: ref 0 aborting False >Thread-15910830::INFO::2013-11-19 21:21:01,803::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000318050384521', 'lastCheck': 1384896053.638303, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00149297714233', 'lastCheck': 1384896054.3564379, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896059.4198689, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000597953796387', 'lastCheck': 1384896059.064239, 'valid': True, 'code': 0}} >Thread-15910832::DEBUG::2013-11-19 21:21:02,498::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910832::INFO::2013-11-19 21:21:02,499::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910832::DEBUG::2013-11-19 21:21:02,499::task::495::TaskManager.Task::(_debug) Task f44d5c0c-113b-47b0-b5fe-6e1997e83206: moving from state init -> state preparing >Thread-15910832::DEBUG::2013-11-19 21:21:02,500::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910832::DEBUG::2013-11-19 21:21:02,501::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910832::DEBUG::2013-11-19 21:21:02,501::task::495::TaskManager.Task::(_debug) Task f44d5c0c-113b-47b0-b5fe-6e1997e83206: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910832::DEBUG::2013-11-19 21:21:02,502::task::495::TaskManager.Task::(_debug) Task f44d5c0c-113b-47b0-b5fe-6e1997e83206: moving from state preparing -> state finished >Thread-15910832::DEBUG::2013-11-19 21:21:02,503::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910832::DEBUG::2013-11-19 21:21:02,503::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910832::DEBUG::2013-11-19 21:21:02,504::task::495::TaskManager.Task::(_debug) Task f44d5c0c-113b-47b0-b5fe-6e1997e83206: ref 0 aborting False >Thread-15910832::INFO::2013-11-19 21:21:02,504::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910833::DEBUG::2013-11-19 21:21:03,510::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910833::INFO::2013-11-19 21:21:03,511::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910833::DEBUG::2013-11-19 21:21:03,511::task::495::TaskManager.Task::(_debug) Task 5b7dafa1-9877-4ca8-8baf-fecf9b06b7df: moving from state init -> state preparing >Thread-15910833::DEBUG::2013-11-19 21:21:03,512::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910833::DEBUG::2013-11-19 21:21:03,512::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910833::DEBUG::2013-11-19 21:21:03,513::task::495::TaskManager.Task::(_debug) Task 5b7dafa1-9877-4ca8-8baf-fecf9b06b7df: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910833::DEBUG::2013-11-19 21:21:03,514::task::495::TaskManager.Task::(_debug) Task 5b7dafa1-9877-4ca8-8baf-fecf9b06b7df: moving from state preparing -> state finished >Thread-15910833::DEBUG::2013-11-19 21:21:03,514::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910833::DEBUG::2013-11-19 21:21:03,515::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910833::DEBUG::2013-11-19 21:21:03,515::task::495::TaskManager.Task::(_debug) Task 5b7dafa1-9877-4ca8-8baf-fecf9b06b7df: ref 0 aborting False >Thread-15910833::INFO::2013-11-19 21:21:03,516::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910835::DEBUG::2013-11-19 21:21:04,521::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910835::INFO::2013-11-19 21:21:04,522::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910835::DEBUG::2013-11-19 21:21:04,539::task::495::TaskManager.Task::(_debug) Task 932131cf-bf82-48ea-a368-89f89a89c97b: moving from state init -> state preparing >Thread-15910835::DEBUG::2013-11-19 21:21:04,539::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910835::DEBUG::2013-11-19 21:21:04,540::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910835::DEBUG::2013-11-19 21:21:04,540::task::495::TaskManager.Task::(_debug) Task 932131cf-bf82-48ea-a368-89f89a89c97b: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910835::DEBUG::2013-11-19 21:21:04,541::task::495::TaskManager.Task::(_debug) Task 932131cf-bf82-48ea-a368-89f89a89c97b: moving from state preparing -> state finished >Thread-15910835::DEBUG::2013-11-19 21:21:04,542::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910835::DEBUG::2013-11-19 21:21:04,542::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910835::DEBUG::2013-11-19 21:21:04,543::task::495::TaskManager.Task::(_debug) Task 932131cf-bf82-48ea-a368-89f89a89c97b: ref 0 aborting False >Thread-15910835::INFO::2013-11-19 21:21:04,543::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910836::DEBUG::2013-11-19 21:21:05,549::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910836::INFO::2013-11-19 21:21:05,550::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910836::DEBUG::2013-11-19 21:21:05,550::task::495::TaskManager.Task::(_debug) Task 70e52133-8750-46d4-9d53-3944f78bddc2: moving from state init -> state preparing >Thread-15910836::DEBUG::2013-11-19 21:21:05,551::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910836::DEBUG::2013-11-19 21:21:05,551::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910836::DEBUG::2013-11-19 21:21:05,552::task::495::TaskManager.Task::(_debug) Task 70e52133-8750-46d4-9d53-3944f78bddc2: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910836::DEBUG::2013-11-19 21:21:05,553::task::495::TaskManager.Task::(_debug) Task 70e52133-8750-46d4-9d53-3944f78bddc2: moving from state preparing -> state finished >Thread-15910836::DEBUG::2013-11-19 21:21:05,553::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910836::DEBUG::2013-11-19 21:21:05,554::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910836::DEBUG::2013-11-19 21:21:05,554::task::495::TaskManager.Task::(_debug) Task 70e52133-8750-46d4-9d53-3944f78bddc2: ref 0 aborting False >Thread-15910836::INFO::2013-11-19 21:21:05,555::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910838::DEBUG::2013-11-19 21:21:06,560::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910838::INFO::2013-11-19 21:21:06,561::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910838::DEBUG::2013-11-19 21:21:06,562::task::495::TaskManager.Task::(_debug) Task 81fcc1d9-f17b-46fe-b428-f989ad5ca468: moving from state init -> state preparing >Thread-15910838::DEBUG::2013-11-19 21:21:06,562::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910838::DEBUG::2013-11-19 21:21:06,563::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910838::DEBUG::2013-11-19 21:21:06,563::task::495::TaskManager.Task::(_debug) Task 81fcc1d9-f17b-46fe-b428-f989ad5ca468: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910838::DEBUG::2013-11-19 21:21:06,564::task::495::TaskManager.Task::(_debug) Task 81fcc1d9-f17b-46fe-b428-f989ad5ca468: moving from state preparing -> state finished >Thread-15910838::DEBUG::2013-11-19 21:21:06,565::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910838::DEBUG::2013-11-19 21:21:06,565::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910838::DEBUG::2013-11-19 21:21:06,566::task::495::TaskManager.Task::(_debug) Task 81fcc1d9-f17b-46fe-b428-f989ad5ca468: ref 0 aborting False >Thread-15910838::INFO::2013-11-19 21:21:06,566::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910839::DEBUG::2013-11-19 21:21:07,572::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910839::INFO::2013-11-19 21:21:07,573::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910839::DEBUG::2013-11-19 21:21:07,573::task::495::TaskManager.Task::(_debug) Task 65f08743-8242-4360-868f-2f520a1b6134: moving from state init -> state preparing >Thread-15910839::DEBUG::2013-11-19 21:21:07,574::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910839::DEBUG::2013-11-19 21:21:07,574::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910839::DEBUG::2013-11-19 21:21:07,575::task::495::TaskManager.Task::(_debug) Task 65f08743-8242-4360-868f-2f520a1b6134: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910839::DEBUG::2013-11-19 21:21:07,576::task::495::TaskManager.Task::(_debug) Task 65f08743-8242-4360-868f-2f520a1b6134: moving from state preparing -> state finished >Thread-15910839::DEBUG::2013-11-19 21:21:07,576::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910839::DEBUG::2013-11-19 21:21:07,577::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910839::DEBUG::2013-11-19 21:21:07,577::task::495::TaskManager.Task::(_debug) Task 65f08743-8242-4360-868f-2f520a1b6134: ref 0 aborting False >Thread-15910839::INFO::2013-11-19 21:21:07,578::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910841::DEBUG::2013-11-19 21:21:08,584::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910841::INFO::2013-11-19 21:21:08,584::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910841::DEBUG::2013-11-19 21:21:08,585::task::495::TaskManager.Task::(_debug) Task c13be546-789b-4d28-9200-404e56a11585: moving from state init -> state preparing >Thread-15910841::DEBUG::2013-11-19 21:21:08,585::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910841::DEBUG::2013-11-19 21:21:08,586::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910841::DEBUG::2013-11-19 21:21:08,587::task::495::TaskManager.Task::(_debug) Task c13be546-789b-4d28-9200-404e56a11585: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910841::DEBUG::2013-11-19 21:21:08,587::task::495::TaskManager.Task::(_debug) Task c13be546-789b-4d28-9200-404e56a11585: moving from state preparing -> state finished >Thread-15910841::DEBUG::2013-11-19 21:21:08,588::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910841::DEBUG::2013-11-19 21:21:08,588::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910841::DEBUG::2013-11-19 21:21:08,589::task::495::TaskManager.Task::(_debug) Task c13be546-789b-4d28-9200-404e56a11585: ref 0 aborting False >Thread-15910841::INFO::2013-11-19 21:21:08,589::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-1024767::ERROR::2013-11-19 21:21:09,432::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910842::DEBUG::2013-11-19 21:21:09,595::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910842::INFO::2013-11-19 21:21:09,605::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910842::DEBUG::2013-11-19 21:21:09,606::task::495::TaskManager.Task::(_debug) Task a6b4799a-fcb5-4269-b313-e6f25a54a456: moving from state init -> state preparing >Thread-15910842::DEBUG::2013-11-19 21:21:09,607::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910842::DEBUG::2013-11-19 21:21:09,607::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910842::DEBUG::2013-11-19 21:21:09,608::task::495::TaskManager.Task::(_debug) Task a6b4799a-fcb5-4269-b313-e6f25a54a456: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910842::DEBUG::2013-11-19 21:21:09,608::task::495::TaskManager.Task::(_debug) Task a6b4799a-fcb5-4269-b313-e6f25a54a456: moving from state preparing -> state finished >Thread-15910842::DEBUG::2013-11-19 21:21:09,609::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910842::DEBUG::2013-11-19 21:21:09,609::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910842::DEBUG::2013-11-19 21:21:09,610::task::495::TaskManager.Task::(_debug) Task a6b4799a-fcb5-4269-b313-e6f25a54a456: ref 0 aborting False >Thread-15910842::INFO::2013-11-19 21:21:09,610::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910844::DEBUG::2013-11-19 21:21:10,616::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910844::INFO::2013-11-19 21:21:10,617::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910844::DEBUG::2013-11-19 21:21:10,617::task::495::TaskManager.Task::(_debug) Task a3602667-dfdb-49c2-baff-085b1cda1eb3: moving from state init -> state preparing >Thread-15910844::DEBUG::2013-11-19 21:21:10,618::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910844::DEBUG::2013-11-19 21:21:10,618::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910844::DEBUG::2013-11-19 21:21:10,619::task::495::TaskManager.Task::(_debug) Task a3602667-dfdb-49c2-baff-085b1cda1eb3: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910844::DEBUG::2013-11-19 21:21:10,620::task::495::TaskManager.Task::(_debug) Task a3602667-dfdb-49c2-baff-085b1cda1eb3: moving from state preparing -> state finished >Thread-15910844::DEBUG::2013-11-19 21:21:10,620::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910844::DEBUG::2013-11-19 21:21:10,621::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910844::DEBUG::2013-11-19 21:21:10,621::task::495::TaskManager.Task::(_debug) Task a3602667-dfdb-49c2-baff-085b1cda1eb3: ref 0 aborting False >Thread-15910844::INFO::2013-11-19 21:21:10,622::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910845::DEBUG::2013-11-19 21:21:11,628::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910845::INFO::2013-11-19 21:21:11,629::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910845::DEBUG::2013-11-19 21:21:11,629::task::495::TaskManager.Task::(_debug) Task 71fef9c5-5b25-4ec5-ac5d-f1264df35c06: moving from state init -> state preparing >Thread-15910845::DEBUG::2013-11-19 21:21:11,630::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910845::DEBUG::2013-11-19 21:21:11,630::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910845::DEBUG::2013-11-19 21:21:11,631::task::495::TaskManager.Task::(_debug) Task 71fef9c5-5b25-4ec5-ac5d-f1264df35c06: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910845::DEBUG::2013-11-19 21:21:11,631::task::495::TaskManager.Task::(_debug) Task 71fef9c5-5b25-4ec5-ac5d-f1264df35c06: moving from state preparing -> state finished >Thread-15910845::DEBUG::2013-11-19 21:21:11,632::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910845::DEBUG::2013-11-19 21:21:11,632::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910845::DEBUG::2013-11-19 21:21:11,633::task::495::TaskManager.Task::(_debug) Task 71fef9c5-5b25-4ec5-ac5d-f1264df35c06: ref 0 aborting False >Thread-15910845::INFO::2013-11-19 21:21:11,633::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910846::INFO::2013-11-19 21:21:11,924::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910846::DEBUG::2013-11-19 21:21:11,924::task::495::TaskManager.Task::(_debug) Task 127b21cd-b650-4a72-8ea2-f5552996048f: moving from state init -> state preparing >Thread-15910846::DEBUG::2013-11-19 21:21:11,925::task::495::TaskManager.Task::(_debug) Task 127b21cd-b650-4a72-8ea2-f5552996048f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000339984893799', 'lastCheck': 1384896063.6513791, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150203704834', 'lastCheck': 1384896064.4249959, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896069.4335749, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000384092330933', 'lastCheck': 1384896069.075789, 'valid': True, 'code': 0}} >Thread-15910846::DEBUG::2013-11-19 21:21:11,926::task::495::TaskManager.Task::(_debug) Task 127b21cd-b650-4a72-8ea2-f5552996048f: moving from state preparing -> state finished >Thread-15910846::DEBUG::2013-11-19 21:21:11,926::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910846::DEBUG::2013-11-19 21:21:11,927::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910846::DEBUG::2013-11-19 21:21:11,927::task::495::TaskManager.Task::(_debug) Task 127b21cd-b650-4a72-8ea2-f5552996048f: ref 0 aborting False >Thread-15910846::INFO::2013-11-19 21:21:11,928::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000339984893799', 'lastCheck': 1384896063.6513791, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150203704834', 'lastCheck': 1384896064.4249959, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896069.4335749, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000384092330933', 'lastCheck': 1384896069.075789, 'valid': True, 'code': 0}} >Thread-15910848::DEBUG::2013-11-19 21:21:12,639::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910848::INFO::2013-11-19 21:21:12,640::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910848::DEBUG::2013-11-19 21:21:12,641::task::495::TaskManager.Task::(_debug) Task e6f0a08d-81b7-4ae5-95d3-cc992b50e941: moving from state init -> state preparing >Thread-15910848::DEBUG::2013-11-19 21:21:12,641::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910848::DEBUG::2013-11-19 21:21:12,642::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910848::DEBUG::2013-11-19 21:21:12,642::task::495::TaskManager.Task::(_debug) Task e6f0a08d-81b7-4ae5-95d3-cc992b50e941: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910848::DEBUG::2013-11-19 21:21:12,643::task::495::TaskManager.Task::(_debug) Task e6f0a08d-81b7-4ae5-95d3-cc992b50e941: moving from state preparing -> state finished >Thread-15910848::DEBUG::2013-11-19 21:21:12,643::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910848::DEBUG::2013-11-19 21:21:12,644::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910848::DEBUG::2013-11-19 21:21:12,644::task::495::TaskManager.Task::(_debug) Task e6f0a08d-81b7-4ae5-95d3-cc992b50e941: ref 0 aborting False >Thread-15910848::INFO::2013-11-19 21:21:12,645::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910849::DEBUG::2013-11-19 21:21:13,651::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910849::INFO::2013-11-19 21:21:13,651::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910849::DEBUG::2013-11-19 21:21:13,652::task::495::TaskManager.Task::(_debug) Task ed8e0f56-ff8a-46fc-bdbf-82605d433e45: moving from state init -> state preparing >Thread-15910849::DEBUG::2013-11-19 21:21:13,653::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910849::DEBUG::2013-11-19 21:21:13,653::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910849::DEBUG::2013-11-19 21:21:13,654::task::495::TaskManager.Task::(_debug) Task ed8e0f56-ff8a-46fc-bdbf-82605d433e45: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910849::DEBUG::2013-11-19 21:21:13,654::task::495::TaskManager.Task::(_debug) Task ed8e0f56-ff8a-46fc-bdbf-82605d433e45: moving from state preparing -> state finished >Thread-15910849::DEBUG::2013-11-19 21:21:13,655::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910849::DEBUG::2013-11-19 21:21:13,655::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910849::DEBUG::2013-11-19 21:21:13,656::task::495::TaskManager.Task::(_debug) Task ed8e0f56-ff8a-46fc-bdbf-82605d433e45: ref 0 aborting False >Thread-15910849::INFO::2013-11-19 21:21:13,656::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910851::DEBUG::2013-11-19 21:21:14,662::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910851::INFO::2013-11-19 21:21:14,663::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910851::DEBUG::2013-11-19 21:21:14,671::task::495::TaskManager.Task::(_debug) Task 8ddee017-4e5f-4b13-8426-6455023a6a52: moving from state init -> state preparing >Thread-15910851::DEBUG::2013-11-19 21:21:14,671::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910851::DEBUG::2013-11-19 21:21:14,672::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910851::DEBUG::2013-11-19 21:21:14,672::task::495::TaskManager.Task::(_debug) Task 8ddee017-4e5f-4b13-8426-6455023a6a52: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910851::DEBUG::2013-11-19 21:21:14,673::task::495::TaskManager.Task::(_debug) Task 8ddee017-4e5f-4b13-8426-6455023a6a52: moving from state preparing -> state finished >Thread-15910851::DEBUG::2013-11-19 21:21:14,673::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910851::DEBUG::2013-11-19 21:21:14,674::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910851::DEBUG::2013-11-19 21:21:14,674::task::495::TaskManager.Task::(_debug) Task 8ddee017-4e5f-4b13-8426-6455023a6a52: ref 0 aborting False >Thread-15910851::INFO::2013-11-19 21:21:14,675::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910852::DEBUG::2013-11-19 21:21:15,681::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910852::INFO::2013-11-19 21:21:15,682::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910852::DEBUG::2013-11-19 21:21:15,682::task::495::TaskManager.Task::(_debug) Task 75905a5c-ec67-459a-bc6a-e7693355faa5: moving from state init -> state preparing >Thread-15910852::DEBUG::2013-11-19 21:21:15,683::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910852::DEBUG::2013-11-19 21:21:15,683::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910852::DEBUG::2013-11-19 21:21:15,684::task::495::TaskManager.Task::(_debug) Task 75905a5c-ec67-459a-bc6a-e7693355faa5: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910852::DEBUG::2013-11-19 21:21:15,684::task::495::TaskManager.Task::(_debug) Task 75905a5c-ec67-459a-bc6a-e7693355faa5: moving from state preparing -> state finished >Thread-15910852::DEBUG::2013-11-19 21:21:15,685::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910852::DEBUG::2013-11-19 21:21:15,685::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910852::DEBUG::2013-11-19 21:21:15,686::task::495::TaskManager.Task::(_debug) Task 75905a5c-ec67-459a-bc6a-e7693355faa5: ref 0 aborting False >Thread-15910852::INFO::2013-11-19 21:21:15,687::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910854::DEBUG::2013-11-19 21:21:16,692::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910854::INFO::2013-11-19 21:21:16,693::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910854::DEBUG::2013-11-19 21:21:16,694::task::495::TaskManager.Task::(_debug) Task 9bf03bef-4319-4f3d-9f05-8b6accf80b0b: moving from state init -> state preparing >Thread-15910854::DEBUG::2013-11-19 21:21:16,694::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910854::DEBUG::2013-11-19 21:21:16,695::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910854::DEBUG::2013-11-19 21:21:16,695::task::495::TaskManager.Task::(_debug) Task 9bf03bef-4319-4f3d-9f05-8b6accf80b0b: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910854::DEBUG::2013-11-19 21:21:16,696::task::495::TaskManager.Task::(_debug) Task 9bf03bef-4319-4f3d-9f05-8b6accf80b0b: moving from state preparing -> state finished >Thread-15910854::DEBUG::2013-11-19 21:21:16,696::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910854::DEBUG::2013-11-19 21:21:16,697::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910854::DEBUG::2013-11-19 21:21:16,697::task::495::TaskManager.Task::(_debug) Task 9bf03bef-4319-4f3d-9f05-8b6accf80b0b: ref 0 aborting False >Thread-15910854::INFO::2013-11-19 21:21:16,698::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910855::DEBUG::2013-11-19 21:21:17,704::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910855::INFO::2013-11-19 21:21:17,705::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910855::DEBUG::2013-11-19 21:21:17,707::task::495::TaskManager.Task::(_debug) Task 9350c3ed-1fb6-43e4-ae60-f6899f296482: moving from state init -> state preparing >Thread-15910855::DEBUG::2013-11-19 21:21:17,708::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910855::DEBUG::2013-11-19 21:21:17,708::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910855::DEBUG::2013-11-19 21:21:17,709::task::495::TaskManager.Task::(_debug) Task 9350c3ed-1fb6-43e4-ae60-f6899f296482: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910855::DEBUG::2013-11-19 21:21:17,710::task::495::TaskManager.Task::(_debug) Task 9350c3ed-1fb6-43e4-ae60-f6899f296482: moving from state preparing -> state finished >Thread-15910855::DEBUG::2013-11-19 21:21:17,712::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910855::DEBUG::2013-11-19 21:21:17,712::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910855::DEBUG::2013-11-19 21:21:17,713::task::495::TaskManager.Task::(_debug) Task 9350c3ed-1fb6-43e4-ae60-f6899f296482: ref 0 aborting False >Thread-15910855::INFO::2013-11-19 21:21:17,714::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910857::DEBUG::2013-11-19 21:21:18,719::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910857::INFO::2013-11-19 21:21:18,720::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910857::DEBUG::2013-11-19 21:21:18,721::task::495::TaskManager.Task::(_debug) Task eba498f3-84a0-46f7-87a1-201ce1e5cf24: moving from state init -> state preparing >Thread-15910857::DEBUG::2013-11-19 21:21:18,721::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910857::DEBUG::2013-11-19 21:21:18,722::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910857::DEBUG::2013-11-19 21:21:18,722::task::495::TaskManager.Task::(_debug) Task eba498f3-84a0-46f7-87a1-201ce1e5cf24: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910857::DEBUG::2013-11-19 21:21:18,723::task::495::TaskManager.Task::(_debug) Task eba498f3-84a0-46f7-87a1-201ce1e5cf24: moving from state preparing -> state finished >Thread-15910857::DEBUG::2013-11-19 21:21:18,724::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910857::DEBUG::2013-11-19 21:21:18,724::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910857::DEBUG::2013-11-19 21:21:18,725::task::495::TaskManager.Task::(_debug) Task eba498f3-84a0-46f7-87a1-201ce1e5cf24: ref 0 aborting False >Thread-15910857::INFO::2013-11-19 21:21:18,725::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:19,281::safelease::61::Storage.Misc.excCmd::(acquire) SUCCESS: <err> = ''; <rc> = 0 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:19,281::safelease::65::ClusterLock::(acquire) Clustered lock acquired successfully >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:19,282::spm::468::Storage.SPM::(start) spm lock acquired successfully >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:19,283::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:19,283::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:19,284::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:19,285::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:19,285::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:19,286::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:21:19,506::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910858::DEBUG::2013-11-19 21:21:19,732::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910858::INFO::2013-11-19 21:21:19,733::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910858::DEBUG::2013-11-19 21:21:19,734::task::495::TaskManager.Task::(_debug) Task c7f3110f-d7e4-4fa3-9b8e-36462ddde7b2: moving from state init -> state preparing >Thread-15910858::DEBUG::2013-11-19 21:21:19,734::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910858::DEBUG::2013-11-19 21:21:19,735::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910858::DEBUG::2013-11-19 21:21:19,736::task::495::TaskManager.Task::(_debug) Task c7f3110f-d7e4-4fa3-9b8e-36462ddde7b2: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910858::DEBUG::2013-11-19 21:21:19,736::task::495::TaskManager.Task::(_debug) Task c7f3110f-d7e4-4fa3-9b8e-36462ddde7b2: moving from state preparing -> state finished >Thread-15910858::DEBUG::2013-11-19 21:21:19,737::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910858::DEBUG::2013-11-19 21:21:19,737::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910858::DEBUG::2013-11-19 21:21:19,738::task::495::TaskManager.Task::(_debug) Task c7f3110f-d7e4-4fa3-9b8e-36462ddde7b2: ref 0 aborting False >Thread-15910858::INFO::2013-11-19 21:21:19,738::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:19,924::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:19,927::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:19,928::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=279', u'_SHA_CKSUM=b5dee775aac07995010b8c6a5db186617df84cf4', u'POOL_SPM_ID=-1'] >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:19,929::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:19,930::persistentDict::146::Storage.PersistentDict::(transaction) Flushing changes >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:19,930::persistentDict::265::Storage.PersistentDict::(flush) about to write lines (VGTagMetadataRW)=[u'CLASS=Data', u'DESCRIPTION=OS', u'IOOPTIMEOUTSEC=10', u'LEASERETRIES=3', u'LEASETIMESEC=60', u'LOCKPOLICY=', u'LOCKRENEWALINTERVALSEC=5', u'LOGBLKSIZE=512', u'MASTER_VERSION=1', u'PHYBLKSIZE=512', u'POOL_DESCRIPTION=Default', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_ID=2', u'POOL_SPM_LVER=280', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'ROLE=Master', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'TYPE=FCP', u'VERSION=2', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', '_SHA_CKSUM=215540c631d1cfd943bc1c89ef09ec9e3251e9bd'] >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:19,931::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:19,932::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:19,932::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:19,933::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:19,934::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:19,935::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:20,584::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:20,587::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:20,588::blockSD::145::storage.Metadata.VGTagMetadataRW::(writelines) Updating metadata adding=MDT__SHA_CKSUM=215540c631d1cfd943bc1c89ef09ec9e3251e9bd, MDT_POOL_SPM_LVER=280, MDT_POOL_SPM_ID=2 removing=MDT_POOL_SPM_ID=-1, MDT__SHA_CKSUM=b5dee775aac07995010b8c6a5db186617df84cf4, MDT_POOL_SPM_LVER=279 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:20,590::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --deltag MDT__SHA_CKSUM=b5dee775aac07995010b8c6a5db186617df84cf4 --deltag MDT_POOL_SPM_LVER=279 --deltag MDT_POOL_SPM_ID=-1 --addtag MDT_POOL_SPM_LVER=280 --addtag MDT__SHA_CKSUM=215540c631d1cfd943bc1c89ef09ec9e3251e9bd --addtag MDT_POOL_SPM_ID=2 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15910860::DEBUG::2013-11-19 21:21:20,812::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910860::INFO::2013-11-19 21:21:20,814::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910860::DEBUG::2013-11-19 21:21:20,815::task::495::TaskManager.Task::(_debug) Task d16c47e8-705d-4004-a4d5-061fb071ccc7: moving from state init -> state preparing >Thread-15910860::DEBUG::2013-11-19 21:21:20,816::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910860::DEBUG::2013-11-19 21:21:20,816::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910860::DEBUG::2013-11-19 21:21:20,817::task::495::TaskManager.Task::(_debug) Task d16c47e8-705d-4004-a4d5-061fb071ccc7: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910860::DEBUG::2013-11-19 21:21:20,818::task::495::TaskManager.Task::(_debug) Task d16c47e8-705d-4004-a4d5-061fb071ccc7: moving from state preparing -> state finished >Thread-15910860::DEBUG::2013-11-19 21:21:20,818::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910860::DEBUG::2013-11-19 21:21:20,819::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910860::DEBUG::2013-11-19 21:21:20,819::task::495::TaskManager.Task::(_debug) Task d16c47e8-705d-4004-a4d5-061fb071ccc7: ref 0 aborting False >Thread-15910860::INFO::2013-11-19 21:21:20,820::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,220::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,221::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,222::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,223::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,224::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f39bd185-4535-444c-bdd3-e7744f66c881`::Request was made in '/usr/share/vdsm/storage/spm.py' line '330' at '_upgradePool' >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,225::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,225::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,226::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f39bd185-4535-444c-bdd3-e7744f66c881`::Granted request >934c4d35-aab6-4fba-9996-9cc07dd7d991::INFO::2013-11-19 21:21:21,227::spm::336::Storage.SPM::(_upgradePool) Trying to upgrade master domain `6b892067-ac7e-4039-a7be-5b7387525b16` >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,228::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`38d6bcc4-c986-47d8-a5e3-25f05bb46151`::Request was made in '/usr/share/vdsm/storage/spm.py' line '337' at '_upgradePool' >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,228::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'exclusive' >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,229::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'exclusive' (1 active user) >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,229::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`38d6bcc4-c986-47d8-a5e3-25f05bb46151`::Granted request >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,230::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,231::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,232::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,232::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,233::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,234::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,813::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,817::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,818::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=280', u'_SHA_CKSUM=215540c631d1cfd943bc1c89ef09ec9e3251e9bd', u'POOL_SPM_ID=2'] >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,819::blockSD::625::Storage.StorageDomain::(upgrade) Trying to upgrade domain `6b892067-ac7e-4039-a7be-5b7387525b16` from version 2 to version 2 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,819::blockSD::631::Storage.StorageDomain::(upgrade) No need to upgrade domain `6b892067-ac7e-4039-a7be-5b7387525b16`, leaving unchanged >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,820::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,821::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,821::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,822::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,822::spm::340::Storage.SPM::(_upgradePool) Marking all domains for upgrade >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,823::spm::347::Storage.SPM::(_upgradePool) Registering with state change event >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,824::spm::349::Storage.SPM::(_upgradePool) Running initial domain upgrade threads >Thread-15910861::DEBUG::2013-11-19 21:21:21,825::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 45cdff8b-89c0-4926-a8c7-80354064bfa6 >Thread-15910863::DEBUG::2013-11-19 21:21:21,827::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 772472cb-2429-466d-96cd-00b3301ef78a >Thread-15910862::DEBUG::2013-11-19 21:21:21,830::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910864::DEBUG::2013-11-19 21:21:21,831::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 5d7b5178-3790-472c-a13b-b481eb27a61e >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,832::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15910864::ERROR::2013-11-19 21:21:21,841::spm::373::Storage.SPM::(_upgradePoolDomain) Error while preparing domain `5d7b5178-3790-472c-a13b-b481eb27a61e` upgrade >Traceback (most recent call last): > File "/usr/share/vdsm/storage/spm.py", line 369, in _upgradePoolDomain >IndexError: list index out of range >Thread-15910863::DEBUG::2013-11-19 21:21:21,843::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`2d7a3930-4fae-4578-beba-389d757607b1`::Request was made in '/usr/share/vdsm/storage/spm.py' line '376' at '_upgradePoolDomain' >Thread-15910862::INFO::2013-11-19 21:21:21,843::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910863::DEBUG::2013-11-19 21:21:21,851::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'exclusive' >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,852::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15910862::DEBUG::2013-11-19 21:21:21,852::task::495::TaskManager.Task::(_debug) Task c41982af-17ac-434d-a324-37cd95930140: moving from state init -> state preparing >Thread-15910861::DEBUG::2013-11-19 21:21:21,854::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`0ac4c005-ee7d-4a90-b20c-f95d924b4099`::Request was made in '/usr/share/vdsm/storage/spm.py' line '376' at '_upgradePoolDomain' >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,854::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15910862::DEBUG::2013-11-19 21:21:21,855::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910861::DEBUG::2013-11-19 21:21:21,855::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' for lock type 'exclusive' >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,856::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15910862::DEBUG::2013-11-19 21:21:21,856::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910863::DEBUG::2013-11-19 21:21:21,857::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'exclusive' (1 active user) >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:21,858::blockSD::791::Storage.Misc.excCmd::(mountMaster) '/usr/bin/sudo -n /sbin/e2fsck -p /dev/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15910862::DEBUG::2013-11-19 21:21:21,859::task::495::TaskManager.Task::(_debug) Task c41982af-17ac-434d-a324-37cd95930140: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910863::DEBUG::2013-11-19 21:21:21,859::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`2d7a3930-4fae-4578-beba-389d757607b1`::Granted request >Thread-15910862::DEBUG::2013-11-19 21:21:21,861::task::495::TaskManager.Task::(_debug) Task c41982af-17ac-434d-a324-37cd95930140: moving from state preparing -> state finished >Thread-15910861::DEBUG::2013-11-19 21:21:21,862::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' is free. Now locking as 'exclusive' (1 active user) >Thread-15910862::DEBUG::2013-11-19 21:21:21,863::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910863::DEBUG::2013-11-19 21:21:22,065::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`62e8e300-bc1a-48cf-8a2c-10534928ef5e`::Request was made in '/usr/share/vdsm/storage/spm.py' line '377' at '_upgradePoolDomain' >Thread-15910861::DEBUG::2013-11-19 21:21:22,067::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`0ac4c005-ee7d-4a90-b20c-f95d924b4099`::Granted request >Thread-15910862::DEBUG::2013-11-19 21:21:22,068::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910863::DEBUG::2013-11-19 21:21:22,069::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'exclusive' >Thread-15910862::DEBUG::2013-11-19 21:21:22,072::task::495::TaskManager.Task::(_debug) Task c41982af-17ac-434d-a324-37cd95930140: ref 0 aborting False >Thread-15910862::INFO::2013-11-19 21:21:22,074::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910861::DEBUG::2013-11-19 21:21:22,077::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`db93c7a6-5b28-4ce1-b14c-b9177c970d7a`::Request was made in '/usr/share/vdsm/storage/spm.py' line '377' at '_upgradePoolDomain' >Thread-15910863::DEBUG::2013-11-19 21:21:22,078::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'exclusive' (1 active user) >Thread-15910861::DEBUG::2013-11-19 21:21:22,079::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' for lock type 'exclusive' >Thread-15910863::DEBUG::2013-11-19 21:21:22,081::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`62e8e300-bc1a-48cf-8a2c-10534928ef5e`::Granted request >Thread-15910861::DEBUG::2013-11-19 21:21:22,083::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' is free. Now locking as 'exclusive' (1 active user) >Thread-15910863::DEBUG::2013-11-19 21:21:22,084::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15910861::DEBUG::2013-11-19 21:21:22,085::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`db93c7a6-5b28-4ce1-b14c-b9177c970d7a`::Granted request >Thread-15910863::DEBUG::2013-11-19 21:21:22,086::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15910861::DEBUG::2013-11-19 21:21:22,087::spm::385::Storage.SPM::(_upgradePoolDomain) Domain `45cdff8b-89c0-4926-a8c7-80354064bfa6` is not a data domain it is an 2 domain, not upgrading >Thread-15910863::DEBUG::2013-11-19 21:21:22,088::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15910861::DEBUG::2013-11-19 21:21:22,089::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' >Thread-15910863::DEBUG::2013-11-19 21:21:22,089::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15910861::DEBUG::2013-11-19 21:21:22,090::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' (0 active users) >Thread-15910863::DEBUG::2013-11-19 21:21:22,091::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15910861::DEBUG::2013-11-19 21:21:22,092::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' is free, finding out if anyone is waiting for it. >Thread-15910863::DEBUG::2013-11-19 21:21:22,093::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15910861::DEBUG::2013-11-19 21:21:22,094::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6', Clearing records. >Thread-15910861::DEBUG::2013-11-19 21:21:22,095::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:22,355::blockSD::791::Storage.Misc.excCmd::(mountMaster) SUCCESS: <err> = ''; <rc> = 0 >Thread-15910861::DEBUG::2013-11-19 21:21:22,357::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' (0 active users) >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:22,358::blockSD::816::Storage.Misc.excCmd::(mountMaster) '/usr/bin/sudo -n /sbin/tune2fs -j /dev/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15910865::INFO::2013-11-19 21:21:22,363::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910861::DEBUG::2013-11-19 21:21:22,364::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' is free, finding out if anyone is waiting for it. >Thread-15910865::DEBUG::2013-11-19 21:21:22,364::task::495::TaskManager.Task::(_debug) Task cba94721-251d-4191-9da3-55bd755ba0bf: moving from state init -> state preparing >Thread-15910861::DEBUG::2013-11-19 21:21:22,714::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6', Clearing records. >Thread-15910865::DEBUG::2013-11-19 21:21:22,716::task::495::TaskManager.Task::(_debug) Task cba94721-251d-4191-9da3-55bd755ba0bf: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000332117080688', 'lastCheck': 1384896073.6639979, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00149703025818', 'lastCheck': 1384896074.440325, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896079.507699, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000433921813965', 'lastCheck': 1384896079.087245, 'valid': True, 'code': 0}} >Thread-15910865::DEBUG::2013-11-19 21:21:22,718::task::495::TaskManager.Task::(_debug) Task cba94721-251d-4191-9da3-55bd755ba0bf: moving from state preparing -> state finished >Thread-15910865::DEBUG::2013-11-19 21:21:22,720::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910865::DEBUG::2013-11-19 21:21:22,721::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910865::DEBUG::2013-11-19 21:21:22,722::task::495::TaskManager.Task::(_debug) Task cba94721-251d-4191-9da3-55bd755ba0bf: ref 0 aborting False >Thread-15910865::INFO::2013-11-19 21:21:22,723::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000332117080688', 'lastCheck': 1384896073.6639979, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00149703025818', 'lastCheck': 1384896074.440325, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896079.507699, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000433921813965', 'lastCheck': 1384896079.087245, 'valid': True, 'code': 0}} >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:22,912::blockSD::816::Storage.Misc.excCmd::(mountMaster) FAILED: <err> = 'The filesystem already has a journal.\n'; <rc> = 1 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:22,914::fileUtils::83::Storage.Misc.excCmd::(mount) '/usr/bin/sudo -n /bin/mount /dev/6b892067-ac7e-4039-a7be-5b7387525b16/master /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15910863::DEBUG::2013-11-19 21:21:23,134::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15910867::DEBUG::2013-11-19 21:21:23,137::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910863::DEBUG::2013-11-19 21:21:23,142::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15910867::INFO::2013-11-19 21:21:23,143::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910863::DEBUG::2013-11-19 21:21:23,144::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'SDUUID=772472cb-2429-466d-96cd-00b3301ef78a', u'VERSION=2', u'LOCKPOLICY=', u'VGUUID=XTb294-NYv6-dVpO-bB1n-4f9P-d0GL-dhd2DI', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'LEASETIMESEC=5', u'LOCKRENEWALINTERVALSEC=5', u'ROLE=Regular', u'PHYBLKSIZE=512', u'DESCRIPTION=Database', u'IOOPTIMEOUTSEC=1', u'CLASS=Data', u'TYPE=FCP', u'PV0=pv:3600c0ff0001520c7414f605001000000,uuid:NujM33-vNiD-sBHY-mrPo-EQ2f-1qsz-HXfn0q,pestart:0,pecount:8929,mapoffset:0', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'MASTER_VERSION=0', u'_SHA_CKSUM=ef8840d9fdef24516a56fd970bed1960d2b1ae92'] >Thread-15910867::DEBUG::2013-11-19 21:21:23,145::task::495::TaskManager.Task::(_debug) Task 45951473-db59-4fb6-bb10-bdb5f17e0a2a: moving from state init -> state preparing >Thread-15910863::DEBUG::2013-11-19 21:21:23,147::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15910867::DEBUG::2013-11-19 21:21:23,148::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910867::DEBUG::2013-11-19 21:21:23,149::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910863::DEBUG::2013-11-19 21:21:23,150::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15910867::DEBUG::2013-11-19 21:21:23,150::task::495::TaskManager.Task::(_debug) Task 45951473-db59-4fb6-bb10-bdb5f17e0a2a: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910863::DEBUG::2013-11-19 21:21:23,151::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15910867::DEBUG::2013-11-19 21:21:23,152::task::495::TaskManager.Task::(_debug) Task 45951473-db59-4fb6-bb10-bdb5f17e0a2a: moving from state preparing -> state finished >Thread-15910863::DEBUG::2013-11-19 21:21:23,152::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15910867::DEBUG::2013-11-19 21:21:23,153::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910863::DEBUG::2013-11-19 21:21:23,154::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15910867::DEBUG::2013-11-19 21:21:23,155::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910863::DEBUG::2013-11-19 21:21:23,156::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15910867::DEBUG::2013-11-19 21:21:23,157::task::495::TaskManager.Task::(_debug) Task 45951473-db59-4fb6-bb10-bdb5f17e0a2a: ref 0 aborting False >Thread-15910863::DEBUG::2013-11-19 21:21:23,157::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15910867::INFO::2013-11-19 21:21:23,158::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910863::DEBUG::2013-11-19 21:21:23,158::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,343::fileUtils::83::Storage.Misc.excCmd::(mount) SUCCESS: <err> = ''; <rc> = 0 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,344::blockSD::833::Storage.Misc.excCmd::(mountMaster) '/usr/bin/sudo -n /bin/chown vdsm:kvm /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,746::blockSD::833::Storage.Misc.excCmd::(mountMaster) SUCCESS: <err> = ''; <rc> = 0 >934c4d35-aab6-4fba-9996-9cc07dd7d991::WARNING::2013-11-19 21:21:23,749::spm::490::Storage.SPM::(start) Backup domain validation failed >Traceback (most recent call last): > File "/usr/share/vdsm/storage/spm.py", line 488, in start > File "/usr/share/vdsm/storage/sp.py", line 1208, in checkBackupDomain > File "/usr/share/vdsm/storage/sd.py", line 666, in isBackup > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,750::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,751::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,752::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,752::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,753::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,754::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,754::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,755::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,755::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,756::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,756::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,757::persistentDict::146::Storage.PersistentDict::(transaction) Flushing changes >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,757::persistentDict::265::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['MASTER_VERSION=0', 'ROLE=Regular', '_SHA_CKSUM=1deb73127f89496e2cb17e51bbfae1df6c1879d1'] >934c4d35-aab6-4fba-9996-9cc07dd7d991::WARNING::2013-11-19 21:21:23,758::persistentDict::150::Storage.PersistentDict::(transaction) Error in transaction, rolling back changes >Traceback (most recent call last): > File "/usr/share/vdsm/storage/persistentDict.py", line 147, in transaction > File "/usr/share/vdsm/storage/persistentDict.py", line 266, in flush > File "/usr/share/vdsm/storage/fileSD.py", line 58, in writelines > File "/usr/share/vdsm/storage/processPool.py", line 33, in wrapper > File "/usr/share/vdsm/storage/processPool.py", line 44, in runExternally > File "/usr/share/vdsm/storage/processPool.py", line 127, in runExternally >IOError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata.new' >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,759::persistentDict::265::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['_SHA_CKSUM=da39a3ee5e6b4b0d3255bfef95601890afd80709'] >934c4d35-aab6-4fba-9996-9cc07dd7d991::ERROR::2013-11-19 21:21:23,760::spm::526::Storage.SPM::(start) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/spm.py", line 493, in start > File "/usr/share/vdsm/storage/sp.py", line 637, in _regulariseOutdatedMasters > File "/usr/share/vdsm/storage/sp.py", line 631, in _setRegularSD > File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ > File "/usr/share/vdsm/storage/persistentDict.py", line 152, in transaction > File "/usr/share/vdsm/storage/persistentDict.py", line 266, in flush > File "/usr/share/vdsm/storage/fileSD.py", line 58, in writelines > File "/usr/share/vdsm/storage/processPool.py", line 33, in wrapper > File "/usr/share/vdsm/storage/processPool.py", line 44, in runExternally > File "/usr/share/vdsm/storage/processPool.py", line 127, in runExternally >IOError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata.new' >934c4d35-aab6-4fba-9996-9cc07dd7d991::ERROR::2013-11-19 21:21:23,761::spm::527::Storage.SPM::(start) failed: [Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata.new' >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,762::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`84330296-faf6-4d34-af3e-638a082fbdb4`::Request was made in '/usr/share/vdsm/storage/spm.py' line '584' at '_stop' >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,763::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,763::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,764::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`84330296-faf6-4d34-af3e-638a082fbdb4`::Granted request >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,765::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`b2fb8e7d-7020-469a-9cc1-2d350516713b`::Request was made in '/usr/share/vdsm/storage/spm.py' line '601' at '_stop' >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,766::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' for lock type 'exclusive' >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,766::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' is free. Now locking as 'exclusive' (1 active user) >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,767::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`b2fb8e7d-7020-469a-9cc1-2d350516713b`::Granted request >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,768::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,769::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' (0 active users) >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,769::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' is free, finding out if anyone is waiting for it. >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,770::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e', Clearing records. >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,770::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,771::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,771::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,772::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,773::taskManager::118::TaskManager::(unloadTasks) Entry. >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,773::taskManager::122::TaskManager::(unloadTasks) Return >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,775::spm::195::Storage.SPM::(__cleanupMasterMount) unmounting /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:23,907::fileUtils::112::Storage.Misc.excCmd::(umount) '/usr/bin/sudo -n /bin/umount -f /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15910868::DEBUG::2013-11-19 21:21:24,168::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910868::INFO::2013-11-19 21:21:24,170::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910868::DEBUG::2013-11-19 21:21:24,171::task::495::TaskManager.Task::(_debug) Task 10e2f76d-7dd0-468e-8bb5-3a615e3a7915: moving from state init -> state preparing >Thread-15910868::DEBUG::2013-11-19 21:21:24,171::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910868::DEBUG::2013-11-19 21:21:24,172::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910868::DEBUG::2013-11-19 21:21:24,173::task::495::TaskManager.Task::(_debug) Task 10e2f76d-7dd0-468e-8bb5-3a615e3a7915: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910868::DEBUG::2013-11-19 21:21:24,173::task::495::TaskManager.Task::(_debug) Task 10e2f76d-7dd0-468e-8bb5-3a615e3a7915: moving from state preparing -> state finished >Thread-15910868::DEBUG::2013-11-19 21:21:24,174::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910868::DEBUG::2013-11-19 21:21:24,174::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910868::DEBUG::2013-11-19 21:21:24,175::task::495::TaskManager.Task::(_debug) Task 10e2f76d-7dd0-468e-8bb5-3a615e3a7915: ref 0 aborting False >Thread-15910868::INFO::2013-11-19 21:21:24,176::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:24,301::fileUtils::112::Storage.Misc.excCmd::(umount) SUCCESS: <err> = ''; <rc> = 0 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:24,304::spm::290::Storage.SPM::(__cleanupSPM) cleaning up SPM: c4e5889a-348d-11e2-83fe-9f72369eab30 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:24,305::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:24,305::persistentDict::146::Storage.PersistentDict::(transaction) Flushing changes >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:24,306::persistentDict::265::Storage.PersistentDict::(flush) about to write lines (VGTagMetadataRW)=[u'CLASS=Data', u'DESCRIPTION=OS', u'IOOPTIMEOUTSEC=10', u'LEASERETRIES=3', u'LEASETIMESEC=60', u'LOCKPOLICY=', u'LOCKRENEWALINTERVALSEC=5', u'LOGBLKSIZE=512', u'MASTER_VERSION=1', u'PHYBLKSIZE=512', u'POOL_DESCRIPTION=Default', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_ID=-1', u'POOL_SPM_LVER=280', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'ROLE=Master', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'TYPE=FCP', u'VERSION=2', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', '_SHA_CKSUM=1cdcb520a5614f101fe172bdfa0b0c9276978abe'] >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:24,307::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:24,308::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:24,308::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:24,309::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:24,310::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:24,311::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:24,918::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:24,921::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:24,922::blockSD::145::storage.Metadata.VGTagMetadataRW::(writelines) Updating metadata adding=MDT_POOL_SPM_ID=-1, MDT__SHA_CKSUM=1cdcb520a5614f101fe172bdfa0b0c9276978abe removing=MDT__SHA_CKSUM=215540c631d1cfd943bc1c89ef09ec9e3251e9bd, MDT_POOL_SPM_ID=2 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:24,923::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --deltag MDT_POOL_SPM_ID=2 --deltag MDT__SHA_CKSUM=215540c631d1cfd943bc1c89ef09ec9e3251e9bd --addtag MDT__SHA_CKSUM=1cdcb520a5614f101fe172bdfa0b0c9276978abe --addtag MDT_POOL_SPM_ID=-1 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15910870::DEBUG::2013-11-19 21:21:25,185::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910870::INFO::2013-11-19 21:21:25,188::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910870::DEBUG::2013-11-19 21:21:25,188::task::495::TaskManager.Task::(_debug) Task 15f3ec47-5ff8-41d8-8e0c-bbd7cfdf0c65: moving from state init -> state preparing >Thread-15910870::DEBUG::2013-11-19 21:21:25,189::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910870::DEBUG::2013-11-19 21:21:25,190::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910870::DEBUG::2013-11-19 21:21:25,190::task::495::TaskManager.Task::(_debug) Task 15f3ec47-5ff8-41d8-8e0c-bbd7cfdf0c65: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910870::DEBUG::2013-11-19 21:21:25,191::task::495::TaskManager.Task::(_debug) Task 15f3ec47-5ff8-41d8-8e0c-bbd7cfdf0c65: moving from state preparing -> state finished >Thread-15910870::DEBUG::2013-11-19 21:21:25,192::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910870::DEBUG::2013-11-19 21:21:25,192::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910870::DEBUG::2013-11-19 21:21:25,193::task::495::TaskManager.Task::(_debug) Task 15f3ec47-5ff8-41d8-8e0c-bbd7cfdf0c65: ref 0 aborting False >Thread-15910870::INFO::2013-11-19 21:21:25,193::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:25,619::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:25,620::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:25,621::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:25,622::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >934c4d35-aab6-4fba-9996-9cc07dd7d991::INFO::2013-11-19 21:21:25,622::safelease::83::ClusterLock::(release) Releasing cluster lock for domain 6b892067-ac7e-4039-a7be-5b7387525b16 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:25,623::safelease::84::Storage.Misc.excCmd::(release) '/usr/libexec/vdsm/spmstop.sh 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd /usr/libexec/vdsm/) >Thread-15910871::DEBUG::2013-11-19 21:21:26,201::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910871::INFO::2013-11-19 21:21:26,203::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910871::DEBUG::2013-11-19 21:21:26,204::task::495::TaskManager.Task::(_debug) Task ad570ec9-799e-439f-b749-80d14bf6e0d9: moving from state init -> state preparing >Thread-15910871::DEBUG::2013-11-19 21:21:26,204::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910871::DEBUG::2013-11-19 21:21:26,205::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910871::DEBUG::2013-11-19 21:21:26,205::task::495::TaskManager.Task::(_debug) Task ad570ec9-799e-439f-b749-80d14bf6e0d9: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910871::DEBUG::2013-11-19 21:21:26,206::task::495::TaskManager.Task::(_debug) Task ad570ec9-799e-439f-b749-80d14bf6e0d9: moving from state preparing -> state finished >Thread-15910871::DEBUG::2013-11-19 21:21:26,207::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910871::DEBUG::2013-11-19 21:21:26,207::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910871::DEBUG::2013-11-19 21:21:26,208::task::495::TaskManager.Task::(_debug) Task ad570ec9-799e-439f-b749-80d14bf6e0d9: ref 0 aborting False >Thread-15910871::INFO::2013-11-19 21:21:26,208::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,095::safelease::84::Storage.Misc.excCmd::(release) SUCCESS: <err> = ''; <rc> = 0 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,095::safelease::87::ClusterLock::(release) Cluster lock released successfully >934c4d35-aab6-4fba-9996-9cc07dd7d991::ERROR::2013-11-19 21:21:27,096::task::868::TaskManager.Task::(_setError) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 876, in _run > File "/usr/share/vdsm/storage/task.py", line 303, in run > File "/usr/share/vdsm/storage/spm.py", line 493, in start > File "/usr/share/vdsm/storage/sp.py", line 637, in _regulariseOutdatedMasters > File "/usr/share/vdsm/storage/sp.py", line 631, in _setRegularSD > File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ > File "/usr/share/vdsm/storage/persistentDict.py", line 152, in transaction > File "/usr/share/vdsm/storage/persistentDict.py", line 266, in flush > File "/usr/share/vdsm/storage/fileSD.py", line 58, in writelines > File "/usr/share/vdsm/storage/processPool.py", line 33, in wrapper > File "/usr/share/vdsm/storage/processPool.py", line 44, in runExternally > File "/usr/share/vdsm/storage/processPool.py", line 127, in runExternally >IOError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata.new' >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,098::task::495::TaskManager.Task::(_debug) Task 934c4d35-aab6-4fba-9996-9cc07dd7d991: Task._run: 934c4d35-aab6-4fba-9996-9cc07dd7d991 () {} failed - stopping task >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,099::task::495::TaskManager.Task::(_debug) Task 934c4d35-aab6-4fba-9996-9cc07dd7d991: stopping in state running (force False) >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,100::task::495::TaskManager.Task::(_debug) Task 934c4d35-aab6-4fba-9996-9cc07dd7d991: ref 1 aborting True >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,100::task::918::TaskManager.Task::(_runJobs) aborting: Task is aborted: "[Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata.new'" - code 100 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,101::task::495::TaskManager.Task::(_debug) Task 934c4d35-aab6-4fba-9996-9cc07dd7d991: ref 0 aborting True >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,101::task::495::TaskManager.Task::(_debug) Task 934c4d35-aab6-4fba-9996-9cc07dd7d991: Task._doAbort: force False >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,102::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,102::task::495::TaskManager.Task::(_debug) Task 934c4d35-aab6-4fba-9996-9cc07dd7d991: moving from state running -> state aborting >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,103::task::495::TaskManager.Task::(_debug) Task 934c4d35-aab6-4fba-9996-9cc07dd7d991: _aborting: recover policy auto >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,104::task::495::TaskManager.Task::(_debug) Task 934c4d35-aab6-4fba-9996-9cc07dd7d991: moving from state aborting -> state racquiring >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,104::task::495::TaskManager.Task::(_debug) Task 934c4d35-aab6-4fba-9996-9cc07dd7d991: moving from state racquiring -> state recovering >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,105::task::495::TaskManager.Task::(_debug) Task 934c4d35-aab6-4fba-9996-9cc07dd7d991: _recover >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,105::task::495::TaskManager.Task::(_debug) Task 934c4d35-aab6-4fba-9996-9cc07dd7d991: running recovery None >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,106::task::495::TaskManager.Task::(_debug) Task 934c4d35-aab6-4fba-9996-9cc07dd7d991: Recover Done: state recovering >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,107::task::495::TaskManager.Task::(_debug) Task 934c4d35-aab6-4fba-9996-9cc07dd7d991: moving from state recovering -> state recovered >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,107::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,108::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,109::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,111::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,113::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,116::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:21:27,118::threadPool::47::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 >Thread-15910873::DEBUG::2013-11-19 21:21:27,214::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910873::INFO::2013-11-19 21:21:27,215::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910873::DEBUG::2013-11-19 21:21:27,216::task::495::TaskManager.Task::(_debug) Task 85a272df-1c72-450e-85df-99cb98eb625c: moving from state init -> state preparing >Thread-15910873::DEBUG::2013-11-19 21:21:27,216::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910873::DEBUG::2013-11-19 21:21:27,217::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 100, 'message': "[Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata.new'", 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'} >Thread-15910873::DEBUG::2013-11-19 21:21:27,217::task::495::TaskManager.Task::(_debug) Task 85a272df-1c72-450e-85df-99cb98eb625c: finished: {'taskStatus': {'code': 100, 'message': "[Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata.new'", 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910873::DEBUG::2013-11-19 21:21:27,218::task::495::TaskManager.Task::(_debug) Task 85a272df-1c72-450e-85df-99cb98eb625c: moving from state preparing -> state finished >Thread-15910873::DEBUG::2013-11-19 21:21:27,218::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910873::DEBUG::2013-11-19 21:21:27,219::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910873::DEBUG::2013-11-19 21:21:27,219::task::495::TaskManager.Task::(_debug) Task 85a272df-1c72-450e-85df-99cb98eb625c: ref 0 aborting False >Thread-15910873::INFO::2013-11-19 21:21:27,220::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 100, 'message': "[Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata.new'", 'taskState': 'finished', 'taskResult': 'cleanSuccess', 'taskID': '934c4d35-aab6-4fba-9996-9cc07dd7d991'}} >Thread-15910874::DEBUG::2013-11-19 21:21:27,225::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910874::INFO::2013-11-19 21:21:27,225::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910874::DEBUG::2013-11-19 21:21:27,226::task::495::TaskManager.Task::(_debug) Task 0ffaa8d6-92ab-4b25-bdf5-95d4a05af8b4: moving from state init -> state preparing >Thread-15910874::DEBUG::2013-11-19 21:21:27,227::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=Free spmLver=280 spmId=-1 >Thread-15910874::DEBUG::2013-11-19 21:21:27,227::task::495::TaskManager.Task::(_debug) Task 0ffaa8d6-92ab-4b25-bdf5-95d4a05af8b4: finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 280}} >Thread-15910874::DEBUG::2013-11-19 21:21:27,228::task::495::TaskManager.Task::(_debug) Task 0ffaa8d6-92ab-4b25-bdf5-95d4a05af8b4: moving from state preparing -> state finished >Thread-15910874::DEBUG::2013-11-19 21:21:27,228::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910874::DEBUG::2013-11-19 21:21:27,229::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910874::DEBUG::2013-11-19 21:21:27,229::task::495::TaskManager.Task::(_debug) Task 0ffaa8d6-92ab-4b25-bdf5-95d4a05af8b4: ref 0 aborting False >Thread-15910874::INFO::2013-11-19 21:21:27,230::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 280}} >Thread-15910875::DEBUG::2013-11-19 21:21:27,236::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910875::INFO::2013-11-19 21:21:27,236::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: clearTask, args: ( taskID=934c4d35-aab6-4fba-9996-9cc07dd7d991) >Thread-15910875::DEBUG::2013-11-19 21:21:27,237::task::495::TaskManager.Task::(_debug) Task 2cd55f3e-3421-4030-85b2-577d769c8868: moving from state init -> state preparing >Thread-15910875::DEBUG::2013-11-19 21:21:27,237::taskManager::146::TaskManager::(clearTask) Entry. taskID: 934c4d35-aab6-4fba-9996-9cc07dd7d991 >Thread-15910875::DEBUG::2013-11-19 21:21:27,238::taskManager::151::TaskManager::(clearTask) Return. >Thread-15910875::DEBUG::2013-11-19 21:21:27,238::task::495::TaskManager.Task::(_debug) Task 2cd55f3e-3421-4030-85b2-577d769c8868: finished: None >Thread-15910875::DEBUG::2013-11-19 21:21:27,239::task::495::TaskManager.Task::(_debug) Task 2cd55f3e-3421-4030-85b2-577d769c8868: moving from state preparing -> state finished >Thread-15910875::DEBUG::2013-11-19 21:21:27,239::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910875::DEBUG::2013-11-19 21:21:27,240::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910875::DEBUG::2013-11-19 21:21:27,240::task::495::TaskManager.Task::(_debug) Task 2cd55f3e-3421-4030-85b2-577d769c8868: ref 0 aborting False >Thread-15910875::INFO::2013-11-19 21:21:27,241::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: clearTask, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15910876::DEBUG::2013-11-19 21:21:27,315::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15910876::INFO::2013-11-19 21:21:27,316::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15910876::DEBUG::2013-11-19 21:21:27,316::task::495::TaskManager.Task::(_debug) Task 7eae8942-543b-42cb-b181-230ba74e7650: moving from state init -> state preparing >Thread-15910876::DEBUG::2013-11-19 21:21:27,317::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=Free spmLver=280 spmId=-1 >Thread-15910876::DEBUG::2013-11-19 21:21:27,318::task::495::TaskManager.Task::(_debug) Task 7eae8942-543b-42cb-b181-230ba74e7650: finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 280}} >Thread-15910876::DEBUG::2013-11-19 21:21:27,318::task::495::TaskManager.Task::(_debug) Task 7eae8942-543b-42cb-b181-230ba74e7650: moving from state preparing -> state finished >Thread-15910876::DEBUG::2013-11-19 21:21:27,319::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910876::DEBUG::2013-11-19 21:21:27,319::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910876::DEBUG::2013-11-19 21:21:27,320::task::495::TaskManager.Task::(_debug) Task 7eae8942-543b-42cb-b181-230ba74e7650: ref 0 aborting False >Thread-15910876::INFO::2013-11-19 21:21:27,320::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 280}} >Thread-1024767::ERROR::2013-11-19 21:21:29,830::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910879::INFO::2013-11-19 21:21:32,880::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910879::DEBUG::2013-11-19 21:21:32,881::task::495::TaskManager.Task::(_debug) Task a077b9c7-d3d2-4e49-8384-36050d5bb046: moving from state init -> state preparing >Thread-15910879::WARNING::2013-11-19 21:21:32,882::hsm::1895::Storage.HSM::(public_repoStats) repoStats detected invalid master:6b892067-ac7e-4039-a7be-5b7387525b16 {'finish': 1384896083.8064029, 'mdasize': '134479872', 'mdathreshold': True, 'mdavalid': True, 'diskfree': '1142595518464', 'disktotal': '3595827150848', 'result': {'delay': '0.000607013702393', 'lastCheck': 1384896083.8064029, 'valid': True, 'code': 0}, 'masterValidate': {'mount': True, 'valid': True}, 'mdafree': '67205632'} >Thread-15910879::DEBUG::2013-11-19 21:21:32,882::task::495::TaskManager.Task::(_debug) Task a077b9c7-d3d2-4e49-8384-36050d5bb046: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000607013702393', 'lastCheck': 1384896083.8064029, 'valid': True, 'code': 355}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144696235657', 'lastCheck': 1384896084.721595, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896089.831394, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000514984130859', 'lastCheck': 1384896089.5651569, 'valid': True, 'code': 0}} >Thread-15910879::DEBUG::2013-11-19 21:21:32,883::task::495::TaskManager.Task::(_debug) Task a077b9c7-d3d2-4e49-8384-36050d5bb046: moving from state preparing -> state finished >Thread-15910879::DEBUG::2013-11-19 21:21:32,884::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910879::DEBUG::2013-11-19 21:21:32,884::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910879::DEBUG::2013-11-19 21:21:32,885::task::495::TaskManager.Task::(_debug) Task a077b9c7-d3d2-4e49-8384-36050d5bb046: ref 0 aborting False >Thread-15910879::INFO::2013-11-19 21:21:32,885::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000607013702393', 'lastCheck': 1384896083.8064029, 'valid': True, 'code': 355}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144696235657', 'lastCheck': 1384896084.721595, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896089.831394, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000514984130859', 'lastCheck': 1384896089.5651569, 'valid': True, 'code': 0}} >Thread-42838::DEBUG::2013-11-19 21:21:33,832::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-42838::DEBUG::2013-11-19 21:21:33,834::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 21:21:34,452::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-42838::DEBUG::2013-11-19 21:21:34,455::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1024767::ERROR::2013-11-19 21:21:40,051::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910885::INFO::2013-11-19 21:21:43,001::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910885::DEBUG::2013-11-19 21:21:43,002::task::495::TaskManager.Task::(_debug) Task b92fffed-1113-4788-8806-ddfd2f2a4116: moving from state init -> state preparing >Thread-15910885::DEBUG::2013-11-19 21:21:43,003::task::495::TaskManager.Task::(_debug) Task b92fffed-1113-4788-8806-ddfd2f2a4116: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000415086746216', 'lastCheck': 1384896094.459101, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00159406661987', 'lastCheck': 1384896094.8393171, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896100.0524311, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000833988189697', 'lastCheck': 1384896099.5780461, 'valid': True, 'code': 0}} >Thread-15910885::DEBUG::2013-11-19 21:21:43,003::task::495::TaskManager.Task::(_debug) Task b92fffed-1113-4788-8806-ddfd2f2a4116: moving from state preparing -> state finished >Thread-15910885::DEBUG::2013-11-19 21:21:43,004::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910885::DEBUG::2013-11-19 21:21:43,005::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910885::DEBUG::2013-11-19 21:21:43,005::task::495::TaskManager.Task::(_debug) Task b92fffed-1113-4788-8806-ddfd2f2a4116: ref 0 aborting False >Thread-15910885::INFO::2013-11-19 21:21:43,006::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000415086746216', 'lastCheck': 1384896094.459101, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00159406661987', 'lastCheck': 1384896094.8393171, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896100.0524311, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000833988189697', 'lastCheck': 1384896099.5780461, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:21:50,065::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910891::INFO::2013-11-19 21:21:53,110::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910891::DEBUG::2013-11-19 21:21:53,111::task::495::TaskManager.Task::(_debug) Task 5d659818-85ad-4bbc-91f7-901d6200c867: moving from state init -> state preparing >Thread-15910891::DEBUG::2013-11-19 21:21:53,111::task::495::TaskManager.Task::(_debug) Task 5d659818-85ad-4bbc-91f7-901d6200c867: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000335931777954', 'lastCheck': 1384896104.471967, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00227117538452', 'lastCheck': 1384896104.856185, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896110.0663061, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000463962554932', 'lastCheck': 1384896109.588975, 'valid': True, 'code': 0}} >Thread-15910891::DEBUG::2013-11-19 21:21:53,112::task::495::TaskManager.Task::(_debug) Task 5d659818-85ad-4bbc-91f7-901d6200c867: moving from state preparing -> state finished >Thread-15910891::DEBUG::2013-11-19 21:21:53,113::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910891::DEBUG::2013-11-19 21:21:53,113::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910891::DEBUG::2013-11-19 21:21:53,114::task::495::TaskManager.Task::(_debug) Task 5d659818-85ad-4bbc-91f7-901d6200c867: ref 0 aborting False >Thread-15910891::INFO::2013-11-19 21:21:53,114::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000335931777954', 'lastCheck': 1384896104.471967, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00227117538452', 'lastCheck': 1384896104.856185, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896110.0663061, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000463962554932', 'lastCheck': 1384896109.588975, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:22:00,079::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910897::INFO::2013-11-19 21:22:03,219::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910897::DEBUG::2013-11-19 21:22:03,220::task::495::TaskManager.Task::(_debug) Task 6deeb172-3836-4f59-85e0-b9ed64c774df: moving from state init -> state preparing >Thread-15910897::DEBUG::2013-11-19 21:22:03,220::task::495::TaskManager.Task::(_debug) Task 6deeb172-3836-4f59-85e0-b9ed64c774df: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000320911407471', 'lastCheck': 1384896114.4847939, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142407417297', 'lastCheck': 1384896114.8720989, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896120.0802431, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000452041625977', 'lastCheck': 1384896119.600466, 'valid': True, 'code': 0}} >Thread-15910897::DEBUG::2013-11-19 21:22:03,221::task::495::TaskManager.Task::(_debug) Task 6deeb172-3836-4f59-85e0-b9ed64c774df: moving from state preparing -> state finished >Thread-15910897::DEBUG::2013-11-19 21:22:03,222::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910897::DEBUG::2013-11-19 21:22:03,222::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910897::DEBUG::2013-11-19 21:22:03,223::task::495::TaskManager.Task::(_debug) Task 6deeb172-3836-4f59-85e0-b9ed64c774df: ref 0 aborting False >Thread-15910897::INFO::2013-11-19 21:22:03,223::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000320911407471', 'lastCheck': 1384896114.4847939, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142407417297', 'lastCheck': 1384896114.8720989, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896120.0802431, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000452041625977', 'lastCheck': 1384896119.600466, 'valid': True, 'code': 0}} >Thread-42840::DEBUG::2013-11-19 21:22:09,611::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:22:10,094::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 21:22:10,335::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15910903::INFO::2013-11-19 21:22:13,403::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910903::DEBUG::2013-11-19 21:22:13,404::task::495::TaskManager.Task::(_debug) Task 6466cc32-feb7-41f7-80d5-d7a9827acb2b: moving from state init -> state preparing >Thread-15910903::DEBUG::2013-11-19 21:22:13,404::task::495::TaskManager.Task::(_debug) Task 6466cc32-feb7-41f7-80d5-d7a9827acb2b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000392913818359', 'lastCheck': 1384896124.4979379, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154399871826', 'lastCheck': 1384896124.887991, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896130.0957601, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000317096710205', 'lastCheck': 1384896130.3372171, 'valid': True, 'code': 0}} >Thread-15910903::DEBUG::2013-11-19 21:22:13,405::task::495::TaskManager.Task::(_debug) Task 6466cc32-feb7-41f7-80d5-d7a9827acb2b: moving from state preparing -> state finished >Thread-15910903::DEBUG::2013-11-19 21:22:13,406::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910903::DEBUG::2013-11-19 21:22:13,406::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910903::DEBUG::2013-11-19 21:22:13,407::task::495::TaskManager.Task::(_debug) Task 6466cc32-feb7-41f7-80d5-d7a9827acb2b: ref 0 aborting False >Thread-15910903::INFO::2013-11-19 21:22:13,408::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000392913818359', 'lastCheck': 1384896124.4979379, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154399871826', 'lastCheck': 1384896124.887991, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896130.0957601, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000317096710205', 'lastCheck': 1384896130.3372171, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:22:20,109::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910909::INFO::2013-11-19 21:22:23,539::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910909::DEBUG::2013-11-19 21:22:23,540::task::495::TaskManager.Task::(_debug) Task 56ba6387-fc78-46b8-8319-16f461b57c6d: moving from state init -> state preparing >Thread-15910909::DEBUG::2013-11-19 21:22:23,540::task::495::TaskManager.Task::(_debug) Task 56ba6387-fc78-46b8-8319-16f461b57c6d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000443935394287', 'lastCheck': 1384896134.5114851, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00145697593689', 'lastCheck': 1384896134.9042151, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896140.109766, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000392198562622', 'lastCheck': 1384896140.348901, 'valid': True, 'code': 0}} >Thread-15910909::DEBUG::2013-11-19 21:22:23,541::task::495::TaskManager.Task::(_debug) Task 56ba6387-fc78-46b8-8319-16f461b57c6d: moving from state preparing -> state finished >Thread-15910909::DEBUG::2013-11-19 21:22:23,542::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910909::DEBUG::2013-11-19 21:22:23,542::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910909::DEBUG::2013-11-19 21:22:23,543::task::495::TaskManager.Task::(_debug) Task 56ba6387-fc78-46b8-8319-16f461b57c6d: ref 0 aborting False >Thread-15910909::INFO::2013-11-19 21:22:23,543::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000443935394287', 'lastCheck': 1384896134.5114851, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00145697593689', 'lastCheck': 1384896134.9042151, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896140.109766, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000392198562622', 'lastCheck': 1384896140.348901, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:22:30,124::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910915::INFO::2013-11-19 21:22:33,650::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910915::DEBUG::2013-11-19 21:22:33,651::task::495::TaskManager.Task::(_debug) Task 13ecfc05-990e-47b7-a348-4d9fe9ebdcc4: moving from state init -> state preparing >Thread-15910915::DEBUG::2013-11-19 21:22:33,652::task::495::TaskManager.Task::(_debug) Task 13ecfc05-990e-47b7-a348-4d9fe9ebdcc4: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000395774841309', 'lastCheck': 1384896144.5244119, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144600868225', 'lastCheck': 1384896144.9200881, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896150.1260059, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000379085540771', 'lastCheck': 1384896150.360539, 'valid': True, 'code': 0}} >Thread-15910915::DEBUG::2013-11-19 21:22:33,653::task::495::TaskManager.Task::(_debug) Task 13ecfc05-990e-47b7-a348-4d9fe9ebdcc4: moving from state preparing -> state finished >Thread-15910915::DEBUG::2013-11-19 21:22:33,653::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910915::DEBUG::2013-11-19 21:22:33,654::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910915::DEBUG::2013-11-19 21:22:33,654::task::495::TaskManager.Task::(_debug) Task 13ecfc05-990e-47b7-a348-4d9fe9ebdcc4: ref 0 aborting False >Thread-15910915::INFO::2013-11-19 21:22:33,655::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000395774841309', 'lastCheck': 1384896144.5244119, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144600868225', 'lastCheck': 1384896144.9200881, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896150.1260059, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000379085540771', 'lastCheck': 1384896150.360539, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:22:40,139::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910921::INFO::2013-11-19 21:22:43,757::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910921::DEBUG::2013-11-19 21:22:43,757::task::495::TaskManager.Task::(_debug) Task 0d52b733-9f6b-4a23-b41a-3d540ff33ef0: moving from state init -> state preparing >Thread-15910921::DEBUG::2013-11-19 21:22:43,758::task::495::TaskManager.Task::(_debug) Task 0d52b733-9f6b-4a23-b41a-3d540ff33ef0: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000375032424927', 'lastCheck': 1384896154.537642, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00153112411499', 'lastCheck': 1384896154.9361269, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896160.140373, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000463008880615', 'lastCheck': 1384896160.372328, 'valid': True, 'code': 0}} >Thread-15910921::DEBUG::2013-11-19 21:22:43,759::task::495::TaskManager.Task::(_debug) Task 0d52b733-9f6b-4a23-b41a-3d540ff33ef0: moving from state preparing -> state finished >Thread-15910921::DEBUG::2013-11-19 21:22:43,759::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910921::DEBUG::2013-11-19 21:22:43,760::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910921::DEBUG::2013-11-19 21:22:43,760::task::495::TaskManager.Task::(_debug) Task 0d52b733-9f6b-4a23-b41a-3d540ff33ef0: ref 0 aborting False >Thread-15910921::INFO::2013-11-19 21:22:43,761::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000375032424927', 'lastCheck': 1384896154.537642, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00153112411499', 'lastCheck': 1384896154.9361269, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896160.140373, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000463008880615', 'lastCheck': 1384896160.372328, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:22:50,153::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910927::INFO::2013-11-19 21:22:53,876::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910927::DEBUG::2013-11-19 21:22:53,876::task::495::TaskManager.Task::(_debug) Task 0a44c657-0190-447d-a802-8d6c2ba6863e: moving from state init -> state preparing >Thread-15910927::DEBUG::2013-11-19 21:22:53,877::task::495::TaskManager.Task::(_debug) Task 0a44c657-0190-447d-a802-8d6c2ba6863e: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.0003821849823', 'lastCheck': 1384896164.5506649, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00227904319763', 'lastCheck': 1384896164.952817, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896170.1539831, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000408887863159', 'lastCheck': 1384896170.3839791, 'valid': True, 'code': 0}} >Thread-15910927::DEBUG::2013-11-19 21:22:53,878::task::495::TaskManager.Task::(_debug) Task 0a44c657-0190-447d-a802-8d6c2ba6863e: moving from state preparing -> state finished >Thread-15910927::DEBUG::2013-11-19 21:22:53,878::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910927::DEBUG::2013-11-19 21:22:53,879::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910927::DEBUG::2013-11-19 21:22:53,879::task::495::TaskManager.Task::(_debug) Task 0a44c657-0190-447d-a802-8d6c2ba6863e: ref 0 aborting False >Thread-15910927::INFO::2013-11-19 21:22:53,880::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.0003821849823', 'lastCheck': 1384896164.5506649, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00227904319763', 'lastCheck': 1384896164.952817, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896170.1539831, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000408887863159', 'lastCheck': 1384896170.3839791, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:23:00,166::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910933::INFO::2013-11-19 21:23:03,985::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910933::DEBUG::2013-11-19 21:23:03,986::task::495::TaskManager.Task::(_debug) Task 3e68bc61-b148-4202-8a20-01a2fce952bc: moving from state init -> state preparing >Thread-15910933::DEBUG::2013-11-19 21:23:03,987::task::495::TaskManager.Task::(_debug) Task 3e68bc61-b148-4202-8a20-01a2fce952bc: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00349116325378', 'lastCheck': 1384896174.567354, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00141096115112', 'lastCheck': 1384896174.9681931, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896180.167511, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000316143035889', 'lastCheck': 1384896180.3956001, 'valid': True, 'code': 0}} >Thread-15910933::DEBUG::2013-11-19 21:23:03,987::task::495::TaskManager.Task::(_debug) Task 3e68bc61-b148-4202-8a20-01a2fce952bc: moving from state preparing -> state finished >Thread-15910933::DEBUG::2013-11-19 21:23:03,988::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910933::DEBUG::2013-11-19 21:23:03,989::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910933::DEBUG::2013-11-19 21:23:03,989::task::495::TaskManager.Task::(_debug) Task 3e68bc61-b148-4202-8a20-01a2fce952bc: ref 0 aborting False >Thread-15910933::INFO::2013-11-19 21:23:03,990::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00349116325378', 'lastCheck': 1384896174.567354, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00141096115112', 'lastCheck': 1384896174.9681931, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896180.167511, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000316143035889', 'lastCheck': 1384896180.3956001, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:23:10,180::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910939::INFO::2013-11-19 21:23:14,103::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910939::DEBUG::2013-11-19 21:23:14,104::task::495::TaskManager.Task::(_debug) Task 07394d7d-4cca-49cf-9765-44b850e508c4: moving from state init -> state preparing >Thread-15910939::DEBUG::2013-11-19 21:23:14,105::task::495::TaskManager.Task::(_debug) Task 07394d7d-4cca-49cf-9765-44b850e508c4: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000401973724365', 'lastCheck': 1384896184.580133, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142002105713', 'lastCheck': 1384896184.9836559, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896190.180896, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000381946563721', 'lastCheck': 1384896190.4072521, 'valid': True, 'code': 0}} >Thread-15910939::DEBUG::2013-11-19 21:23:14,105::task::495::TaskManager.Task::(_debug) Task 07394d7d-4cca-49cf-9765-44b850e508c4: moving from state preparing -> state finished >Thread-15910939::DEBUG::2013-11-19 21:23:14,106::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910939::DEBUG::2013-11-19 21:23:14,107::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910939::DEBUG::2013-11-19 21:23:14,107::task::495::TaskManager.Task::(_debug) Task 07394d7d-4cca-49cf-9765-44b850e508c4: ref 0 aborting False >Thread-15910939::INFO::2013-11-19 21:23:14,108::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000401973724365', 'lastCheck': 1384896184.580133, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142002105713', 'lastCheck': 1384896184.9836559, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896190.180896, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000381946563721', 'lastCheck': 1384896190.4072521, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:23:20,194::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910945::INFO::2013-11-19 21:23:24,211::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910945::DEBUG::2013-11-19 21:23:24,211::task::495::TaskManager.Task::(_debug) Task 80d5bec1-7256-43d6-ab0b-e89ea1e988fc: moving from state init -> state preparing >Thread-15910945::DEBUG::2013-11-19 21:23:24,212::task::495::TaskManager.Task::(_debug) Task 80d5bec1-7256-43d6-ab0b-e89ea1e988fc: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00031590461731', 'lastCheck': 1384896194.5912099, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00140500068665', 'lastCheck': 1384896194.9991951, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896200.195117, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000307083129883', 'lastCheck': 1384896200.418812, 'valid': True, 'code': 0}} >Thread-15910945::DEBUG::2013-11-19 21:23:24,213::task::495::TaskManager.Task::(_debug) Task 80d5bec1-7256-43d6-ab0b-e89ea1e988fc: moving from state preparing -> state finished >Thread-15910945::DEBUG::2013-11-19 21:23:24,213::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910945::DEBUG::2013-11-19 21:23:24,214::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910945::DEBUG::2013-11-19 21:23:24,214::task::495::TaskManager.Task::(_debug) Task 80d5bec1-7256-43d6-ab0b-e89ea1e988fc: ref 0 aborting False >Thread-15910945::INFO::2013-11-19 21:23:24,215::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00031590461731', 'lastCheck': 1384896194.5912099, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00140500068665', 'lastCheck': 1384896194.9991951, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896200.195117, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000307083129883', 'lastCheck': 1384896200.418812, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:23:30,208::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910951::INFO::2013-11-19 21:23:34,360::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910951::DEBUG::2013-11-19 21:23:34,361::task::495::TaskManager.Task::(_debug) Task 21369c81-34ba-473b-8274-fa3ec7f0b785: moving from state init -> state preparing >Thread-15910951::DEBUG::2013-11-19 21:23:34,361::task::495::TaskManager.Task::(_debug) Task 21369c81-34ba-473b-8274-fa3ec7f0b785: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000311136245728', 'lastCheck': 1384896204.6038909, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144982337952', 'lastCheck': 1384896205.014539, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896210.208775, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000296115875244', 'lastCheck': 1384896210.430301, 'valid': True, 'code': 0}} >Thread-15910951::DEBUG::2013-11-19 21:23:34,362::task::495::TaskManager.Task::(_debug) Task 21369c81-34ba-473b-8274-fa3ec7f0b785: moving from state preparing -> state finished >Thread-15910951::DEBUG::2013-11-19 21:23:34,363::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910951::DEBUG::2013-11-19 21:23:34,363::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910951::DEBUG::2013-11-19 21:23:34,364::task::495::TaskManager.Task::(_debug) Task 21369c81-34ba-473b-8274-fa3ec7f0b785: ref 0 aborting False >Thread-15910951::INFO::2013-11-19 21:23:34,364::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000311136245728', 'lastCheck': 1384896204.6038909, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144982337952', 'lastCheck': 1384896205.014539, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896210.208775, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000296115875244', 'lastCheck': 1384896210.430301, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:23:40,221::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910957::INFO::2013-11-19 21:23:44,464::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910957::DEBUG::2013-11-19 21:23:44,465::task::495::TaskManager.Task::(_debug) Task 7eaaad37-7428-487e-ac28-1fe6bba66769: moving from state init -> state preparing >Thread-15910957::DEBUG::2013-11-19 21:23:44,465::task::495::TaskManager.Task::(_debug) Task 7eaaad37-7428-487e-ac28-1fe6bba66769: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00039005279541', 'lastCheck': 1384896214.6168921, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00134086608887', 'lastCheck': 1384896215.030549, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896220.222296, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000423908233643', 'lastCheck': 1384896220.4417379, 'valid': True, 'code': 0}} >Thread-15910957::DEBUG::2013-11-19 21:23:44,466::task::495::TaskManager.Task::(_debug) Task 7eaaad37-7428-487e-ac28-1fe6bba66769: moving from state preparing -> state finished >Thread-15910957::DEBUG::2013-11-19 21:23:44,467::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910957::DEBUG::2013-11-19 21:23:44,467::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910957::DEBUG::2013-11-19 21:23:44,468::task::495::TaskManager.Task::(_debug) Task 7eaaad37-7428-487e-ac28-1fe6bba66769: ref 0 aborting False >Thread-15910957::INFO::2013-11-19 21:23:44,468::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00039005279541', 'lastCheck': 1384896214.6168921, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00134086608887', 'lastCheck': 1384896215.030549, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896220.222296, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000423908233643', 'lastCheck': 1384896220.4417379, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:23:50,235::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910963::INFO::2013-11-19 21:23:54,574::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910963::DEBUG::2013-11-19 21:23:54,580::task::495::TaskManager.Task::(_debug) Task 25c06ea8-5164-43e7-ab0f-59b1cac68971: moving from state init -> state preparing >Thread-15910963::DEBUG::2013-11-19 21:23:54,582::task::495::TaskManager.Task::(_debug) Task 25c06ea8-5164-43e7-ab0f-59b1cac68971: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000354051589966', 'lastCheck': 1384896224.629945, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00227999687195', 'lastCheck': 1384896225.047545, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896230.236021, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000418186187744', 'lastCheck': 1384896230.4534709, 'valid': True, 'code': 0}} >Thread-15910963::DEBUG::2013-11-19 21:23:54,583::task::495::TaskManager.Task::(_debug) Task 25c06ea8-5164-43e7-ab0f-59b1cac68971: moving from state preparing -> state finished >Thread-15910963::DEBUG::2013-11-19 21:23:54,584::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910963::DEBUG::2013-11-19 21:23:54,585::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910963::DEBUG::2013-11-19 21:23:54,587::task::495::TaskManager.Task::(_debug) Task 25c06ea8-5164-43e7-ab0f-59b1cac68971: ref 0 aborting False >Thread-15910963::INFO::2013-11-19 21:23:54,587::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000354051589966', 'lastCheck': 1384896224.629945, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00227999687195', 'lastCheck': 1384896225.047545, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896230.236021, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000418186187744', 'lastCheck': 1384896230.4534709, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:24:00,249::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910969::INFO::2013-11-19 21:24:04,692::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910969::DEBUG::2013-11-19 21:24:04,693::task::495::TaskManager.Task::(_debug) Task 10c66dc0-3d4c-4eff-bd3b-800dba1440b5: moving from state init -> state preparing >Thread-15910969::DEBUG::2013-11-19 21:24:04,693::task::495::TaskManager.Task::(_debug) Task 10c66dc0-3d4c-4eff-bd3b-800dba1440b5: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000418901443481', 'lastCheck': 1384896244.654932, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00141406059265', 'lastCheck': 1384896235.0631771, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896240.2496951, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000428915023804', 'lastCheck': 1384896240.465271, 'valid': True, 'code': 0}} >Thread-15910969::DEBUG::2013-11-19 21:24:04,694::task::495::TaskManager.Task::(_debug) Task 10c66dc0-3d4c-4eff-bd3b-800dba1440b5: moving from state preparing -> state finished >Thread-15910969::DEBUG::2013-11-19 21:24:04,695::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910969::DEBUG::2013-11-19 21:24:04,695::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910969::DEBUG::2013-11-19 21:24:04,696::task::495::TaskManager.Task::(_debug) Task 10c66dc0-3d4c-4eff-bd3b-800dba1440b5: ref 0 aborting False >Thread-15910969::INFO::2013-11-19 21:24:04,696::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000418901443481', 'lastCheck': 1384896244.654932, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00141406059265', 'lastCheck': 1384896235.0631771, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896240.2496951, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000428915023804', 'lastCheck': 1384896240.465271, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:24:10,262::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910975::INFO::2013-11-19 21:24:14,798::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910975::DEBUG::2013-11-19 21:24:14,799::task::495::TaskManager.Task::(_debug) Task d380e398-88f5-497a-8e9c-6ac8cb67e86e: moving from state init -> state preparing >Thread-15910975::DEBUG::2013-11-19 21:24:14,799::task::495::TaskManager.Task::(_debug) Task d380e398-88f5-497a-8e9c-6ac8cb67e86e: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000379085540771', 'lastCheck': 1384896254.667459, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147199630737', 'lastCheck': 1384896245.0791991, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896250.263453, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000432968139648', 'lastCheck': 1384896250.4769809, 'valid': True, 'code': 0}} >Thread-15910975::DEBUG::2013-11-19 21:24:14,800::task::495::TaskManager.Task::(_debug) Task d380e398-88f5-497a-8e9c-6ac8cb67e86e: moving from state preparing -> state finished >Thread-15910975::DEBUG::2013-11-19 21:24:14,801::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910975::DEBUG::2013-11-19 21:24:14,801::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910975::DEBUG::2013-11-19 21:24:14,802::task::495::TaskManager.Task::(_debug) Task d380e398-88f5-497a-8e9c-6ac8cb67e86e: ref 0 aborting False >Thread-15910975::INFO::2013-11-19 21:24:14,802::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000379085540771', 'lastCheck': 1384896254.667459, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147199630737', 'lastCheck': 1384896245.0791991, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896250.263453, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000432968139648', 'lastCheck': 1384896250.4769809, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:24:20,276::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910981::INFO::2013-11-19 21:24:24,956::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910981::DEBUG::2013-11-19 21:24:24,957::task::495::TaskManager.Task::(_debug) Task dda289da-37c7-485b-8731-149e7de23922: moving from state init -> state preparing >Thread-15910981::DEBUG::2013-11-19 21:24:24,957::task::495::TaskManager.Task::(_debug) Task dda289da-37c7-485b-8731-149e7de23922: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000313997268677', 'lastCheck': 1384896264.6799359, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142598152161', 'lastCheck': 1384896255.095057, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896260.2771511, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000421047210693', 'lastCheck': 1384896260.4887309, 'valid': True, 'code': 0}} >Thread-15910981::DEBUG::2013-11-19 21:24:24,958::task::495::TaskManager.Task::(_debug) Task dda289da-37c7-485b-8731-149e7de23922: moving from state preparing -> state finished >Thread-15910981::DEBUG::2013-11-19 21:24:24,958::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910981::DEBUG::2013-11-19 21:24:24,959::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910981::DEBUG::2013-11-19 21:24:24,959::task::495::TaskManager.Task::(_debug) Task dda289da-37c7-485b-8731-149e7de23922: ref 0 aborting False >Thread-15910981::INFO::2013-11-19 21:24:24,960::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000313997268677', 'lastCheck': 1384896264.6799359, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142598152161', 'lastCheck': 1384896255.095057, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896260.2771511, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000421047210693', 'lastCheck': 1384896260.4887309, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:24:30,290::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 21:24:34,691::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15910987::INFO::2013-11-19 21:24:35,067::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910987::DEBUG::2013-11-19 21:24:35,068::task::495::TaskManager.Task::(_debug) Task 2be574eb-57aa-4c63-93b6-b949e5de9c88: moving from state init -> state preparing >Thread-15910987::DEBUG::2013-11-19 21:24:35,069::task::495::TaskManager.Task::(_debug) Task 2be574eb-57aa-4c63-93b6-b949e5de9c88: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000313997268677', 'lastCheck': 1384896264.6799359, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144600868225', 'lastCheck': 1384896265.111063, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896270.2910099, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000360012054443', 'lastCheck': 1384896270.5004399, 'valid': True, 'code': 0}} >Thread-15910987::DEBUG::2013-11-19 21:24:35,070::task::495::TaskManager.Task::(_debug) Task 2be574eb-57aa-4c63-93b6-b949e5de9c88: moving from state preparing -> state finished >Thread-15910987::DEBUG::2013-11-19 21:24:35,071::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910987::DEBUG::2013-11-19 21:24:35,071::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910987::DEBUG::2013-11-19 21:24:35,072::task::495::TaskManager.Task::(_debug) Task 2be574eb-57aa-4c63-93b6-b949e5de9c88: ref 0 aborting False >Thread-15910987::INFO::2013-11-19 21:24:35,072::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000313997268677', 'lastCheck': 1384896264.6799359, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144600868225', 'lastCheck': 1384896265.111063, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896270.2910099, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000360012054443', 'lastCheck': 1384896270.5004399, 'valid': True, 'code': 0}} >Thread-42838::DEBUG::2013-11-19 21:24:35,399::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:24:40,304::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910993::INFO::2013-11-19 21:24:45,204::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910993::DEBUG::2013-11-19 21:24:45,204::task::495::TaskManager.Task::(_debug) Task d8d9df0b-164d-4280-a366-55d2e1601171: moving from state init -> state preparing >Thread-15910993::DEBUG::2013-11-19 21:24:45,205::task::495::TaskManager.Task::(_debug) Task d8d9df0b-164d-4280-a366-55d2e1601171: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000316143035889', 'lastCheck': 1384896275.402132, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00249290466309', 'lastCheck': 1384896285.145678, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896280.3052809, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000444173812866', 'lastCheck': 1384896280.5122001, 'valid': True, 'code': 0}} >Thread-15910993::DEBUG::2013-11-19 21:24:45,206::task::495::TaskManager.Task::(_debug) Task d8d9df0b-164d-4280-a366-55d2e1601171: moving from state preparing -> state finished >Thread-15910993::DEBUG::2013-11-19 21:24:45,206::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910993::DEBUG::2013-11-19 21:24:45,207::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910993::DEBUG::2013-11-19 21:24:45,207::task::495::TaskManager.Task::(_debug) Task d8d9df0b-164d-4280-a366-55d2e1601171: ref 0 aborting False >Thread-15910993::INFO::2013-11-19 21:24:45,208::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000316143035889', 'lastCheck': 1384896275.402132, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00249290466309', 'lastCheck': 1384896285.145678, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896280.3052809, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000444173812866', 'lastCheck': 1384896280.5122001, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:24:50,317::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15910999::INFO::2013-11-19 21:24:55,315::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15910999::DEBUG::2013-11-19 21:24:55,315::task::495::TaskManager.Task::(_debug) Task ca555fbf-150a-42b4-967a-89d475b807b9: moving from state init -> state preparing >Thread-15910999::DEBUG::2013-11-19 21:24:55,316::task::495::TaskManager.Task::(_debug) Task ca555fbf-150a-42b4-967a-89d475b807b9: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000219106674194', 'lastCheck': 1384896285.4141891, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00143218040466', 'lastCheck': 1384896295.1599159, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896290.31884, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000364065170288', 'lastCheck': 1384896290.5236421, 'valid': True, 'code': 0}} >Thread-15910999::DEBUG::2013-11-19 21:24:55,317::task::495::TaskManager.Task::(_debug) Task ca555fbf-150a-42b4-967a-89d475b807b9: moving from state preparing -> state finished >Thread-15910999::DEBUG::2013-11-19 21:24:55,317::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15910999::DEBUG::2013-11-19 21:24:55,318::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15910999::DEBUG::2013-11-19 21:24:55,318::task::495::TaskManager.Task::(_debug) Task ca555fbf-150a-42b4-967a-89d475b807b9: ref 0 aborting False >Thread-15910999::INFO::2013-11-19 21:24:55,327::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000219106674194', 'lastCheck': 1384896285.4141891, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00143218040466', 'lastCheck': 1384896295.1599159, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896290.31884, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000364065170288', 'lastCheck': 1384896290.5236421, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:25:00,331::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911005::INFO::2013-11-19 21:25:05,438::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911005::DEBUG::2013-11-19 21:25:05,440::task::495::TaskManager.Task::(_debug) Task 7e705f0f-13cb-46e7-8f5e-59bba9d5b1b0: moving from state init -> state preparing >Thread-15911005::DEBUG::2013-11-19 21:25:05,440::task::495::TaskManager.Task::(_debug) Task 7e705f0f-13cb-46e7-8f5e-59bba9d5b1b0: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000633955001831', 'lastCheck': 1384896305.4404781, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00149202346802', 'lastCheck': 1384896305.176949, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896300.3321509, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000327110290527', 'lastCheck': 1384896300.5350029, 'valid': True, 'code': 0}} >Thread-15911005::DEBUG::2013-11-19 21:25:05,441::task::495::TaskManager.Task::(_debug) Task 7e705f0f-13cb-46e7-8f5e-59bba9d5b1b0: moving from state preparing -> state finished >Thread-15911005::DEBUG::2013-11-19 21:25:05,442::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911005::DEBUG::2013-11-19 21:25:05,442::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911005::DEBUG::2013-11-19 21:25:05,443::task::495::TaskManager.Task::(_debug) Task 7e705f0f-13cb-46e7-8f5e-59bba9d5b1b0: ref 0 aborting False >Thread-15911005::INFO::2013-11-19 21:25:05,443::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000633955001831', 'lastCheck': 1384896305.4404781, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00149202346802', 'lastCheck': 1384896305.176949, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896300.3321509, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000327110290527', 'lastCheck': 1384896300.5350029, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:25:10,345::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911011::INFO::2013-11-19 21:25:15,549::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911011::DEBUG::2013-11-19 21:25:15,550::task::495::TaskManager.Task::(_debug) Task 42b01541-e880-41c0-97e3-cf2f45b981f2: moving from state init -> state preparing >Thread-15911011::DEBUG::2013-11-19 21:25:15,550::task::495::TaskManager.Task::(_debug) Task 42b01541-e880-41c0-97e3-cf2f45b981f2: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000323057174683', 'lastCheck': 1384896315.453151, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00143718719482', 'lastCheck': 1384896315.1922719, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896310.3458619, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000452041625977', 'lastCheck': 1384896310.5464771, 'valid': True, 'code': 0}} >Thread-15911011::DEBUG::2013-11-19 21:25:15,551::task::495::TaskManager.Task::(_debug) Task 42b01541-e880-41c0-97e3-cf2f45b981f2: moving from state preparing -> state finished >Thread-15911011::DEBUG::2013-11-19 21:25:15,552::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911011::DEBUG::2013-11-19 21:25:15,552::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911011::DEBUG::2013-11-19 21:25:15,553::task::495::TaskManager.Task::(_debug) Task 42b01541-e880-41c0-97e3-cf2f45b981f2: ref 0 aborting False >Thread-15911011::INFO::2013-11-19 21:25:15,553::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000323057174683', 'lastCheck': 1384896315.453151, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00143718719482', 'lastCheck': 1384896315.1922719, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896310.3458619, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000452041625977', 'lastCheck': 1384896310.5464771, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:25:20,358::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911017::INFO::2013-11-19 21:25:25,665::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911017::DEBUG::2013-11-19 21:25:25,666::task::495::TaskManager.Task::(_debug) Task 397d33ce-a1c4-4b2a-9537-853782052a17: moving from state init -> state preparing >Thread-15911017::DEBUG::2013-11-19 21:25:25,676::task::495::TaskManager.Task::(_debug) Task 397d33ce-a1c4-4b2a-9537-853782052a17: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000262022018433', 'lastCheck': 1384896325.465745, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00149488449097', 'lastCheck': 1384896325.2081001, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896320.3593431, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000322818756104', 'lastCheck': 1384896320.558249, 'valid': True, 'code': 0}} >Thread-15911017::DEBUG::2013-11-19 21:25:25,677::task::495::TaskManager.Task::(_debug) Task 397d33ce-a1c4-4b2a-9537-853782052a17: moving from state preparing -> state finished >Thread-15911017::DEBUG::2013-11-19 21:25:25,677::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911017::DEBUG::2013-11-19 21:25:25,678::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911017::DEBUG::2013-11-19 21:25:25,678::task::495::TaskManager.Task::(_debug) Task 397d33ce-a1c4-4b2a-9537-853782052a17: ref 0 aborting False >Thread-15911017::INFO::2013-11-19 21:25:25,679::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000262022018433', 'lastCheck': 1384896325.465745, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00149488449097', 'lastCheck': 1384896325.2081001, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896320.3593431, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000322818756104', 'lastCheck': 1384896320.558249, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:25:30,372::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911023::INFO::2013-11-19 21:25:35,785::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911023::DEBUG::2013-11-19 21:25:35,786::task::495::TaskManager.Task::(_debug) Task 522eae04-da3e-4e10-b195-cdb0fc8b8026: moving from state init -> state preparing >Thread-15911023::DEBUG::2013-11-19 21:25:35,786::task::495::TaskManager.Task::(_debug) Task 522eae04-da3e-4e10-b195-cdb0fc8b8026: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000318050384521', 'lastCheck': 1384896335.4784651, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152707099915', 'lastCheck': 1384896335.2240939, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896330.3729451, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000387907028198', 'lastCheck': 1384896330.569663, 'valid': True, 'code': 0}} >Thread-15911023::DEBUG::2013-11-19 21:25:35,787::task::495::TaskManager.Task::(_debug) Task 522eae04-da3e-4e10-b195-cdb0fc8b8026: moving from state preparing -> state finished >Thread-15911023::DEBUG::2013-11-19 21:25:35,788::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911023::DEBUG::2013-11-19 21:25:35,788::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911023::DEBUG::2013-11-19 21:25:35,789::task::495::TaskManager.Task::(_debug) Task 522eae04-da3e-4e10-b195-cdb0fc8b8026: ref 0 aborting False >Thread-15911023::INFO::2013-11-19 21:25:35,789::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000318050384521', 'lastCheck': 1384896335.4784651, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152707099915', 'lastCheck': 1384896335.2240939, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896330.3729451, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000387907028198', 'lastCheck': 1384896330.569663, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:25:40,385::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911029::INFO::2013-11-19 21:25:45,901::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911029::DEBUG::2013-11-19 21:25:45,902::task::495::TaskManager.Task::(_debug) Task 6345fee6-bb46-45c2-bb98-985b33b18f08: moving from state init -> state preparing >Thread-15911029::DEBUG::2013-11-19 21:25:45,911::task::495::TaskManager.Task::(_debug) Task 6345fee6-bb46-45c2-bb98-985b33b18f08: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000273942947388', 'lastCheck': 1384896345.4910541, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00222897529602', 'lastCheck': 1384896345.2406721, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896340.3866911, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000345945358276', 'lastCheck': 1384896340.580718, 'valid': True, 'code': 0}} >Thread-15911029::DEBUG::2013-11-19 21:25:45,912::task::495::TaskManager.Task::(_debug) Task 6345fee6-bb46-45c2-bb98-985b33b18f08: moving from state preparing -> state finished >Thread-15911029::DEBUG::2013-11-19 21:25:45,913::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911029::DEBUG::2013-11-19 21:25:45,913::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911029::DEBUG::2013-11-19 21:25:45,914::task::495::TaskManager.Task::(_debug) Task 6345fee6-bb46-45c2-bb98-985b33b18f08: ref 0 aborting False >Thread-15911029::INFO::2013-11-19 21:25:45,914::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000273942947388', 'lastCheck': 1384896345.4910541, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00222897529602', 'lastCheck': 1384896345.2406721, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896340.3866911, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000345945358276', 'lastCheck': 1384896340.580718, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:25:50,399::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911035::INFO::2013-11-19 21:25:56,021::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911035::DEBUG::2013-11-19 21:25:56,022::task::495::TaskManager.Task::(_debug) Task 6a5c5a5f-6a45-4470-858e-e0caff9574d5: moving from state init -> state preparing >Thread-15911035::DEBUG::2013-11-19 21:25:56,022::task::495::TaskManager.Task::(_debug) Task 6a5c5a5f-6a45-4470-858e-e0caff9574d5: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000296115875244', 'lastCheck': 1384896355.503885, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142002105713', 'lastCheck': 1384896355.2560771, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896350.400012, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000488996505737', 'lastCheck': 1384896350.5918469, 'valid': True, 'code': 0}} >Thread-15911035::DEBUG::2013-11-19 21:25:56,023::task::495::TaskManager.Task::(_debug) Task 6a5c5a5f-6a45-4470-858e-e0caff9574d5: moving from state preparing -> state finished >Thread-15911035::DEBUG::2013-11-19 21:25:56,024::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911035::DEBUG::2013-11-19 21:25:56,024::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911035::DEBUG::2013-11-19 21:25:56,025::task::495::TaskManager.Task::(_debug) Task 6a5c5a5f-6a45-4470-858e-e0caff9574d5: ref 0 aborting False >Thread-15911035::INFO::2013-11-19 21:25:56,025::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000296115875244', 'lastCheck': 1384896355.503885, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142002105713', 'lastCheck': 1384896355.2560771, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896350.400012, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000488996505737', 'lastCheck': 1384896350.5918469, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:26:00,412::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911041::INFO::2013-11-19 21:26:06,160::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911041::DEBUG::2013-11-19 21:26:06,161::task::495::TaskManager.Task::(_debug) Task be4e21fe-1b39-4996-b929-069a1e84e955: moving from state init -> state preparing >Thread-15911041::DEBUG::2013-11-19 21:26:06,162::task::495::TaskManager.Task::(_debug) Task be4e21fe-1b39-4996-b929-069a1e84e955: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000296115875244', 'lastCheck': 1384896365.5164969, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00206708908081', 'lastCheck': 1384896365.272527, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896360.413337, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000367879867554', 'lastCheck': 1384896360.602582, 'valid': True, 'code': 0}} >Thread-15911041::DEBUG::2013-11-19 21:26:06,163::task::495::TaskManager.Task::(_debug) Task be4e21fe-1b39-4996-b929-069a1e84e955: moving from state preparing -> state finished >Thread-15911041::DEBUG::2013-11-19 21:26:06,163::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911041::DEBUG::2013-11-19 21:26:06,164::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911041::DEBUG::2013-11-19 21:26:06,164::task::495::TaskManager.Task::(_debug) Task be4e21fe-1b39-4996-b929-069a1e84e955: ref 0 aborting False >Thread-15911041::INFO::2013-11-19 21:26:06,165::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000296115875244', 'lastCheck': 1384896365.5164969, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00206708908081', 'lastCheck': 1384896365.272527, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896360.413337, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000367879867554', 'lastCheck': 1384896360.602582, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:26:10,426::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911047::INFO::2013-11-19 21:26:16,243::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911047::DEBUG::2013-11-19 21:26:16,244::task::495::TaskManager.Task::(_debug) Task e95e7d70-5209-44d0-84db-48c9cd0c82b1: moving from state init -> state preparing >Thread-15911047::DEBUG::2013-11-19 21:26:16,244::task::495::TaskManager.Task::(_debug) Task e95e7d70-5209-44d0-84db-48c9cd0c82b1: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000295162200928', 'lastCheck': 1384896375.5289631, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150609016418', 'lastCheck': 1384896375.288049, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896370.426806, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000501155853271', 'lastCheck': 1384896370.6144941, 'valid': True, 'code': 0}} >Thread-15911047::DEBUG::2013-11-19 21:26:16,245::task::495::TaskManager.Task::(_debug) Task e95e7d70-5209-44d0-84db-48c9cd0c82b1: moving from state preparing -> state finished >Thread-15911047::DEBUG::2013-11-19 21:26:16,245::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911047::DEBUG::2013-11-19 21:26:16,246::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911047::DEBUG::2013-11-19 21:26:16,246::task::495::TaskManager.Task::(_debug) Task e95e7d70-5209-44d0-84db-48c9cd0c82b1: ref 0 aborting False >Thread-15911047::INFO::2013-11-19 21:26:16,247::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000295162200928', 'lastCheck': 1384896375.5289631, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150609016418', 'lastCheck': 1384896375.288049, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896370.426806, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000501155853271', 'lastCheck': 1384896370.6144941, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:26:20,439::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911053::INFO::2013-11-19 21:26:26,358::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911053::DEBUG::2013-11-19 21:26:26,358::task::495::TaskManager.Task::(_debug) Task 49a7c2bc-0d7f-4baf-8ce8-f9ccd2591e6a: moving from state init -> state preparing >Thread-15911053::DEBUG::2013-11-19 21:26:26,359::task::495::TaskManager.Task::(_debug) Task 49a7c2bc-0d7f-4baf-8ce8-f9ccd2591e6a: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000281095504761', 'lastCheck': 1384896385.5414951, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146007537842', 'lastCheck': 1384896385.303916, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896380.440177, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000383853912354', 'lastCheck': 1384896380.626647, 'valid': True, 'code': 0}} >Thread-15911053::DEBUG::2013-11-19 21:26:26,360::task::495::TaskManager.Task::(_debug) Task 49a7c2bc-0d7f-4baf-8ce8-f9ccd2591e6a: moving from state preparing -> state finished >Thread-15911053::DEBUG::2013-11-19 21:26:26,360::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911053::DEBUG::2013-11-19 21:26:26,361::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911053::DEBUG::2013-11-19 21:26:26,361::task::495::TaskManager.Task::(_debug) Task 49a7c2bc-0d7f-4baf-8ce8-f9ccd2591e6a: ref 0 aborting False >Thread-15911053::INFO::2013-11-19 21:26:26,362::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000281095504761', 'lastCheck': 1384896385.5414951, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146007537842', 'lastCheck': 1384896385.303916, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896380.440177, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000383853912354', 'lastCheck': 1384896380.626647, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:26:30,452::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911059::INFO::2013-11-19 21:26:36,573::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911059::DEBUG::2013-11-19 21:26:36,574::task::495::TaskManager.Task::(_debug) Task ba0094fa-0099-47bd-ba40-bc7e8e7ecb21: moving from state init -> state preparing >Thread-15911059::DEBUG::2013-11-19 21:26:36,575::task::495::TaskManager.Task::(_debug) Task ba0094fa-0099-47bd-ba40-bc7e8e7ecb21: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000355005264282', 'lastCheck': 1384896395.5538001, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00133395195007', 'lastCheck': 1384896395.319736, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896390.453069, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000386953353882', 'lastCheck': 1384896390.6383491, 'valid': True, 'code': 0}} >Thread-15911059::DEBUG::2013-11-19 21:26:36,576::task::495::TaskManager.Task::(_debug) Task ba0094fa-0099-47bd-ba40-bc7e8e7ecb21: moving from state preparing -> state finished >Thread-15911059::DEBUG::2013-11-19 21:26:36,577::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911059::DEBUG::2013-11-19 21:26:36,578::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911059::DEBUG::2013-11-19 21:26:36,579::task::495::TaskManager.Task::(_debug) Task ba0094fa-0099-47bd-ba40-bc7e8e7ecb21: ref 0 aborting False >Thread-15911059::INFO::2013-11-19 21:26:36,581::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000355005264282', 'lastCheck': 1384896395.5538001, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00133395195007', 'lastCheck': 1384896395.319736, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896390.453069, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000386953353882', 'lastCheck': 1384896390.6383491, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:26:40,465::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911065::INFO::2013-11-19 21:26:46,696::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911065::DEBUG::2013-11-19 21:26:46,697::task::495::TaskManager.Task::(_debug) Task 60af5da1-e709-4b33-b69b-19fa6b5af200: moving from state init -> state preparing >Thread-15911065::DEBUG::2013-11-19 21:26:46,698::task::495::TaskManager.Task::(_debug) Task 60af5da1-e709-4b33-b69b-19fa6b5af200: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000277042388916', 'lastCheck': 1384896405.566467, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00227808952332', 'lastCheck': 1384896405.3356709, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896400.46649, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00047492980957', 'lastCheck': 1384896400.6497071, 'valid': True, 'code': 0}} >Thread-15911065::DEBUG::2013-11-19 21:26:46,698::task::495::TaskManager.Task::(_debug) Task 60af5da1-e709-4b33-b69b-19fa6b5af200: moving from state preparing -> state finished >Thread-15911065::DEBUG::2013-11-19 21:26:46,699::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911065::DEBUG::2013-11-19 21:26:46,699::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911065::DEBUG::2013-11-19 21:26:46,700::task::495::TaskManager.Task::(_debug) Task 60af5da1-e709-4b33-b69b-19fa6b5af200: ref 0 aborting False >Thread-15911065::INFO::2013-11-19 21:26:46,701::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000277042388916', 'lastCheck': 1384896405.566467, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00227808952332', 'lastCheck': 1384896405.3356709, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896400.46649, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00047492980957', 'lastCheck': 1384896400.6497071, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:26:50,479::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911071::INFO::2013-11-19 21:26:56,909::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911071::DEBUG::2013-11-19 21:26:56,909::task::495::TaskManager.Task::(_debug) Task 8b429815-ffab-4f55-8917-2787d5102ed0: moving from state init -> state preparing >Thread-15911071::DEBUG::2013-11-19 21:26:56,910::task::495::TaskManager.Task::(_debug) Task 8b429815-ffab-4f55-8917-2787d5102ed0: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000290155410767', 'lastCheck': 1384896415.5789361, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142598152161', 'lastCheck': 1384896415.3514221, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896410.4797771, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000448942184448', 'lastCheck': 1384896410.6611559, 'valid': True, 'code': 0}} >Thread-15911071::DEBUG::2013-11-19 21:26:56,911::task::495::TaskManager.Task::(_debug) Task 8b429815-ffab-4f55-8917-2787d5102ed0: moving from state preparing -> state finished >Thread-15911071::DEBUG::2013-11-19 21:26:56,911::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911071::DEBUG::2013-11-19 21:26:56,912::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911071::DEBUG::2013-11-19 21:26:56,912::task::495::TaskManager.Task::(_debug) Task 8b429815-ffab-4f55-8917-2787d5102ed0: ref 0 aborting False >Thread-15911071::INFO::2013-11-19 21:26:56,913::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000290155410767', 'lastCheck': 1384896415.5789361, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142598152161', 'lastCheck': 1384896415.3514221, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896410.4797771, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000448942184448', 'lastCheck': 1384896410.6611559, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:27:00,492::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911077::INFO::2013-11-19 21:27:07,021::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911077::DEBUG::2013-11-19 21:27:07,021::task::495::TaskManager.Task::(_debug) Task 165c30d0-b19d-4716-bf84-00ade4ebc32f: moving from state init -> state preparing >Thread-15911077::DEBUG::2013-11-19 21:27:07,022::task::495::TaskManager.Task::(_debug) Task 165c30d0-b19d-4716-bf84-00ade4ebc32f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000279903411865', 'lastCheck': 1384896425.591224, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00145196914673', 'lastCheck': 1384896425.3669081, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896420.4930091, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000477075576782', 'lastCheck': 1384896420.672648, 'valid': True, 'code': 0}} >Thread-15911077::DEBUG::2013-11-19 21:27:07,023::task::495::TaskManager.Task::(_debug) Task 165c30d0-b19d-4716-bf84-00ade4ebc32f: moving from state preparing -> state finished >Thread-15911077::DEBUG::2013-11-19 21:27:07,023::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911077::DEBUG::2013-11-19 21:27:07,024::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911077::DEBUG::2013-11-19 21:27:07,024::task::495::TaskManager.Task::(_debug) Task 165c30d0-b19d-4716-bf84-00ade4ebc32f: ref 0 aborting False >Thread-15911077::INFO::2013-11-19 21:27:07,025::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000279903411865', 'lastCheck': 1384896425.591224, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00145196914673', 'lastCheck': 1384896425.3669081, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896420.4930091, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000477075576782', 'lastCheck': 1384896420.672648, 'valid': True, 'code': 0}} >Thread-15911079::DEBUG::2013-11-19 21:27:08,656::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911079::INFO::2013-11-19 21:27:08,657::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911079::DEBUG::2013-11-19 21:27:08,657::task::495::TaskManager.Task::(_debug) Task 39b2886b-616c-455c-9f88-f859600fe2e7: moving from state init -> state preparing >Thread-15911079::DEBUG::2013-11-19 21:27:08,658::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=Free spmLver=280 spmId=-1 >Thread-15911079::DEBUG::2013-11-19 21:27:08,659::task::495::TaskManager.Task::(_debug) Task 39b2886b-616c-455c-9f88-f859600fe2e7: finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 280}} >Thread-15911079::DEBUG::2013-11-19 21:27:08,659::task::495::TaskManager.Task::(_debug) Task 39b2886b-616c-455c-9f88-f859600fe2e7: moving from state preparing -> state finished >Thread-15911079::DEBUG::2013-11-19 21:27:08,660::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911079::DEBUG::2013-11-19 21:27:08,660::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911079::DEBUG::2013-11-19 21:27:08,661::task::495::TaskManager.Task::(_debug) Task 39b2886b-616c-455c-9f88-f859600fe2e7: ref 0 aborting False >Thread-15911079::INFO::2013-11-19 21:27:08,661::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 280}} >Thread-15911080::DEBUG::2013-11-19 21:27:08,691::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911080::INFO::2013-11-19 21:27:08,692::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: spmStart, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 prevID=-1 prevLVER=280 recoveryMode=0 scsiFencing=false maxHostID=250 domVersion=2) >Thread-15911080::DEBUG::2013-11-19 21:27:08,692::task::495::TaskManager.Task::(_debug) Task 3c078476-ceba-49ba-8c4f-cfc47f187488: moving from state init -> state preparing >Thread-15911080::DEBUG::2013-11-19 21:27:08,694::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e0040fbc-ff91-4bc3-826a-a8d60258cf99`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '330' at 'public_spmStart' >Thread-15911080::DEBUG::2013-11-19 21:27:08,694::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >Thread-15911080::DEBUG::2013-11-19 21:27:08,695::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >Thread-15911080::DEBUG::2013-11-19 21:27:08,695::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e0040fbc-ff91-4bc3-826a-a8d60258cf99`::Granted request >Thread-15911080::DEBUG::2013-11-19 21:27:08,696::task::495::TaskManager.Task::(_debug) Task 3c078476-ceba-49ba-8c4f-cfc47f187488: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (exclusive) >Thread-15911080::DEBUG::2013-11-19 21:27:08,697::task::495::TaskManager.Task::(_debug) Task 3c078476-ceba-49ba-8c4f-cfc47f187488: ref 1 aborting False >Thread-15911080::DEBUG::2013-11-19 21:27:08,697::task::495::TaskManager.Task::(_debug) Task 3c078476-ceba-49ba-8c4f-cfc47f187488: Prepare: 1 jobs exist, move to acquiring >Thread-15911080::DEBUG::2013-11-19 21:27:08,698::task::495::TaskManager.Task::(_debug) Task 3c078476-ceba-49ba-8c4f-cfc47f187488: moving from state preparing -> state acquiring >Thread-15911080::DEBUG::2013-11-19 21:27:08,698::task::495::TaskManager.Task::(_debug) Task 3c078476-ceba-49ba-8c4f-cfc47f187488: moving from state acquiring -> state queued >Thread-15911080::DEBUG::2013-11-19 21:27:08,699::taskManager::37::TaskManager::(_queueTask) queueing task: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911080::DEBUG::2013-11-19 21:27:08,700::taskManager::43::TaskManager::(_queueTask) task queued: 3c078476-ceba-49ba-8c4f-cfc47f187488 >21b4b46d-58c8-40ca-8f03-a8063e50f1b9::DEBUG::2013-11-19 21:27:08,700::threadPool::47::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-15911080::DEBUG::2013-11-19 21:27:08,700::task::1183::TaskManager.Task::(prepare) returning >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:08,701::threadPool::192::Misc.ThreadPool.WorkerThread::(run) Task: 3c078476-ceba-49ba-8c4f-cfc47f187488 running: <bound method Task.commit of <storage.task.Task instance at 0x7f487c12bdd0>> with: None >Thread-15911080::DEBUG::2013-11-19 21:27:08,702::task::495::TaskManager.Task::(_debug) Task 3c078476-ceba-49ba-8c4f-cfc47f187488: ref 0 aborting False >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:08,703::task::1195::TaskManager.Task::(commit) Commiting task: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911080::INFO::2013-11-19 21:27:08,704::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: spmStart, Return response: {'status': {'message': 'OK', 'code': 0}, 'uuid': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:08,704::task::495::TaskManager.Task::(_debug) Task 3c078476-ceba-49ba-8c4f-cfc47f187488: moving from state queued -> state running >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:08,706::task::907::TaskManager.Task::(_runJobs) Task.run: running job 0: spmStart: <bound method SPM.start of <storage.spm.SPM instance at 0x1bf9f38>> (args: ('c4e5889a-348d-11e2-83fe-9f72369eab30', -1, '280', 0, 'false', 250, 2) kwargs: {}) >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:08,706::task::300::TaskManager.Task::(run) Job.run: running spmStart: <bound method SPM.start of <storage.spm.SPM instance at 0x1bf9f38>> (args: ('c4e5889a-348d-11e2-83fe-9f72369eab30', -1, '280', 0, 'false', 250, 2) kwargs: {}) callback None >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:08,707::spm::433::Storage.SPM::(start) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:08,707::misc::1010::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:08,708::misc::1012::SamplingMethod::(__call__) Got in to sampling method >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:08,709::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:08,709::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:08,710::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:08,711::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:08,711::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:08,712::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:09,398::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:09,401::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:09,402::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=281', u'POOL_SPM_ID=-1', u'_SHA_CKSUM=52809f588a9aef1f1ae8cf2b75391d6f337f51aa'] >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:09,404::misc::1020::SamplingMethod::(__call__) Returning last result >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:09,405::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:09,405::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:09,406::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:09,406::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:09,407::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:09,408::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15911081::DEBUG::2013-11-19 21:27:09,713::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911081::INFO::2013-11-19 21:27:09,715::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911081::DEBUG::2013-11-19 21:27:09,715::task::495::TaskManager.Task::(_debug) Task ab0fff4e-6da3-4f90-93a9-296d0f2a869a: moving from state init -> state preparing >Thread-15911081::DEBUG::2013-11-19 21:27:09,716::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911081::DEBUG::2013-11-19 21:27:09,717::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911081::DEBUG::2013-11-19 21:27:09,717::task::495::TaskManager.Task::(_debug) Task ab0fff4e-6da3-4f90-93a9-296d0f2a869a: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911081::DEBUG::2013-11-19 21:27:09,718::task::495::TaskManager.Task::(_debug) Task ab0fff4e-6da3-4f90-93a9-296d0f2a869a: moving from state preparing -> state finished >Thread-15911081::DEBUG::2013-11-19 21:27:09,719::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911081::DEBUG::2013-11-19 21:27:09,719::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911081::DEBUG::2013-11-19 21:27:09,720::task::495::TaskManager.Task::(_debug) Task ab0fff4e-6da3-4f90-93a9-296d0f2a869a: ref 0 aborting False >Thread-15911081::INFO::2013-11-19 21:27:09,720::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:10,071::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:10,074::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:10,075::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=281', u'POOL_SPM_ID=-1', u'_SHA_CKSUM=52809f588a9aef1f1ae8cf2b75391d6f337f51aa'] >3c078476-ceba-49ba-8c4f-cfc47f187488::INFO::2013-11-19 21:27:10,077::spm::457::Storage.SPM::(start) expected previd:-1 lver:281 got request for previd:-1 lver:280 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:10,078::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:10,078::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:10,079::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:10,080::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:10,080::persistentDict::163::Storage.PersistentDict::(__init__) Created a persistant dict with LvMetadataRW backend >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:10,081::blockSD::164::Storage.Misc.excCmd::(readlines) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:10,476::blockSD::164::Storage.Misc.excCmd::(readlines) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000231298 s, 8.9 MB/s\n'; <rc> = 0 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:10,477::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000231298 s, 8.9 MB/s'], size: 2048 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:10,478::persistentDict::204::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >3c078476-ceba-49ba-8c4f-cfc47f187488::WARNING::2013-11-19 21:27:10,478::persistentDict::226::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:10,479::persistentDict::163::Storage.PersistentDict::(__init__) Created a persistant dict with VGTagMetadataRW backend >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:10,480::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:10,480::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:10,481::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:10,482::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:10,482::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:10,484::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42840::DEBUG::2013-11-19 21:27:10,686::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:27:10,687::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911082::DEBUG::2013-11-19 21:27:11,044::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911082::INFO::2013-11-19 21:27:11,046::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911082::DEBUG::2013-11-19 21:27:11,046::task::495::TaskManager.Task::(_debug) Task 7a90951f-c45e-403b-ba18-ccd649397564: moving from state init -> state preparing >Thread-15911082::DEBUG::2013-11-19 21:27:11,047::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911082::DEBUG::2013-11-19 21:27:11,047::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911082::DEBUG::2013-11-19 21:27:11,047::task::495::TaskManager.Task::(_debug) Task 7a90951f-c45e-403b-ba18-ccd649397564: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911082::DEBUG::2013-11-19 21:27:11,048::task::495::TaskManager.Task::(_debug) Task 7a90951f-c45e-403b-ba18-ccd649397564: moving from state preparing -> state finished >Thread-15911082::DEBUG::2013-11-19 21:27:11,048::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911082::DEBUG::2013-11-19 21:27:11,048::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911082::DEBUG::2013-11-19 21:27:11,049::task::495::TaskManager.Task::(_debug) Task 7a90951f-c45e-403b-ba18-ccd649397564: ref 0 aborting False >Thread-15911082::INFO::2013-11-19 21:27:11,049::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:11,246::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:11,249::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:11,250::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=281', u'POOL_SPM_ID=-1', u'_SHA_CKSUM=52809f588a9aef1f1ae8cf2b75391d6f337f51aa'] >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:11,252::safelease::52::ClusterLock::(acquire) Acquiring cluster lock for domain 6b892067-ac7e-4039-a7be-5b7387525b16 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:11,253::safelease::61::Storage.Misc.excCmd::(acquire) '/usr/bin/sudo -n /usr/bin/setsid /usr/bin/ionice -c1 -n0 /bin/su vdsm -s /bin/sh -c "/usr/libexec/vdsm/spmprotect.sh start 6b892067-ac7e-4039-a7be-5b7387525b16 2 5 /dev/6b892067-ac7e-4039-a7be-5b7387525b16/leases 60000 10000 3"' (cwd /usr/libexec/vdsm/) >Thread-42840::DEBUG::2013-11-19 21:27:11,466::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911083::DEBUG::2013-11-19 21:27:12,056::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911083::INFO::2013-11-19 21:27:12,058::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911083::DEBUG::2013-11-19 21:27:12,058::task::495::TaskManager.Task::(_debug) Task a6241060-9310-442a-90c3-5c6e778f6cdf: moving from state init -> state preparing >Thread-15911083::DEBUG::2013-11-19 21:27:12,059::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911083::DEBUG::2013-11-19 21:27:12,059::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911083::DEBUG::2013-11-19 21:27:12,060::task::495::TaskManager.Task::(_debug) Task a6241060-9310-442a-90c3-5c6e778f6cdf: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911083::DEBUG::2013-11-19 21:27:12,061::task::495::TaskManager.Task::(_debug) Task a6241060-9310-442a-90c3-5c6e778f6cdf: moving from state preparing -> state finished >Thread-15911083::DEBUG::2013-11-19 21:27:12,061::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911083::DEBUG::2013-11-19 21:27:12,062::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911083::DEBUG::2013-11-19 21:27:12,062::task::495::TaskManager.Task::(_debug) Task a6241060-9310-442a-90c3-5c6e778f6cdf: ref 0 aborting False >Thread-15911083::INFO::2013-11-19 21:27:12,063::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911084::DEBUG::2013-11-19 21:27:13,069::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911084::INFO::2013-11-19 21:27:13,070::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911084::DEBUG::2013-11-19 21:27:13,070::task::495::TaskManager.Task::(_debug) Task 039ed82c-1b8c-4acd-b5f9-cfadd5deb8a4: moving from state init -> state preparing >Thread-15911084::DEBUG::2013-11-19 21:27:13,071::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911084::DEBUG::2013-11-19 21:27:13,071::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911084::DEBUG::2013-11-19 21:27:13,072::task::495::TaskManager.Task::(_debug) Task 039ed82c-1b8c-4acd-b5f9-cfadd5deb8a4: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911084::DEBUG::2013-11-19 21:27:13,072::task::495::TaskManager.Task::(_debug) Task 039ed82c-1b8c-4acd-b5f9-cfadd5deb8a4: moving from state preparing -> state finished >Thread-15911084::DEBUG::2013-11-19 21:27:13,073::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911084::DEBUG::2013-11-19 21:27:13,073::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911084::DEBUG::2013-11-19 21:27:13,074::task::495::TaskManager.Task::(_debug) Task 039ed82c-1b8c-4acd-b5f9-cfadd5deb8a4: ref 0 aborting False >Thread-15911084::INFO::2013-11-19 21:27:13,075::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911085::DEBUG::2013-11-19 21:27:14,080::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911085::INFO::2013-11-19 21:27:14,081::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911085::DEBUG::2013-11-19 21:27:14,082::task::495::TaskManager.Task::(_debug) Task 05a308c5-a979-441f-990f-2a7cedf85ccd: moving from state init -> state preparing >Thread-15911085::DEBUG::2013-11-19 21:27:14,082::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911085::DEBUG::2013-11-19 21:27:14,083::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911085::DEBUG::2013-11-19 21:27:14,083::task::495::TaskManager.Task::(_debug) Task 05a308c5-a979-441f-990f-2a7cedf85ccd: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911085::DEBUG::2013-11-19 21:27:14,084::task::495::TaskManager.Task::(_debug) Task 05a308c5-a979-441f-990f-2a7cedf85ccd: moving from state preparing -> state finished >Thread-15911085::DEBUG::2013-11-19 21:27:14,084::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911085::DEBUG::2013-11-19 21:27:14,085::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911085::DEBUG::2013-11-19 21:27:14,085::task::495::TaskManager.Task::(_debug) Task 05a308c5-a979-441f-990f-2a7cedf85ccd: ref 0 aborting False >Thread-15911085::INFO::2013-11-19 21:27:14,086::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911086::DEBUG::2013-11-19 21:27:15,092::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911086::INFO::2013-11-19 21:27:15,093::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911086::DEBUG::2013-11-19 21:27:15,093::task::495::TaskManager.Task::(_debug) Task 2e93a975-b7cd-4192-a643-c031252f361b: moving from state init -> state preparing >Thread-15911086::DEBUG::2013-11-19 21:27:15,094::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911086::DEBUG::2013-11-19 21:27:15,094::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911086::DEBUG::2013-11-19 21:27:15,095::task::495::TaskManager.Task::(_debug) Task 2e93a975-b7cd-4192-a643-c031252f361b: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911086::DEBUG::2013-11-19 21:27:15,095::task::495::TaskManager.Task::(_debug) Task 2e93a975-b7cd-4192-a643-c031252f361b: moving from state preparing -> state finished >Thread-15911086::DEBUG::2013-11-19 21:27:15,096::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911086::DEBUG::2013-11-19 21:27:15,096::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911086::DEBUG::2013-11-19 21:27:15,097::task::495::TaskManager.Task::(_debug) Task 2e93a975-b7cd-4192-a643-c031252f361b: ref 0 aborting False >Thread-15911086::INFO::2013-11-19 21:27:15,098::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911087::DEBUG::2013-11-19 21:27:16,103::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911087::INFO::2013-11-19 21:27:16,104::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911087::DEBUG::2013-11-19 21:27:16,105::task::495::TaskManager.Task::(_debug) Task f84929d7-0215-4ea5-848f-b5bb33bd30a4: moving from state init -> state preparing >Thread-15911087::DEBUG::2013-11-19 21:27:16,105::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911087::DEBUG::2013-11-19 21:27:16,106::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911087::DEBUG::2013-11-19 21:27:16,106::task::495::TaskManager.Task::(_debug) Task f84929d7-0215-4ea5-848f-b5bb33bd30a4: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911087::DEBUG::2013-11-19 21:27:16,107::task::495::TaskManager.Task::(_debug) Task f84929d7-0215-4ea5-848f-b5bb33bd30a4: moving from state preparing -> state finished >Thread-15911087::DEBUG::2013-11-19 21:27:16,107::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911087::DEBUG::2013-11-19 21:27:16,108::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911087::DEBUG::2013-11-19 21:27:16,108::task::495::TaskManager.Task::(_debug) Task f84929d7-0215-4ea5-848f-b5bb33bd30a4: ref 0 aborting False >Thread-15911087::INFO::2013-11-19 21:27:16,109::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911088::DEBUG::2013-11-19 21:27:17,114::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911088::INFO::2013-11-19 21:27:17,115::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911088::DEBUG::2013-11-19 21:27:17,116::task::495::TaskManager.Task::(_debug) Task dfbed9f2-8c7c-4642-a516-6d4748cc7bd5: moving from state init -> state preparing >Thread-15911088::DEBUG::2013-11-19 21:27:17,116::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911088::DEBUG::2013-11-19 21:27:17,117::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911088::DEBUG::2013-11-19 21:27:17,117::task::495::TaskManager.Task::(_debug) Task dfbed9f2-8c7c-4642-a516-6d4748cc7bd5: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911088::DEBUG::2013-11-19 21:27:17,118::task::495::TaskManager.Task::(_debug) Task dfbed9f2-8c7c-4642-a516-6d4748cc7bd5: moving from state preparing -> state finished >Thread-15911088::DEBUG::2013-11-19 21:27:17,118::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911088::DEBUG::2013-11-19 21:27:17,119::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911088::DEBUG::2013-11-19 21:27:17,119::task::495::TaskManager.Task::(_debug) Task dfbed9f2-8c7c-4642-a516-6d4748cc7bd5: ref 0 aborting False >Thread-15911088::INFO::2013-11-19 21:27:17,120::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911089::DEBUG::2013-11-19 21:27:18,126::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911089::INFO::2013-11-19 21:27:18,127::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911089::DEBUG::2013-11-19 21:27:18,127::task::495::TaskManager.Task::(_debug) Task a1f9cc6f-636c-4679-8340-f92c7ceb48cc: moving from state init -> state preparing >Thread-15911089::DEBUG::2013-11-19 21:27:18,128::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911089::DEBUG::2013-11-19 21:27:18,128::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911089::DEBUG::2013-11-19 21:27:18,129::task::495::TaskManager.Task::(_debug) Task a1f9cc6f-636c-4679-8340-f92c7ceb48cc: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911089::DEBUG::2013-11-19 21:27:18,130::task::495::TaskManager.Task::(_debug) Task a1f9cc6f-636c-4679-8340-f92c7ceb48cc: moving from state preparing -> state finished >Thread-15911089::DEBUG::2013-11-19 21:27:18,131::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911089::DEBUG::2013-11-19 21:27:18,131::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911089::DEBUG::2013-11-19 21:27:18,132::task::495::TaskManager.Task::(_debug) Task a1f9cc6f-636c-4679-8340-f92c7ceb48cc: ref 0 aborting False >Thread-15911089::INFO::2013-11-19 21:27:18,133::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911090::DEBUG::2013-11-19 21:27:19,138::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911090::INFO::2013-11-19 21:27:19,139::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911090::DEBUG::2013-11-19 21:27:19,140::task::495::TaskManager.Task::(_debug) Task c9555e53-0709-4811-a547-dd8a8ebb6885: moving from state init -> state preparing >Thread-15911090::DEBUG::2013-11-19 21:27:19,140::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911090::DEBUG::2013-11-19 21:27:19,141::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911090::DEBUG::2013-11-19 21:27:19,141::task::495::TaskManager.Task::(_debug) Task c9555e53-0709-4811-a547-dd8a8ebb6885: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911090::DEBUG::2013-11-19 21:27:19,142::task::495::TaskManager.Task::(_debug) Task c9555e53-0709-4811-a547-dd8a8ebb6885: moving from state preparing -> state finished >Thread-15911090::DEBUG::2013-11-19 21:27:19,142::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911090::DEBUG::2013-11-19 21:27:19,143::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911090::DEBUG::2013-11-19 21:27:19,143::task::495::TaskManager.Task::(_debug) Task c9555e53-0709-4811-a547-dd8a8ebb6885: ref 0 aborting False >Thread-15911090::INFO::2013-11-19 21:27:19,144::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911091::DEBUG::2013-11-19 21:27:20,150::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911091::INFO::2013-11-19 21:27:20,150::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911091::DEBUG::2013-11-19 21:27:20,151::task::495::TaskManager.Task::(_debug) Task 953d93aa-bc07-492d-8edb-914b9923877a: moving from state init -> state preparing >Thread-15911091::DEBUG::2013-11-19 21:27:20,151::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911091::DEBUG::2013-11-19 21:27:20,152::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911091::DEBUG::2013-11-19 21:27:20,153::task::495::TaskManager.Task::(_debug) Task 953d93aa-bc07-492d-8edb-914b9923877a: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911091::DEBUG::2013-11-19 21:27:20,153::task::495::TaskManager.Task::(_debug) Task 953d93aa-bc07-492d-8edb-914b9923877a: moving from state preparing -> state finished >Thread-15911091::DEBUG::2013-11-19 21:27:20,154::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911091::DEBUG::2013-11-19 21:27:20,154::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911091::DEBUG::2013-11-19 21:27:20,155::task::495::TaskManager.Task::(_debug) Task 953d93aa-bc07-492d-8edb-914b9923877a: ref 0 aborting False >Thread-15911091::INFO::2013-11-19 21:27:20,155::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-1024767::ERROR::2013-11-19 21:27:20,702::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911092::DEBUG::2013-11-19 21:27:21,161::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911092::INFO::2013-11-19 21:27:21,163::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911092::DEBUG::2013-11-19 21:27:21,163::task::495::TaskManager.Task::(_debug) Task 75d82b22-5362-4fe0-b810-39b53fbebb78: moving from state init -> state preparing >Thread-15911092::DEBUG::2013-11-19 21:27:21,164::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911092::DEBUG::2013-11-19 21:27:21,164::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911092::DEBUG::2013-11-19 21:27:21,165::task::495::TaskManager.Task::(_debug) Task 75d82b22-5362-4fe0-b810-39b53fbebb78: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911092::DEBUG::2013-11-19 21:27:21,166::task::495::TaskManager.Task::(_debug) Task 75d82b22-5362-4fe0-b810-39b53fbebb78: moving from state preparing -> state finished >Thread-15911092::DEBUG::2013-11-19 21:27:21,166::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911092::DEBUG::2013-11-19 21:27:21,167::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911092::DEBUG::2013-11-19 21:27:21,167::task::495::TaskManager.Task::(_debug) Task 75d82b22-5362-4fe0-b810-39b53fbebb78: ref 0 aborting False >Thread-15911092::INFO::2013-11-19 21:27:21,168::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911093::DEBUG::2013-11-19 21:27:22,174::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911093::INFO::2013-11-19 21:27:22,174::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911093::DEBUG::2013-11-19 21:27:22,175::task::495::TaskManager.Task::(_debug) Task 532ffec1-75cc-4714-88de-aa16d29258d0: moving from state init -> state preparing >Thread-15911093::DEBUG::2013-11-19 21:27:22,175::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911093::DEBUG::2013-11-19 21:27:22,176::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911093::DEBUG::2013-11-19 21:27:22,177::task::495::TaskManager.Task::(_debug) Task 532ffec1-75cc-4714-88de-aa16d29258d0: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911093::DEBUG::2013-11-19 21:27:22,177::task::495::TaskManager.Task::(_debug) Task 532ffec1-75cc-4714-88de-aa16d29258d0: moving from state preparing -> state finished >Thread-15911093::DEBUG::2013-11-19 21:27:22,178::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911093::DEBUG::2013-11-19 21:27:22,178::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911093::DEBUG::2013-11-19 21:27:22,179::task::495::TaskManager.Task::(_debug) Task 532ffec1-75cc-4714-88de-aa16d29258d0: ref 0 aborting False >Thread-15911093::INFO::2013-11-19 21:27:22,179::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911094::DEBUG::2013-11-19 21:27:23,185::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911094::INFO::2013-11-19 21:27:23,186::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911094::DEBUG::2013-11-19 21:27:23,186::task::495::TaskManager.Task::(_debug) Task fed80f6f-3740-49c6-9f48-32b6846d6401: moving from state init -> state preparing >Thread-15911094::DEBUG::2013-11-19 21:27:23,187::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911094::DEBUG::2013-11-19 21:27:23,187::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911094::DEBUG::2013-11-19 21:27:23,188::task::495::TaskManager.Task::(_debug) Task fed80f6f-3740-49c6-9f48-32b6846d6401: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911094::DEBUG::2013-11-19 21:27:23,188::task::495::TaskManager.Task::(_debug) Task fed80f6f-3740-49c6-9f48-32b6846d6401: moving from state preparing -> state finished >Thread-15911094::DEBUG::2013-11-19 21:27:23,189::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911094::DEBUG::2013-11-19 21:27:23,189::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911094::DEBUG::2013-11-19 21:27:23,190::task::495::TaskManager.Task::(_debug) Task fed80f6f-3740-49c6-9f48-32b6846d6401: ref 0 aborting False >Thread-15911094::INFO::2013-11-19 21:27:23,190::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911095::DEBUG::2013-11-19 21:27:24,196::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911095::INFO::2013-11-19 21:27:24,197::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911095::DEBUG::2013-11-19 21:27:24,197::task::495::TaskManager.Task::(_debug) Task ced09e13-e1c5-41a6-9b59-dd9f059963f9: moving from state init -> state preparing >Thread-15911095::DEBUG::2013-11-19 21:27:24,198::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911095::DEBUG::2013-11-19 21:27:24,198::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911095::DEBUG::2013-11-19 21:27:24,199::task::495::TaskManager.Task::(_debug) Task ced09e13-e1c5-41a6-9b59-dd9f059963f9: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911095::DEBUG::2013-11-19 21:27:24,200::task::495::TaskManager.Task::(_debug) Task ced09e13-e1c5-41a6-9b59-dd9f059963f9: moving from state preparing -> state finished >Thread-15911095::DEBUG::2013-11-19 21:27:24,200::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911095::DEBUG::2013-11-19 21:27:24,201::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911095::DEBUG::2013-11-19 21:27:24,201::task::495::TaskManager.Task::(_debug) Task ced09e13-e1c5-41a6-9b59-dd9f059963f9: ref 0 aborting False >Thread-15911095::INFO::2013-11-19 21:27:24,202::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911096::DEBUG::2013-11-19 21:27:25,207::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911096::INFO::2013-11-19 21:27:25,208::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911096::DEBUG::2013-11-19 21:27:25,217::task::495::TaskManager.Task::(_debug) Task 2d893693-686d-4780-a573-9d6145122ec8: moving from state init -> state preparing >Thread-15911096::DEBUG::2013-11-19 21:27:25,217::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911096::DEBUG::2013-11-19 21:27:25,218::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911096::DEBUG::2013-11-19 21:27:25,218::task::495::TaskManager.Task::(_debug) Task 2d893693-686d-4780-a573-9d6145122ec8: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911096::DEBUG::2013-11-19 21:27:25,219::task::495::TaskManager.Task::(_debug) Task 2d893693-686d-4780-a573-9d6145122ec8: moving from state preparing -> state finished >Thread-15911096::DEBUG::2013-11-19 21:27:25,219::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911096::DEBUG::2013-11-19 21:27:25,220::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911096::DEBUG::2013-11-19 21:27:25,220::task::495::TaskManager.Task::(_debug) Task 2d893693-686d-4780-a573-9d6145122ec8: ref 0 aborting False >Thread-15911096::INFO::2013-11-19 21:27:25,221::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911097::DEBUG::2013-11-19 21:27:26,226::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911097::INFO::2013-11-19 21:27:26,227::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911097::DEBUG::2013-11-19 21:27:26,228::task::495::TaskManager.Task::(_debug) Task 8d08fd7f-84f8-4058-a655-ce3bc8fc91e2: moving from state init -> state preparing >Thread-15911097::DEBUG::2013-11-19 21:27:26,228::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911097::DEBUG::2013-11-19 21:27:26,229::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911097::DEBUG::2013-11-19 21:27:26,229::task::495::TaskManager.Task::(_debug) Task 8d08fd7f-84f8-4058-a655-ce3bc8fc91e2: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911097::DEBUG::2013-11-19 21:27:26,230::task::495::TaskManager.Task::(_debug) Task 8d08fd7f-84f8-4058-a655-ce3bc8fc91e2: moving from state preparing -> state finished >Thread-15911097::DEBUG::2013-11-19 21:27:26,231::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911097::DEBUG::2013-11-19 21:27:26,231::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911097::DEBUG::2013-11-19 21:27:26,232::task::495::TaskManager.Task::(_debug) Task 8d08fd7f-84f8-4058-a655-ce3bc8fc91e2: ref 0 aborting False >Thread-15911097::INFO::2013-11-19 21:27:26,232::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911098::DEBUG::2013-11-19 21:27:27,238::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911098::INFO::2013-11-19 21:27:27,238::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911098::DEBUG::2013-11-19 21:27:27,239::task::495::TaskManager.Task::(_debug) Task f1bc18e5-e890-4a66-9f42-98fa552f29bd: moving from state init -> state preparing >Thread-15911098::DEBUG::2013-11-19 21:27:27,240::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911098::DEBUG::2013-11-19 21:27:27,240::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911098::DEBUG::2013-11-19 21:27:27,241::task::495::TaskManager.Task::(_debug) Task f1bc18e5-e890-4a66-9f42-98fa552f29bd: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911098::DEBUG::2013-11-19 21:27:27,241::task::495::TaskManager.Task::(_debug) Task f1bc18e5-e890-4a66-9f42-98fa552f29bd: moving from state preparing -> state finished >Thread-15911098::DEBUG::2013-11-19 21:27:27,242::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911098::DEBUG::2013-11-19 21:27:27,242::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911098::DEBUG::2013-11-19 21:27:27,243::task::495::TaskManager.Task::(_debug) Task f1bc18e5-e890-4a66-9f42-98fa552f29bd: ref 0 aborting False >Thread-15911098::INFO::2013-11-19 21:27:27,243::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911099::DEBUG::2013-11-19 21:27:28,249::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911099::INFO::2013-11-19 21:27:28,250::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911099::DEBUG::2013-11-19 21:27:28,251::task::495::TaskManager.Task::(_debug) Task 801c9fdd-86f9-4882-a7f9-7267f2a1233a: moving from state init -> state preparing >Thread-15911099::DEBUG::2013-11-19 21:27:28,251::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911099::DEBUG::2013-11-19 21:27:28,252::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911099::DEBUG::2013-11-19 21:27:28,252::task::495::TaskManager.Task::(_debug) Task 801c9fdd-86f9-4882-a7f9-7267f2a1233a: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911099::DEBUG::2013-11-19 21:27:28,253::task::495::TaskManager.Task::(_debug) Task 801c9fdd-86f9-4882-a7f9-7267f2a1233a: moving from state preparing -> state finished >Thread-15911099::DEBUG::2013-11-19 21:27:28,253::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911099::DEBUG::2013-11-19 21:27:28,254::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911099::DEBUG::2013-11-19 21:27:28,254::task::495::TaskManager.Task::(_debug) Task 801c9fdd-86f9-4882-a7f9-7267f2a1233a: ref 0 aborting False >Thread-15911099::INFO::2013-11-19 21:27:28,255::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911100::DEBUG::2013-11-19 21:27:29,261::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911100::INFO::2013-11-19 21:27:29,262::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911100::DEBUG::2013-11-19 21:27:29,262::task::495::TaskManager.Task::(_debug) Task 036550bd-c864-4d2b-9feb-516c072d876a: moving from state init -> state preparing >Thread-15911100::DEBUG::2013-11-19 21:27:29,263::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911100::DEBUG::2013-11-19 21:27:29,263::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911100::DEBUG::2013-11-19 21:27:29,264::task::495::TaskManager.Task::(_debug) Task 036550bd-c864-4d2b-9feb-516c072d876a: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911100::DEBUG::2013-11-19 21:27:29,265::task::495::TaskManager.Task::(_debug) Task 036550bd-c864-4d2b-9feb-516c072d876a: moving from state preparing -> state finished >Thread-15911100::DEBUG::2013-11-19 21:27:29,265::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911100::DEBUG::2013-11-19 21:27:29,266::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911100::DEBUG::2013-11-19 21:27:29,266::task::495::TaskManager.Task::(_debug) Task 036550bd-c864-4d2b-9feb-516c072d876a: ref 0 aborting False >Thread-15911100::INFO::2013-11-19 21:27:29,267::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911101::DEBUG::2013-11-19 21:27:30,272::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911101::INFO::2013-11-19 21:27:30,273::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911101::DEBUG::2013-11-19 21:27:30,280::task::495::TaskManager.Task::(_debug) Task 5f4c2174-0019-45f1-849a-e6159da182a7: moving from state init -> state preparing >Thread-15911101::DEBUG::2013-11-19 21:27:30,280::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911101::DEBUG::2013-11-19 21:27:30,281::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911101::DEBUG::2013-11-19 21:27:30,281::task::495::TaskManager.Task::(_debug) Task 5f4c2174-0019-45f1-849a-e6159da182a7: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911101::DEBUG::2013-11-19 21:27:30,282::task::495::TaskManager.Task::(_debug) Task 5f4c2174-0019-45f1-849a-e6159da182a7: moving from state preparing -> state finished >Thread-15911101::DEBUG::2013-11-19 21:27:30,283::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911101::DEBUG::2013-11-19 21:27:30,283::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911101::DEBUG::2013-11-19 21:27:30,284::task::495::TaskManager.Task::(_debug) Task 5f4c2174-0019-45f1-849a-e6159da182a7: ref 0 aborting False >Thread-15911101::INFO::2013-11-19 21:27:30,284::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-1024767::ERROR::2013-11-19 21:27:30,716::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911102::DEBUG::2013-11-19 21:27:31,290::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911102::INFO::2013-11-19 21:27:31,291::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911102::DEBUG::2013-11-19 21:27:31,291::task::495::TaskManager.Task::(_debug) Task 8f091748-5407-4968-8751-98d23afbd6f6: moving from state init -> state preparing >Thread-15911102::DEBUG::2013-11-19 21:27:31,292::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911102::DEBUG::2013-11-19 21:27:31,292::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911102::DEBUG::2013-11-19 21:27:31,293::task::495::TaskManager.Task::(_debug) Task 8f091748-5407-4968-8751-98d23afbd6f6: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911102::DEBUG::2013-11-19 21:27:31,293::task::495::TaskManager.Task::(_debug) Task 8f091748-5407-4968-8751-98d23afbd6f6: moving from state preparing -> state finished >Thread-15911102::DEBUG::2013-11-19 21:27:31,294::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911102::DEBUG::2013-11-19 21:27:31,294::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911102::DEBUG::2013-11-19 21:27:31,295::task::495::TaskManager.Task::(_debug) Task 8f091748-5407-4968-8751-98d23afbd6f6: ref 0 aborting False >Thread-15911102::INFO::2013-11-19 21:27:31,295::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:31,654::safelease::61::Storage.Misc.excCmd::(acquire) SUCCESS: <err> = ''; <rc> = 0 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:31,655::safelease::65::ClusterLock::(acquire) Clustered lock acquired successfully >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:31,655::spm::468::Storage.SPM::(start) spm lock acquired successfully >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:31,656::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:31,657::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:31,657::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:31,658::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:31,659::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:31,660::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:32,293::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:32,300::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:32,302::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=281', u'POOL_SPM_ID=-1', u'_SHA_CKSUM=52809f588a9aef1f1ae8cf2b75391d6f337f51aa'] >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:32,304::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >Thread-15911103::DEBUG::2013-11-19 21:27:32,306::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:32,307::persistentDict::146::Storage.PersistentDict::(transaction) Flushing changes >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:32,308::persistentDict::265::Storage.PersistentDict::(flush) about to write lines (VGTagMetadataRW)=[u'CLASS=Data', u'DESCRIPTION=OS', u'IOOPTIMEOUTSEC=10', u'LEASERETRIES=3', u'LEASETIMESEC=60', u'LOCKPOLICY=', u'LOCKRENEWALINTERVALSEC=5', u'LOGBLKSIZE=512', u'MASTER_VERSION=1', u'PHYBLKSIZE=512', u'POOL_DESCRIPTION=Default', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_ID=2', u'POOL_SPM_LVER=282', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'ROLE=Master', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'TYPE=FCP', u'VERSION=2', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', '_SHA_CKSUM=de56f40c15838d78cba198e8c045a60fe83f87c3'] >Thread-15911103::INFO::2013-11-19 21:27:32,309::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:32,310::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15911103::DEBUG::2013-11-19 21:27:32,310::task::495::TaskManager.Task::(_debug) Task 5d2cc08e-eff7-4bd1-83c9-4480c598674a: moving from state init -> state preparing >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:32,311::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15911103::DEBUG::2013-11-19 21:27:32,312::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:32,313::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15911103::DEBUG::2013-11-19 21:27:32,313::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:32,314::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15911103::DEBUG::2013-11-19 21:27:32,315::task::495::TaskManager.Task::(_debug) Task 5d2cc08e-eff7-4bd1-83c9-4480c598674a: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:32,316::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15911103::DEBUG::2013-11-19 21:27:32,316::task::495::TaskManager.Task::(_debug) Task 5d2cc08e-eff7-4bd1-83c9-4480c598674a: moving from state preparing -> state finished >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:32,317::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15911103::DEBUG::2013-11-19 21:27:32,318::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911103::DEBUG::2013-11-19 21:27:32,319::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911103::DEBUG::2013-11-19 21:27:32,548::task::495::TaskManager.Task::(_debug) Task 5d2cc08e-eff7-4bd1-83c9-4480c598674a: ref 0 aborting False >Thread-15911103::INFO::2013-11-19 21:27:32,552::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:32,968::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:32,982::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:32,983::blockSD::145::storage.Metadata.VGTagMetadataRW::(writelines) Updating metadata adding=MDT__SHA_CKSUM=de56f40c15838d78cba198e8c045a60fe83f87c3, MDT_POOL_SPM_LVER=282, MDT_POOL_SPM_ID=2 removing=MDT_POOL_SPM_ID=-1, MDT__SHA_CKSUM=52809f588a9aef1f1ae8cf2b75391d6f337f51aa, MDT_POOL_SPM_LVER=281 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:32,985::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --deltag MDT_POOL_SPM_LVER=281 --deltag MDT_POOL_SPM_ID=-1 --deltag MDT__SHA_CKSUM=52809f588a9aef1f1ae8cf2b75391d6f337f51aa --addtag MDT_POOL_SPM_LVER=282 --addtag MDT__SHA_CKSUM=de56f40c15838d78cba198e8c045a60fe83f87c3 --addtag MDT_POOL_SPM_ID=2 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15911104::DEBUG::2013-11-19 21:27:33,561::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911104::INFO::2013-11-19 21:27:33,562::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911104::DEBUG::2013-11-19 21:27:33,563::task::495::TaskManager.Task::(_debug) Task 96b0234c-e8c5-42d6-93ab-15a508d61288: moving from state init -> state preparing >Thread-15911104::DEBUG::2013-11-19 21:27:33,564::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911104::DEBUG::2013-11-19 21:27:33,564::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911104::DEBUG::2013-11-19 21:27:33,565::task::495::TaskManager.Task::(_debug) Task 96b0234c-e8c5-42d6-93ab-15a508d61288: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911104::DEBUG::2013-11-19 21:27:33,565::task::495::TaskManager.Task::(_debug) Task 96b0234c-e8c5-42d6-93ab-15a508d61288: moving from state preparing -> state finished >Thread-15911104::DEBUG::2013-11-19 21:27:33,566::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911104::DEBUG::2013-11-19 21:27:33,567::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911104::DEBUG::2013-11-19 21:27:33,567::task::495::TaskManager.Task::(_debug) Task 96b0234c-e8c5-42d6-93ab-15a508d61288: ref 0 aborting False >Thread-15911104::INFO::2013-11-19 21:27:33,568::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:33,618::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:33,619::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:33,620::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:33,620::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:33,621::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d70ce20c-ee3a-43fe-8e64-cebbf35fc887`::Request was made in '/usr/share/vdsm/storage/spm.py' line '330' at '_upgradePool' >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:33,622::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:33,623::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:33,623::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d70ce20c-ee3a-43fe-8e64-cebbf35fc887`::Granted request >3c078476-ceba-49ba-8c4f-cfc47f187488::INFO::2013-11-19 21:27:33,624::spm::336::Storage.SPM::(_upgradePool) Trying to upgrade master domain `6b892067-ac7e-4039-a7be-5b7387525b16` >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:33,625::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`8093ac85-edeb-4915-9436-da96688a5090`::Request was made in '/usr/share/vdsm/storage/spm.py' line '337' at '_upgradePool' >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:33,626::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'exclusive' >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:33,626::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'exclusive' (1 active user) >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:33,627::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`8093ac85-edeb-4915-9436-da96688a5090`::Granted request >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:33,628::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:33,629::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:33,629::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:33,630::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:33,630::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:33,632::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:34,280::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:34,283::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:34,284::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=282', u'_SHA_CKSUM=de56f40c15838d78cba198e8c045a60fe83f87c3', u'POOL_SPM_ID=2'] >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:34,285::blockSD::625::Storage.StorageDomain::(upgrade) Trying to upgrade domain `6b892067-ac7e-4039-a7be-5b7387525b16` from version 2 to version 2 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:34,286::blockSD::631::Storage.StorageDomain::(upgrade) No need to upgrade domain `6b892067-ac7e-4039-a7be-5b7387525b16`, leaving unchanged >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:34,287::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:34,287::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:34,288::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:34,288::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:34,289::spm::340::Storage.SPM::(_upgradePool) Marking all domains for upgrade >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:34,290::spm::347::Storage.SPM::(_upgradePool) Registering with state change event >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:34,290::spm::349::Storage.SPM::(_upgradePool) Running initial domain upgrade threads >Thread-15911105::DEBUG::2013-11-19 21:27:34,291::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 45cdff8b-89c0-4926-a8c7-80354064bfa6 >Thread-15911107::DEBUG::2013-11-19 21:27:34,293::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 5d7b5178-3790-472c-a13b-b481eb27a61e >Thread-15911106::DEBUG::2013-11-19 21:27:34,294::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 772472cb-2429-466d-96cd-00b3301ef78a >Thread-15911107::ERROR::2013-11-19 21:27:34,296::spm::373::Storage.SPM::(_upgradePoolDomain) Error while preparing domain `5d7b5178-3790-472c-a13b-b481eb27a61e` upgrade >Traceback (most recent call last): > File "/usr/share/vdsm/storage/spm.py", line 369, in _upgradePoolDomain >IndexError: list index out of range >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:34,296::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911105::DEBUG::2013-11-19 21:27:34,298::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`d6b8afd5-fcdf-43b2-b88a-b1e755f3435d`::Request was made in '/usr/share/vdsm/storage/spm.py' line '376' at '_upgradePoolDomain' >Thread-15911106::DEBUG::2013-11-19 21:27:34,299::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`a3eb70c5-f6c4-43b6-9376-0207e30b5eeb`::Request was made in '/usr/share/vdsm/storage/spm.py' line '376' at '_upgradePoolDomain' >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:34,300::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911105::DEBUG::2013-11-19 21:27:34,301::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' for lock type 'exclusive' >Thread-15911106::DEBUG::2013-11-19 21:27:34,302::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'exclusive' >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:34,302::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:34,304::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911105::DEBUG::2013-11-19 21:27:34,305::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' is free. Now locking as 'exclusive' (1 active user) >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:34,306::blockSD::791::Storage.Misc.excCmd::(mountMaster) '/usr/bin/sudo -n /sbin/e2fsck -p /dev/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15911105::DEBUG::2013-11-19 21:27:34,306::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`d6b8afd5-fcdf-43b2-b88a-b1e755f3435d`::Granted request >Thread-15911106::DEBUG::2013-11-19 21:27:34,308::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'exclusive' (1 active user) >Thread-15911106::DEBUG::2013-11-19 21:27:34,520::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`a3eb70c5-f6c4-43b6-9376-0207e30b5eeb`::Granted request >Thread-15911105::DEBUG::2013-11-19 21:27:34,525::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`536b7109-8f80-44f2-83a5-c58b6fefa2e7`::Request was made in '/usr/share/vdsm/storage/spm.py' line '377' at '_upgradePoolDomain' >Thread-15911106::DEBUG::2013-11-19 21:27:34,525::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`d2e38165-3a6b-4ca1-b044-fa09fbe056c1`::Request was made in '/usr/share/vdsm/storage/spm.py' line '377' at '_upgradePoolDomain' >Thread-15911105::DEBUG::2013-11-19 21:27:34,527::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' for lock type 'exclusive' >Thread-15911106::DEBUG::2013-11-19 21:27:34,527::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'exclusive' >Thread-15911105::DEBUG::2013-11-19 21:27:34,529::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' is free. Now locking as 'exclusive' (1 active user) >Thread-15911105::DEBUG::2013-11-19 21:27:34,530::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`536b7109-8f80-44f2-83a5-c58b6fefa2e7`::Granted request >Thread-15911106::DEBUG::2013-11-19 21:27:34,531::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'exclusive' (1 active user) >Thread-15911105::DEBUG::2013-11-19 21:27:34,533::spm::385::Storage.SPM::(_upgradePoolDomain) Domain `45cdff8b-89c0-4926-a8c7-80354064bfa6` is not a data domain it is an 2 domain, not upgrading >Thread-15911106::DEBUG::2013-11-19 21:27:34,534::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`d2e38165-3a6b-4ca1-b044-fa09fbe056c1`::Granted request >Thread-15911105::DEBUG::2013-11-19 21:27:34,535::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' >Thread-15911106::DEBUG::2013-11-19 21:27:34,537::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15911105::DEBUG::2013-11-19 21:27:34,537::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' (0 active users) >Thread-15911106::DEBUG::2013-11-19 21:27:34,539::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15911105::DEBUG::2013-11-19 21:27:34,539::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' is free, finding out if anyone is waiting for it. >Thread-15911106::DEBUG::2013-11-19 21:27:34,540::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15911105::DEBUG::2013-11-19 21:27:34,541::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6', Clearing records. >Thread-15911106::DEBUG::2013-11-19 21:27:34,542::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15911105::DEBUG::2013-11-19 21:27:34,543::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' >Thread-15911106::DEBUG::2013-11-19 21:27:34,544::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15911105::DEBUG::2013-11-19 21:27:34,545::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' (0 active users) >Thread-15911106::DEBUG::2013-11-19 21:27:34,547::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15911105::DEBUG::2013-11-19 21:27:34,547::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' is free, finding out if anyone is waiting for it. >Thread-15911105::DEBUG::2013-11-19 21:27:34,548::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6', Clearing records. >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:34,896::blockSD::791::Storage.Misc.excCmd::(mountMaster) SUCCESS: <err> = ''; <rc> = 0 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:34,897::blockSD::816::Storage.Misc.excCmd::(mountMaster) '/usr/bin/sudo -n /sbin/tune2fs -j /dev/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15911108::DEBUG::2013-11-19 21:27:34,899::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911108::INFO::2013-11-19 21:27:35,254::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911108::DEBUG::2013-11-19 21:27:35,256::task::495::TaskManager.Task::(_debug) Task 1e66ef65-7ef4-47c3-8b3e-4ceb9cfd384d: moving from state init -> state preparing >Thread-15911108::DEBUG::2013-11-19 21:27:35,257::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911108::DEBUG::2013-11-19 21:27:35,258::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911108::DEBUG::2013-11-19 21:27:35,258::task::495::TaskManager.Task::(_debug) Task 1e66ef65-7ef4-47c3-8b3e-4ceb9cfd384d: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911108::DEBUG::2013-11-19 21:27:35,259::task::495::TaskManager.Task::(_debug) Task 1e66ef65-7ef4-47c3-8b3e-4ceb9cfd384d: moving from state preparing -> state finished >Thread-15911108::DEBUG::2013-11-19 21:27:35,260::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911108::DEBUG::2013-11-19 21:27:35,260::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911108::DEBUG::2013-11-19 21:27:35,261::task::495::TaskManager.Task::(_debug) Task 1e66ef65-7ef4-47c3-8b3e-4ceb9cfd384d: ref 0 aborting False >Thread-15911108::INFO::2013-11-19 21:27:35,262::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:35,455::blockSD::816::Storage.Misc.excCmd::(mountMaster) FAILED: <err> = 'The filesystem already has a journal.\n'; <rc> = 1 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:35,457::fileUtils::83::Storage.Misc.excCmd::(mount) '/usr/bin/sudo -n /bin/mount /dev/6b892067-ac7e-4039-a7be-5b7387525b16/master /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15911106::DEBUG::2013-11-19 21:27:35,691::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911106::DEBUG::2013-11-19 21:27:35,696::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15911106::DEBUG::2013-11-19 21:27:35,697::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'SDUUID=772472cb-2429-466d-96cd-00b3301ef78a', u'VERSION=2', u'LOCKPOLICY=', u'VGUUID=XTb294-NYv6-dVpO-bB1n-4f9P-d0GL-dhd2DI', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'LEASETIMESEC=5', u'LOCKRENEWALINTERVALSEC=5', u'ROLE=Regular', u'PHYBLKSIZE=512', u'DESCRIPTION=Database', u'IOOPTIMEOUTSEC=1', u'CLASS=Data', u'TYPE=FCP', u'PV0=pv:3600c0ff0001520c7414f605001000000,uuid:NujM33-vNiD-sBHY-mrPo-EQ2f-1qsz-HXfn0q,pestart:0,pecount:8929,mapoffset:0', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'MASTER_VERSION=0', u'_SHA_CKSUM=ef8840d9fdef24516a56fd970bed1960d2b1ae92'] >Thread-15911106::DEBUG::2013-11-19 21:27:35,699::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15911106::DEBUG::2013-11-19 21:27:35,700::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15911106::DEBUG::2013-11-19 21:27:35,700::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15911106::DEBUG::2013-11-19 21:27:35,701::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15911106::DEBUG::2013-11-19 21:27:35,702::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15911106::DEBUG::2013-11-19 21:27:35,702::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15911106::DEBUG::2013-11-19 21:27:35,703::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15911106::DEBUG::2013-11-19 21:27:35,704::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:35,878::fileUtils::83::Storage.Misc.excCmd::(mount) SUCCESS: <err> = ''; <rc> = 0 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:35,879::blockSD::833::Storage.Misc.excCmd::(mountMaster) '/usr/bin/sudo -n /bin/chown vdsm:kvm /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15911109::DEBUG::2013-11-19 21:27:36,269::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911109::INFO::2013-11-19 21:27:36,271::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911109::DEBUG::2013-11-19 21:27:36,271::task::495::TaskManager.Task::(_debug) Task 40c77a93-bebc-4b76-9f60-0fa7e7d2746c: moving from state init -> state preparing >Thread-15911109::DEBUG::2013-11-19 21:27:36,272::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911109::DEBUG::2013-11-19 21:27:36,272::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911109::DEBUG::2013-11-19 21:27:36,273::task::495::TaskManager.Task::(_debug) Task 40c77a93-bebc-4b76-9f60-0fa7e7d2746c: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911109::DEBUG::2013-11-19 21:27:36,274::task::495::TaskManager.Task::(_debug) Task 40c77a93-bebc-4b76-9f60-0fa7e7d2746c: moving from state preparing -> state finished >Thread-15911109::DEBUG::2013-11-19 21:27:36,274::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911109::DEBUG::2013-11-19 21:27:36,275::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:36,276::blockSD::833::Storage.Misc.excCmd::(mountMaster) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911109::DEBUG::2013-11-19 21:27:36,277::task::495::TaskManager.Task::(_debug) Task 40c77a93-bebc-4b76-9f60-0fa7e7d2746c: ref 0 aborting False >Thread-15911109::INFO::2013-11-19 21:27:36,279::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >3c078476-ceba-49ba-8c4f-cfc47f187488::WARNING::2013-11-19 21:27:36,279::spm::490::Storage.SPM::(start) Backup domain validation failed >Traceback (most recent call last): > File "/usr/share/vdsm/storage/spm.py", line 488, in start > File "/usr/share/vdsm/storage/sp.py", line 1208, in checkBackupDomain > File "/usr/share/vdsm/storage/sd.py", line 666, in isBackup > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:36,282::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:36,282::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:36,283::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:36,283::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:36,284::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:36,285::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:36,285::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:36,286::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:36,286::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:36,287::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:36,287::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:36,287::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:36,289::threadPool::25::Misc.ThreadPool::(__init__) Enter - numThreads: 5.0, waitTimeout: 3, maxTasks: 500.0 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:36,293::storage_mailbox::486::Storage.MailBox.SpmMailMonitor::(__init__) SPM_MailMonitor - clearing outgoing mail, command is: ['dd', 'of=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/outbox', 'oflag=direct', 'iflag=fullblock', 'conv=notrunc', 'count=1'] >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:36,294::storage_mailbox::488::Storage.Misc.excCmd::(__init__) 'dd of=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/outbox oflag=direct iflag=fullblock conv=notrunc count=1 bs=1024000' (cwd None) >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:36,752::storage_mailbox::488::Storage.Misc.excCmd::(__init__) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0409812 s, 25.0 MB/s\n'; <rc> = 0 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:36,753::storage_mailbox::493::Storage.MailBox.SpmMailMonitor::(__init__) SPM_MailMonitor created for pool c4e5889a-348d-11e2-83fe-9f72369eab30 >Dummy-15911115::DEBUG::2013-11-19 21:27:36,754::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:36,756::spm::523::Storage.SPM::(start) ended. >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:36,758::task::495::TaskManager.Task::(_debug) Task 3c078476-ceba-49ba-8c4f-cfc47f187488: moving from state running -> state finished >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:36,997::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:37,002::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:37,003::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:37,004::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:37,005::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:37,005::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:37,006::task::915::TaskManager.Task::(_runJobs) Task.run: exit - success: result >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:37,007::task::495::TaskManager.Task::(_debug) Task 3c078476-ceba-49ba-8c4f-cfc47f187488: ref 0 aborting False >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-19 21:27:37,007::threadPool::47::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 >Dummy-15911115::DEBUG::2013-11-19 21:27:37,273::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0726459 s, 14.1 MB/s\n'; <rc> = 0 >Thread-15911116::DEBUG::2013-11-19 21:27:37,311::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911116::INFO::2013-11-19 21:27:37,312::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911116::DEBUG::2013-11-19 21:27:37,323::task::495::TaskManager.Task::(_debug) Task 82a2b289-f4ed-4475-91fa-6d21eb4e1365: moving from state init -> state preparing >Thread-15911116::DEBUG::2013-11-19 21:27:37,323::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911116::DEBUG::2013-11-19 21:27:37,327::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': '1 jobs completed successfuly', 'taskState': 'finished', 'taskResult': 'success', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'} >Thread-15911116::DEBUG::2013-11-19 21:27:37,328::task::495::TaskManager.Task::(_debug) Task 82a2b289-f4ed-4475-91fa-6d21eb4e1365: finished: {'taskStatus': {'code': 0, 'message': '1 jobs completed successfuly', 'taskState': 'finished', 'taskResult': 'success', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911116::DEBUG::2013-11-19 21:27:37,337::task::495::TaskManager.Task::(_debug) Task 82a2b289-f4ed-4475-91fa-6d21eb4e1365: moving from state preparing -> state finished >Thread-15911116::DEBUG::2013-11-19 21:27:37,348::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911116::DEBUG::2013-11-19 21:27:37,351::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911116::DEBUG::2013-11-19 21:27:37,352::task::495::TaskManager.Task::(_debug) Task 82a2b289-f4ed-4475-91fa-6d21eb4e1365: ref 0 aborting False >Thread-15911116::INFO::2013-11-19 21:27:37,352::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': '1 jobs completed successfuly', 'taskState': 'finished', 'taskResult': 'success', 'taskID': '3c078476-ceba-49ba-8c4f-cfc47f187488'}} >Thread-15911117::DEBUG::2013-11-19 21:27:37,392::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911117::INFO::2013-11-19 21:27:37,396::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911117::DEBUG::2013-11-19 21:27:37,400::task::495::TaskManager.Task::(_debug) Task aac575fb-e9f8-40b9-90a7-d9cc469b549e: moving from state init -> state preparing >Thread-15911117::DEBUG::2013-11-19 21:27:37,412::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911117::DEBUG::2013-11-19 21:27:37,417::task::495::TaskManager.Task::(_debug) Task aac575fb-e9f8-40b9-90a7-d9cc469b549e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911117::DEBUG::2013-11-19 21:27:37,434::task::495::TaskManager.Task::(_debug) Task aac575fb-e9f8-40b9-90a7-d9cc469b549e: moving from state preparing -> state finished >Thread-15911117::DEBUG::2013-11-19 21:27:37,448::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911117::DEBUG::2013-11-19 21:27:37,457::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911117::DEBUG::2013-11-19 21:27:37,457::task::495::TaskManager.Task::(_debug) Task aac575fb-e9f8-40b9-90a7-d9cc469b549e: ref 0 aborting False >Thread-15911117::INFO::2013-11-19 21:27:37,458::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911118::DEBUG::2013-11-19 21:27:37,502::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911118::INFO::2013-11-19 21:27:37,512::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: clearTask, args: ( taskID=3c078476-ceba-49ba-8c4f-cfc47f187488) >Thread-15911118::DEBUG::2013-11-19 21:27:37,514::task::495::TaskManager.Task::(_debug) Task abf8fba0-a9c2-4710-8ea0-cbca5cef098e: moving from state init -> state preparing >Thread-15911118::DEBUG::2013-11-19 21:27:37,517::taskManager::146::TaskManager::(clearTask) Entry. taskID: 3c078476-ceba-49ba-8c4f-cfc47f187488 >Thread-15911118::DEBUG::2013-11-19 21:27:37,520::taskManager::151::TaskManager::(clearTask) Return. >Thread-15911118::DEBUG::2013-11-19 21:27:37,526::task::495::TaskManager.Task::(_debug) Task abf8fba0-a9c2-4710-8ea0-cbca5cef098e: finished: None >Thread-15911118::DEBUG::2013-11-19 21:27:37,527::task::495::TaskManager.Task::(_debug) Task abf8fba0-a9c2-4710-8ea0-cbca5cef098e: moving from state preparing -> state finished >Thread-15911118::DEBUG::2013-11-19 21:27:37,531::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911118::DEBUG::2013-11-19 21:27:37,539::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911118::DEBUG::2013-11-19 21:27:37,547::task::495::TaskManager.Task::(_debug) Task abf8fba0-a9c2-4710-8ea0-cbca5cef098e: ref 0 aborting False >Thread-15911118::INFO::2013-11-19 21:27:37,550::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: clearTask, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15911119::DEBUG::2013-11-19 21:27:37,646::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911119::INFO::2013-11-19 21:27:37,655::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: deactivateStorageDomain, args: ( sdUUID=5d7b5178-3790-472c-a13b-b481eb27a61e spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 msdUUID=00000000-0000-0000-0000-000000000000 masterVersion=1) >Thread-15911119::DEBUG::2013-11-19 21:27:37,665::task::495::TaskManager.Task::(_debug) Task 332ce334-59da-4235-95f9-b0b20113b537: moving from state init -> state preparing >Thread-15911119::DEBUG::2013-11-19 21:27:37,683::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1beaebe7-35e9-4c58-bfe7-5ebfcbb82552`::Request was made in '/usr/share/vdsm/storage/spm.py' line '1112' at 'public_deactivateStorageDomain' >Thread-15911119::DEBUG::2013-11-19 21:27:37,697::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >Thread-15911119::DEBUG::2013-11-19 21:27:37,698::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >Thread-15911119::DEBUG::2013-11-19 21:27:37,698::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1beaebe7-35e9-4c58-bfe7-5ebfcbb82552`::Granted request >Thread-15911119::DEBUG::2013-11-19 21:27:37,705::task::495::TaskManager.Task::(_debug) Task 332ce334-59da-4235-95f9-b0b20113b537: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (exclusive) >Thread-15911119::DEBUG::2013-11-19 21:27:37,710::task::495::TaskManager.Task::(_debug) Task 332ce334-59da-4235-95f9-b0b20113b537: ref 1 aborting False >Thread-15911119::DEBUG::2013-11-19 21:27:37,719::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`7702d961-8576-4465-b53a-6dbfab16f035`::Request was made in '/usr/share/vdsm/storage/spm.py' line '1113' at 'public_deactivateStorageDomain' >Thread-15911119::DEBUG::2013-11-19 21:27:37,744::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' for lock type 'exclusive' >Thread-15911119::DEBUG::2013-11-19 21:27:37,750::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' is free. Now locking as 'exclusive' (1 active user) >Thread-15911119::DEBUG::2013-11-19 21:27:37,755::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`7702d961-8576-4465-b53a-6dbfab16f035`::Granted request >Thread-15911119::DEBUG::2013-11-19 21:27:37,764::task::495::TaskManager.Task::(_debug) Task 332ce334-59da-4235-95f9-b0b20113b537: _resourcesAcquired: Storage.5d7b5178-3790-472c-a13b-b481eb27a61e (exclusive) >Thread-15911119::DEBUG::2013-11-19 21:27:37,775::task::495::TaskManager.Task::(_debug) Task 332ce334-59da-4235-95f9-b0b20113b537: ref 1 aborting False >Thread-15911119::INFO::2013-11-19 21:27:37,785::sp::775::Storage.StoragePool::(deactivateSD) sdUUID=5d7b5178-3790-472c-a13b-b481eb27a61e spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 new_msdUUID=00000000-0000-0000-0000-000000000000 >Thread-15911119::INFO::2013-11-19 21:27:37,797::fileSD::176::Storage.StorageDomain::(validate) sdUUID=5d7b5178-3790-472c-a13b-b481eb27a61e >Thread-15911119::DEBUG::2013-11-19 21:27:37,808::persistentDict::204::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=[] >Thread-15911119::WARNING::2013-11-19 21:27:37,811::persistentDict::226::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-15911119::ERROR::2013-11-19 21:27:37,811::task::868::TaskManager.Task::(_setError) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 876, in _run > File "/usr/share/vdsm/storage/spm.py", line 115, in run > File "/usr/share/vdsm/storage/spm.py", line 1115, in public_deactivateStorageDomain > File "/usr/share/vdsm/storage/sp.py", line 798, in deactivateSD > File "/usr/share/vdsm/storage/sd.py", line 666, in isBackup > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911119::DEBUG::2013-11-19 21:27:37,830::task::495::TaskManager.Task::(_debug) Task 332ce334-59da-4235-95f9-b0b20113b537: Task._run: 332ce334-59da-4235-95f9-b0b20113b537 ('5d7b5178-3790-472c-a13b-b481eb27a61e', 'c4e5889a-348d-11e2-83fe-9f72369eab30', '00000000-0000-0000-0000-000000000000', 1) {} failed - stopping task >Thread-15911119::DEBUG::2013-11-19 21:27:37,846::task::495::TaskManager.Task::(_debug) Task 332ce334-59da-4235-95f9-b0b20113b537: stopping in state preparing (force False) >Thread-15911119::DEBUG::2013-11-19 21:27:37,846::task::495::TaskManager.Task::(_debug) Task 332ce334-59da-4235-95f9-b0b20113b537: ref 1 aborting True >Thread-15911119::INFO::2013-11-19 21:27:37,853::task::1171::TaskManager.Task::(prepare) aborting: Task is aborted: "'CLASS'" - code 100 >Thread-15911119::DEBUG::2013-11-19 21:27:37,854::task::495::TaskManager.Task::(_debug) Task 332ce334-59da-4235-95f9-b0b20113b537: Prepare: aborted: 'CLASS' >Thread-15911119::DEBUG::2013-11-19 21:27:37,854::task::495::TaskManager.Task::(_debug) Task 332ce334-59da-4235-95f9-b0b20113b537: ref 0 aborting True >Thread-15911119::DEBUG::2013-11-19 21:27:37,857::task::495::TaskManager.Task::(_debug) Task 332ce334-59da-4235-95f9-b0b20113b537: Task._doAbort: force False >Thread-15911119::DEBUG::2013-11-19 21:27:37,858::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911119::DEBUG::2013-11-19 21:27:37,861::task::495::TaskManager.Task::(_debug) Task 332ce334-59da-4235-95f9-b0b20113b537: moving from state preparing -> state aborting >Thread-15911119::DEBUG::2013-11-19 21:27:37,861::task::495::TaskManager.Task::(_debug) Task 332ce334-59da-4235-95f9-b0b20113b537: _aborting: recover policy none >Thread-15911119::DEBUG::2013-11-19 21:27:37,862::task::495::TaskManager.Task::(_debug) Task 332ce334-59da-4235-95f9-b0b20113b537: moving from state aborting -> state failed >Thread-15911119::DEBUG::2013-11-19 21:27:37,863::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>, 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e': < ResourceRef 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e', isValid: 'True' obj: 'None'>} >Thread-15911119::DEBUG::2013-11-19 21:27:37,866::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911119::DEBUG::2013-11-19 21:27:37,869::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911119::DEBUG::2013-11-19 21:27:37,873::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911119::DEBUG::2013-11-19 21:27:37,879::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911119::DEBUG::2013-11-19 21:27:37,879::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911119::DEBUG::2013-11-19 21:27:37,883::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' >Thread-15911119::DEBUG::2013-11-19 21:27:37,884::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' (0 active users) >Thread-15911119::DEBUG::2013-11-19 21:27:37,889::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' is free, finding out if anyone is waiting for it. >Thread-15911119::DEBUG::2013-11-19 21:27:37,889::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e', Clearing records. >Thread-15911119::ERROR::2013-11-19 21:27:37,890::dispatcher::106::Storage.Dispatcher.Protect::(run) 'CLASS' >Thread-15911119::ERROR::2013-11-19 21:27:37,891::dispatcher::107::Storage.Dispatcher.Protect::(run) Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 96, in run > File "/usr/share/vdsm/storage/task.py", line 1178, in prepare >KeyError: 'CLASS' > >Thread-15911120::DEBUG::2013-11-19 21:27:37,917::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911120::INFO::2013-11-19 21:27:37,918::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksInfo, args: () >Thread-15911120::DEBUG::2013-11-19 21:27:37,918::task::495::TaskManager.Task::(_debug) Task de0e06d1-14d7-4e6c-a16e-c621d6b7ab2a: moving from state init -> state preparing >Thread-15911120::DEBUG::2013-11-19 21:27:37,919::taskManager::168::TaskManager::(getAllTasksInfo) Entry. >Thread-15911120::DEBUG::2013-11-19 21:27:37,924::taskManager::173::TaskManager::(getAllTasksInfo) Return. Response: {} >Thread-15911120::DEBUG::2013-11-19 21:27:37,924::task::495::TaskManager.Task::(_debug) Task de0e06d1-14d7-4e6c-a16e-c621d6b7ab2a: finished: {'allTasksInfo': {}} >Thread-15911120::DEBUG::2013-11-19 21:27:37,925::task::495::TaskManager.Task::(_debug) Task de0e06d1-14d7-4e6c-a16e-c621d6b7ab2a: moving from state preparing -> state finished >Thread-15911120::DEBUG::2013-11-19 21:27:37,930::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911120::DEBUG::2013-11-19 21:27:37,935::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911120::DEBUG::2013-11-19 21:27:37,936::task::495::TaskManager.Task::(_debug) Task de0e06d1-14d7-4e6c-a16e-c621d6b7ab2a: ref 0 aborting False >Thread-15911120::INFO::2013-11-19 21:27:37,936::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'allTasksInfo': {}} >Thread-15911121::DEBUG::2013-11-19 21:27:37,976::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911121::INFO::2013-11-19 21:27:37,980::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911121::DEBUG::2013-11-19 21:27:37,981::task::495::TaskManager.Task::(_debug) Task 946f43a2-a4d7-4e9b-8c22-885ae40b10f2: moving from state init -> state preparing >Thread-15911121::DEBUG::2013-11-19 21:27:37,986::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911121::DEBUG::2013-11-19 21:27:37,986::task::495::TaskManager.Task::(_debug) Task 946f43a2-a4d7-4e9b-8c22-885ae40b10f2: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911121::DEBUG::2013-11-19 21:27:37,987::task::495::TaskManager.Task::(_debug) Task 946f43a2-a4d7-4e9b-8c22-885ae40b10f2: moving from state preparing -> state finished >Thread-15911121::DEBUG::2013-11-19 21:27:37,993::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911121::DEBUG::2013-11-19 21:27:37,993::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911121::DEBUG::2013-11-19 21:27:37,994::task::495::TaskManager.Task::(_debug) Task 946f43a2-a4d7-4e9b-8c22-885ae40b10f2: ref 0 aborting False >Thread-15911121::INFO::2013-11-19 21:27:37,995::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911122::DEBUG::2013-11-19 21:27:38,026::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911122::INFO::2013-11-19 21:27:38,031::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911122::DEBUG::2013-11-19 21:27:38,031::task::495::TaskManager.Task::(_debug) Task a4962d61-42e7-46d8-a26f-1360e43651e9: moving from state init -> state preparing >Thread-15911122::DEBUG::2013-11-19 21:27:38,042::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9a8bbb37-59ab-48ea-bb3a-5df4da3436a4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911122::DEBUG::2013-11-19 21:27:38,046::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911122::DEBUG::2013-11-19 21:27:38,047::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911122::DEBUG::2013-11-19 21:27:38,050::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9a8bbb37-59ab-48ea-bb3a-5df4da3436a4`::Granted request >Thread-15911122::DEBUG::2013-11-19 21:27:38,059::task::495::TaskManager.Task::(_debug) Task a4962d61-42e7-46d8-a26f-1360e43651e9: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911122::DEBUG::2013-11-19 21:27:38,060::task::495::TaskManager.Task::(_debug) Task a4962d61-42e7-46d8-a26f-1360e43651e9: ref 1 aborting False >Thread-15911122::WARNING::2013-11-19 21:27:38,083::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911122::DEBUG::2013-11-19 21:27:38,101::task::495::TaskManager.Task::(_debug) Task a4962d61-42e7-46d8-a26f-1360e43651e9: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911122::DEBUG::2013-11-19 21:27:38,118::task::495::TaskManager.Task::(_debug) Task a4962d61-42e7-46d8-a26f-1360e43651e9: moving from state preparing -> state finished >Thread-15911122::DEBUG::2013-11-19 21:27:38,129::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911122::DEBUG::2013-11-19 21:27:38,137::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911122::DEBUG::2013-11-19 21:27:38,138::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911122::DEBUG::2013-11-19 21:27:38,141::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911122::DEBUG::2013-11-19 21:27:38,142::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911122::DEBUG::2013-11-19 21:27:38,147::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911122::DEBUG::2013-11-19 21:27:38,148::task::495::TaskManager.Task::(_debug) Task a4962d61-42e7-46d8-a26f-1360e43651e9: ref 0 aborting False >Thread-15911122::INFO::2013-11-19 21:27:38,156::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911123::DEBUG::2013-11-19 21:27:38,330::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911123::INFO::2013-11-19 21:27:38,339::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksInfo, args: () >Thread-15911123::DEBUG::2013-11-19 21:27:38,339::task::495::TaskManager.Task::(_debug) Task 59fbbf91-b166-437f-9b13-5e671437cbc6: moving from state init -> state preparing >Thread-15911123::DEBUG::2013-11-19 21:27:38,340::taskManager::168::TaskManager::(getAllTasksInfo) Entry. >Thread-15911123::DEBUG::2013-11-19 21:27:38,340::taskManager::173::TaskManager::(getAllTasksInfo) Return. Response: {} >Thread-15911123::DEBUG::2013-11-19 21:27:38,341::task::495::TaskManager.Task::(_debug) Task 59fbbf91-b166-437f-9b13-5e671437cbc6: finished: {'allTasksInfo': {}} >Thread-15911123::DEBUG::2013-11-19 21:27:38,342::task::495::TaskManager.Task::(_debug) Task 59fbbf91-b166-437f-9b13-5e671437cbc6: moving from state preparing -> state finished >Thread-15911123::DEBUG::2013-11-19 21:27:38,342::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911123::DEBUG::2013-11-19 21:27:38,352::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911123::DEBUG::2013-11-19 21:27:38,355::task::495::TaskManager.Task::(_debug) Task 59fbbf91-b166-437f-9b13-5e671437cbc6: ref 0 aborting False >Thread-15911123::INFO::2013-11-19 21:27:38,356::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'allTasksInfo': {}} >Dummy-15911115::DEBUG::2013-11-19 21:27:39,289::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:27:39,830::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0741424 s, 13.8 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:27:41,096::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:27:41,846::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:27:42,377::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0732462 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:27:44,391::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:27:44,938::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0733825 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:27:46,954::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:27:47,486::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0733239 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15911128::INFO::2013-11-19 21:27:48,188::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911128::DEBUG::2013-11-19 21:27:48,190::task::495::TaskManager.Task::(_debug) Task 0dac1488-d47d-4625-9408-4765b1a597c7: moving from state init -> state preparing >Thread-15911128::DEBUG::2013-11-19 21:27:48,190::task::495::TaskManager.Task::(_debug) Task 0dac1488-d47d-4625-9408-4765b1a597c7: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000728845596313', 'lastCheck': 1384896466.1345589, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00233507156372', 'lastCheck': 1384896465.7145, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896461.0970981, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000458955764771', 'lastCheck': 1384896461.6780951, 'valid': True, 'code': 0}} >Thread-15911128::DEBUG::2013-11-19 21:27:48,207::task::495::TaskManager.Task::(_debug) Task 0dac1488-d47d-4625-9408-4765b1a597c7: moving from state preparing -> state finished >Thread-15911128::DEBUG::2013-11-19 21:27:48,217::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911128::DEBUG::2013-11-19 21:27:48,228::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911128::DEBUG::2013-11-19 21:27:48,234::task::495::TaskManager.Task::(_debug) Task 0dac1488-d47d-4625-9408-4765b1a597c7: ref 0 aborting False >Thread-15911128::INFO::2013-11-19 21:27:48,244::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000728845596313', 'lastCheck': 1384896466.1345589, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00233507156372', 'lastCheck': 1384896465.7145, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896461.0970981, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000458955764771', 'lastCheck': 1384896461.6780951, 'valid': True, 'code': 0}} >Thread-15911130::DEBUG::2013-11-19 21:27:48,323::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911130::INFO::2013-11-19 21:27:48,330::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911130::DEBUG::2013-11-19 21:27:48,334::task::495::TaskManager.Task::(_debug) Task 8eb88e78-826a-46cd-bd99-79ab1b47fb8e: moving from state init -> state preparing >Thread-15911130::DEBUG::2013-11-19 21:27:48,335::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911130::DEBUG::2013-11-19 21:27:48,336::task::495::TaskManager.Task::(_debug) Task 8eb88e78-826a-46cd-bd99-79ab1b47fb8e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911130::DEBUG::2013-11-19 21:27:48,336::task::495::TaskManager.Task::(_debug) Task 8eb88e78-826a-46cd-bd99-79ab1b47fb8e: moving from state preparing -> state finished >Thread-15911130::DEBUG::2013-11-19 21:27:48,337::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911130::DEBUG::2013-11-19 21:27:48,343::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911130::DEBUG::2013-11-19 21:27:48,347::task::495::TaskManager.Task::(_debug) Task 8eb88e78-826a-46cd-bd99-79ab1b47fb8e: ref 0 aborting False >Thread-15911130::INFO::2013-11-19 21:27:48,351::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911131::DEBUG::2013-11-19 21:27:48,394::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911131::INFO::2013-11-19 21:27:48,398::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911131::DEBUG::2013-11-19 21:27:48,398::task::495::TaskManager.Task::(_debug) Task 09f5de93-ff56-4db5-bd43-84697aa36d28: moving from state init -> state preparing >Thread-15911131::DEBUG::2013-11-19 21:27:48,399::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6f1334bb-a250-4d4f-97ad-bbb35aa91bbc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911131::DEBUG::2013-11-19 21:27:48,415::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911131::DEBUG::2013-11-19 21:27:48,421::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911131::DEBUG::2013-11-19 21:27:48,427::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6f1334bb-a250-4d4f-97ad-bbb35aa91bbc`::Granted request >Thread-15911131::DEBUG::2013-11-19 21:27:48,435::task::495::TaskManager.Task::(_debug) Task 09f5de93-ff56-4db5-bd43-84697aa36d28: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911131::DEBUG::2013-11-19 21:27:48,450::task::495::TaskManager.Task::(_debug) Task 09f5de93-ff56-4db5-bd43-84697aa36d28: ref 1 aborting False >Thread-15911131::WARNING::2013-11-19 21:27:48,465::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911131::DEBUG::2013-11-19 21:27:48,479::task::495::TaskManager.Task::(_debug) Task 09f5de93-ff56-4db5-bd43-84697aa36d28: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911131::DEBUG::2013-11-19 21:27:48,515::task::495::TaskManager.Task::(_debug) Task 09f5de93-ff56-4db5-bd43-84697aa36d28: moving from state preparing -> state finished >Thread-15911131::DEBUG::2013-11-19 21:27:48,517::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911131::DEBUG::2013-11-19 21:27:48,517::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911131::DEBUG::2013-11-19 21:27:48,518::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911131::DEBUG::2013-11-19 21:27:48,519::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911131::DEBUG::2013-11-19 21:27:48,519::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911131::DEBUG::2013-11-19 21:27:48,525::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911131::DEBUG::2013-11-19 21:27:48,525::task::495::TaskManager.Task::(_debug) Task 09f5de93-ff56-4db5-bd43-84697aa36d28: ref 0 aborting False >Thread-15911131::INFO::2013-11-19 21:27:48,533::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:27:49,502::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:27:50,019::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0732388 s, 14.0 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:27:51,194::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:27:52,034::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:27:52,572::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0737843 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:27:54,588::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:27:55,142::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.102934 s, 9.9 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:27:57,157::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:27:57,644::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0704051 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15911136::INFO::2013-11-19 21:27:58,509::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911136::DEBUG::2013-11-19 21:27:58,510::task::495::TaskManager.Task::(_debug) Task f605972b-9bda-4649-9c32-156f938935ef: moving from state init -> state preparing >Thread-15911136::DEBUG::2013-11-19 21:27:58,516::task::495::TaskManager.Task::(_debug) Task f605972b-9bda-4649-9c32-156f938935ef: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000345945358276', 'lastCheck': 1384896476.2618029, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00156188011169', 'lastCheck': 1384896475.801497, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896471.1948969, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000381946563721', 'lastCheck': 1384896471.7057941, 'valid': True, 'code': 0}} >Thread-15911136::DEBUG::2013-11-19 21:27:58,525::task::495::TaskManager.Task::(_debug) Task f605972b-9bda-4649-9c32-156f938935ef: moving from state preparing -> state finished >Thread-15911136::DEBUG::2013-11-19 21:27:58,531::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911136::DEBUG::2013-11-19 21:27:58,535::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911136::DEBUG::2013-11-19 21:27:58,538::task::495::TaskManager.Task::(_debug) Task f605972b-9bda-4649-9c32-156f938935ef: ref 0 aborting False >Thread-15911136::INFO::2013-11-19 21:27:58,541::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000345945358276', 'lastCheck': 1384896476.2618029, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00156188011169', 'lastCheck': 1384896475.801497, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896471.1948969, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000381946563721', 'lastCheck': 1384896471.7057941, 'valid': True, 'code': 0}} >Thread-15911138::DEBUG::2013-11-19 21:27:58,689::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911138::INFO::2013-11-19 21:27:58,694::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911138::DEBUG::2013-11-19 21:27:58,695::task::495::TaskManager.Task::(_debug) Task 2276a252-8fe7-4f3c-bb89-a13183e151f5: moving from state init -> state preparing >Thread-15911138::DEBUG::2013-11-19 21:27:58,696::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911138::DEBUG::2013-11-19 21:27:58,700::task::495::TaskManager.Task::(_debug) Task 2276a252-8fe7-4f3c-bb89-a13183e151f5: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911138::DEBUG::2013-11-19 21:27:58,700::task::495::TaskManager.Task::(_debug) Task 2276a252-8fe7-4f3c-bb89-a13183e151f5: moving from state preparing -> state finished >Thread-15911138::DEBUG::2013-11-19 21:27:58,701::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911138::DEBUG::2013-11-19 21:27:58,706::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911138::DEBUG::2013-11-19 21:27:58,707::task::495::TaskManager.Task::(_debug) Task 2276a252-8fe7-4f3c-bb89-a13183e151f5: ref 0 aborting False >Thread-15911138::INFO::2013-11-19 21:27:58,707::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911139::DEBUG::2013-11-19 21:27:58,760::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911139::INFO::2013-11-19 21:27:58,761::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911139::DEBUG::2013-11-19 21:27:58,761::task::495::TaskManager.Task::(_debug) Task 9026fdae-aae0-4932-bd74-55d362093df7: moving from state init -> state preparing >Thread-15911139::DEBUG::2013-11-19 21:27:58,771::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c0a24abb-9dab-460b-a53f-9034aa569527`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911139::DEBUG::2013-11-19 21:27:58,777::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911139::DEBUG::2013-11-19 21:27:58,778::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911139::DEBUG::2013-11-19 21:27:58,778::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c0a24abb-9dab-460b-a53f-9034aa569527`::Granted request >Thread-15911139::DEBUG::2013-11-19 21:27:58,782::task::495::TaskManager.Task::(_debug) Task 9026fdae-aae0-4932-bd74-55d362093df7: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911139::DEBUG::2013-11-19 21:27:58,783::task::495::TaskManager.Task::(_debug) Task 9026fdae-aae0-4932-bd74-55d362093df7: ref 1 aborting False >Thread-15911139::WARNING::2013-11-19 21:27:58,794::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911139::DEBUG::2013-11-19 21:27:58,809::task::495::TaskManager.Task::(_debug) Task 9026fdae-aae0-4932-bd74-55d362093df7: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911139::DEBUG::2013-11-19 21:27:58,817::task::495::TaskManager.Task::(_debug) Task 9026fdae-aae0-4932-bd74-55d362093df7: moving from state preparing -> state finished >Thread-15911139::DEBUG::2013-11-19 21:27:58,821::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911139::DEBUG::2013-11-19 21:27:58,824::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911139::DEBUG::2013-11-19 21:27:58,829::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911139::DEBUG::2013-11-19 21:27:58,840::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911139::DEBUG::2013-11-19 21:27:58,841::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911139::DEBUG::2013-11-19 21:27:58,841::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911139::DEBUG::2013-11-19 21:27:58,842::task::495::TaskManager.Task::(_debug) Task 9026fdae-aae0-4932-bd74-55d362093df7: ref 0 aborting False >Thread-15911139::INFO::2013-11-19 21:27:58,847::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:27:59,660::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:00,218::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0834281 s, 12.3 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:28:01,399::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:28:02,233::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:02,772::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0897622 s, 11.4 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:28:04,787::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:05,322::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0736538 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:28:07,338::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:07,863::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0739876 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15911144::INFO::2013-11-19 21:28:08,762::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911144::DEBUG::2013-11-19 21:28:08,763::task::495::TaskManager.Task::(_debug) Task 51dd015a-5eb7-4ca4-805d-9ff18190059a: moving from state init -> state preparing >Thread-15911144::DEBUG::2013-11-19 21:28:08,764::task::495::TaskManager.Task::(_debug) Task 51dd015a-5eb7-4ca4-805d-9ff18190059a: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000401973724365', 'lastCheck': 1384896486.4470141, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00157189369202', 'lastCheck': 1384896486.0126951, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896481.4001191, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000561952590942', 'lastCheck': 1384896481.8916891, 'valid': True, 'code': 0}} >Thread-15911144::DEBUG::2013-11-19 21:28:08,770::task::495::TaskManager.Task::(_debug) Task 51dd015a-5eb7-4ca4-805d-9ff18190059a: moving from state preparing -> state finished >Thread-15911144::DEBUG::2013-11-19 21:28:08,770::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911144::DEBUG::2013-11-19 21:28:08,777::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911144::DEBUG::2013-11-19 21:28:08,777::task::495::TaskManager.Task::(_debug) Task 51dd015a-5eb7-4ca4-805d-9ff18190059a: ref 0 aborting False >Thread-15911144::INFO::2013-11-19 21:28:08,778::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000401973724365', 'lastCheck': 1384896486.4470141, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00157189369202', 'lastCheck': 1384896486.0126951, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896481.4001191, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000561952590942', 'lastCheck': 1384896481.8916891, 'valid': True, 'code': 0}} >Thread-15911146::DEBUG::2013-11-19 21:28:08,990::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911146::INFO::2013-11-19 21:28:08,991::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911146::DEBUG::2013-11-19 21:28:08,991::task::495::TaskManager.Task::(_debug) Task d0c028cf-f34a-44b2-b271-6ced113b08ac: moving from state init -> state preparing >Thread-15911146::DEBUG::2013-11-19 21:28:08,992::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911146::DEBUG::2013-11-19 21:28:08,993::task::495::TaskManager.Task::(_debug) Task d0c028cf-f34a-44b2-b271-6ced113b08ac: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911146::DEBUG::2013-11-19 21:28:08,997::task::495::TaskManager.Task::(_debug) Task d0c028cf-f34a-44b2-b271-6ced113b08ac: moving from state preparing -> state finished >Thread-15911146::DEBUG::2013-11-19 21:28:09,003::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911146::DEBUG::2013-11-19 21:28:09,003::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911146::DEBUG::2013-11-19 21:28:09,004::task::495::TaskManager.Task::(_debug) Task d0c028cf-f34a-44b2-b271-6ced113b08ac: ref 0 aborting False >Thread-15911146::INFO::2013-11-19 21:28:09,009::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911147::DEBUG::2013-11-19 21:28:09,039::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911147::INFO::2013-11-19 21:28:09,040::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911147::DEBUG::2013-11-19 21:28:09,041::task::495::TaskManager.Task::(_debug) Task 8b2dbed3-cf6f-4a57-a71e-290614f99032: moving from state init -> state preparing >Thread-15911147::DEBUG::2013-11-19 21:28:09,044::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ac163f13-782f-45fb-be97-e3266ae29eba`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911147::DEBUG::2013-11-19 21:28:09,055::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911147::DEBUG::2013-11-19 21:28:09,056::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911147::DEBUG::2013-11-19 21:28:09,057::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ac163f13-782f-45fb-be97-e3266ae29eba`::Granted request >Thread-15911147::DEBUG::2013-11-19 21:28:09,058::task::495::TaskManager.Task::(_debug) Task 8b2dbed3-cf6f-4a57-a71e-290614f99032: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911147::DEBUG::2013-11-19 21:28:09,064::task::495::TaskManager.Task::(_debug) Task 8b2dbed3-cf6f-4a57-a71e-290614f99032: ref 1 aborting False >Thread-15911147::WARNING::2013-11-19 21:28:09,081::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911147::DEBUG::2013-11-19 21:28:09,102::task::495::TaskManager.Task::(_debug) Task 8b2dbed3-cf6f-4a57-a71e-290614f99032: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911147::DEBUG::2013-11-19 21:28:09,134::task::495::TaskManager.Task::(_debug) Task 8b2dbed3-cf6f-4a57-a71e-290614f99032: moving from state preparing -> state finished >Thread-15911147::DEBUG::2013-11-19 21:28:09,135::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911147::DEBUG::2013-11-19 21:28:09,135::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911147::DEBUG::2013-11-19 21:28:09,143::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911147::DEBUG::2013-11-19 21:28:09,144::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911147::DEBUG::2013-11-19 21:28:09,150::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911147::DEBUG::2013-11-19 21:28:09,151::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911147::DEBUG::2013-11-19 21:28:09,151::task::495::TaskManager.Task::(_debug) Task 8b2dbed3-cf6f-4a57-a71e-290614f99032: ref 0 aborting False >Thread-15911147::INFO::2013-11-19 21:28:09,152::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:28:09,878::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:10,395::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0726088 s, 14.1 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:28:11,551::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:28:12,410::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:12,986::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.09985 s, 10.3 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:28:15,001::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:15,533::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0836241 s, 12.2 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:28:17,549::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:18,094::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0799157 s, 12.8 MB/s\n'; <rc> = 0 >Thread-15911152::INFO::2013-11-19 21:28:18,987::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911152::DEBUG::2013-11-19 21:28:18,993::task::495::TaskManager.Task::(_debug) Task a5b6d3e1-77fe-40b8-9758-ca8f155e576e: moving from state init -> state preparing >Thread-15911152::DEBUG::2013-11-19 21:28:18,994::task::495::TaskManager.Task::(_debug) Task a5b6d3e1-77fe-40b8-9758-ca8f155e576e: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00037693977356', 'lastCheck': 1384896496.64905, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00151991844177', 'lastCheck': 1384896496.2208431, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896491.5521519, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00053596496582', 'lastCheck': 1384896492.0845261, 'valid': True, 'code': 0}} >Thread-15911152::DEBUG::2013-11-19 21:28:19,001::task::495::TaskManager.Task::(_debug) Task a5b6d3e1-77fe-40b8-9758-ca8f155e576e: moving from state preparing -> state finished >Thread-15911152::DEBUG::2013-11-19 21:28:19,001::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911152::DEBUG::2013-11-19 21:28:19,006::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911152::DEBUG::2013-11-19 21:28:19,007::task::495::TaskManager.Task::(_debug) Task a5b6d3e1-77fe-40b8-9758-ca8f155e576e: ref 0 aborting False >Thread-15911152::INFO::2013-11-19 21:28:19,007::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00037693977356', 'lastCheck': 1384896496.64905, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00151991844177', 'lastCheck': 1384896496.2208431, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896491.5521519, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00053596496582', 'lastCheck': 1384896492.0845261, 'valid': True, 'code': 0}} >Thread-15911154::DEBUG::2013-11-19 21:28:19,282::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911154::INFO::2013-11-19 21:28:19,283::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911154::DEBUG::2013-11-19 21:28:19,287::task::495::TaskManager.Task::(_debug) Task b7db5e3d-df52-43e6-8908-506215ad9cf6: moving from state init -> state preparing >Thread-15911154::DEBUG::2013-11-19 21:28:19,293::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911154::DEBUG::2013-11-19 21:28:19,293::task::495::TaskManager.Task::(_debug) Task b7db5e3d-df52-43e6-8908-506215ad9cf6: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911154::DEBUG::2013-11-19 21:28:19,294::task::495::TaskManager.Task::(_debug) Task b7db5e3d-df52-43e6-8908-506215ad9cf6: moving from state preparing -> state finished >Thread-15911154::DEBUG::2013-11-19 21:28:19,294::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911154::DEBUG::2013-11-19 21:28:19,295::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911154::DEBUG::2013-11-19 21:28:19,295::task::495::TaskManager.Task::(_debug) Task b7db5e3d-df52-43e6-8908-506215ad9cf6: ref 0 aborting False >Thread-15911154::INFO::2013-11-19 21:28:19,300::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911155::DEBUG::2013-11-19 21:28:19,341::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911155::INFO::2013-11-19 21:28:19,347::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911155::DEBUG::2013-11-19 21:28:19,351::task::495::TaskManager.Task::(_debug) Task 92e56e6d-50d1-493e-a388-92c9ed9f6264: moving from state init -> state preparing >Thread-15911155::DEBUG::2013-11-19 21:28:19,356::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f9dbfcce-3181-4bca-a718-37abd97baf8f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911155::DEBUG::2013-11-19 21:28:19,357::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911155::DEBUG::2013-11-19 21:28:19,363::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911155::DEBUG::2013-11-19 21:28:19,364::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f9dbfcce-3181-4bca-a718-37abd97baf8f`::Granted request >Thread-15911155::DEBUG::2013-11-19 21:28:19,370::task::495::TaskManager.Task::(_debug) Task 92e56e6d-50d1-493e-a388-92c9ed9f6264: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911155::DEBUG::2013-11-19 21:28:19,371::task::495::TaskManager.Task::(_debug) Task 92e56e6d-50d1-493e-a388-92c9ed9f6264: ref 1 aborting False >Thread-15911155::WARNING::2013-11-19 21:28:19,388::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911155::DEBUG::2013-11-19 21:28:19,392::task::495::TaskManager.Task::(_debug) Task 92e56e6d-50d1-493e-a388-92c9ed9f6264: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911155::DEBUG::2013-11-19 21:28:19,402::task::495::TaskManager.Task::(_debug) Task 92e56e6d-50d1-493e-a388-92c9ed9f6264: moving from state preparing -> state finished >Thread-15911155::DEBUG::2013-11-19 21:28:19,405::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911155::DEBUG::2013-11-19 21:28:19,408::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911155::DEBUG::2013-11-19 21:28:19,416::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911155::DEBUG::2013-11-19 21:28:19,417::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911155::DEBUG::2013-11-19 21:28:19,418::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911155::DEBUG::2013-11-19 21:28:19,418::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911155::DEBUG::2013-11-19 21:28:19,419::task::495::TaskManager.Task::(_debug) Task 92e56e6d-50d1-493e-a388-92c9ed9f6264: ref 0 aborting False >Thread-15911155::INFO::2013-11-19 21:28:19,425::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:28:20,110::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:20,628::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0731599 s, 14.0 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:28:21,776::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:28:22,642::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:23,206::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925146 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:28:25,222::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:25,744::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0740442 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:28:27,759::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:28,288::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0741539 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15911160::INFO::2013-11-19 21:28:29,198::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911160::DEBUG::2013-11-19 21:28:29,199::task::495::TaskManager.Task::(_debug) Task a5d06c92-2c98-45b8-ba91-22909f6867cb: moving from state init -> state preparing >Thread-15911160::DEBUG::2013-11-19 21:28:29,201::task::495::TaskManager.Task::(_debug) Task a5d06c92-2c98-45b8-ba91-22909f6867cb: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000377893447876', 'lastCheck': 1384896506.878119, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00151705741882', 'lastCheck': 1384896506.4331961, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896501.7776141, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000483989715576', 'lastCheck': 1384896502.3138571, 'valid': True, 'code': 0}} >Thread-15911160::DEBUG::2013-11-19 21:28:29,211::task::495::TaskManager.Task::(_debug) Task a5d06c92-2c98-45b8-ba91-22909f6867cb: moving from state preparing -> state finished >Thread-15911160::DEBUG::2013-11-19 21:28:29,211::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911160::DEBUG::2013-11-19 21:28:29,212::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911160::DEBUG::2013-11-19 21:28:29,212::task::495::TaskManager.Task::(_debug) Task a5d06c92-2c98-45b8-ba91-22909f6867cb: ref 0 aborting False >Thread-15911160::INFO::2013-11-19 21:28:29,213::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000377893447876', 'lastCheck': 1384896506.878119, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00151705741882', 'lastCheck': 1384896506.4331961, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896501.7776141, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000483989715576', 'lastCheck': 1384896502.3138571, 'valid': True, 'code': 0}} >Thread-15911162::DEBUG::2013-11-19 21:28:29,567::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911162::INFO::2013-11-19 21:28:29,568::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911162::DEBUG::2013-11-19 21:28:29,571::task::495::TaskManager.Task::(_debug) Task 921dd74e-f1e2-443b-9eee-fca423e5551e: moving from state init -> state preparing >Thread-15911162::DEBUG::2013-11-19 21:28:29,571::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911162::DEBUG::2013-11-19 21:28:29,574::task::495::TaskManager.Task::(_debug) Task 921dd74e-f1e2-443b-9eee-fca423e5551e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911162::DEBUG::2013-11-19 21:28:29,575::task::495::TaskManager.Task::(_debug) Task 921dd74e-f1e2-443b-9eee-fca423e5551e: moving from state preparing -> state finished >Thread-15911162::DEBUG::2013-11-19 21:28:29,575::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911162::DEBUG::2013-11-19 21:28:29,576::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911162::DEBUG::2013-11-19 21:28:29,576::task::495::TaskManager.Task::(_debug) Task 921dd74e-f1e2-443b-9eee-fca423e5551e: ref 0 aborting False >Thread-15911162::INFO::2013-11-19 21:28:29,582::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911163::DEBUG::2013-11-19 21:28:29,596::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911163::INFO::2013-11-19 21:28:29,597::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911163::DEBUG::2013-11-19 21:28:29,601::task::495::TaskManager.Task::(_debug) Task fc13eb4c-38f4-48fb-bcdb-8f0d3cbafcd1: moving from state init -> state preparing >Thread-15911163::DEBUG::2013-11-19 21:28:29,602::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f62d47b4-e79a-4af4-a801-57b6bcdbcc8b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911163::DEBUG::2013-11-19 21:28:29,603::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911163::DEBUG::2013-11-19 21:28:29,605::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911163::DEBUG::2013-11-19 21:28:29,605::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f62d47b4-e79a-4af4-a801-57b6bcdbcc8b`::Granted request >Thread-15911163::DEBUG::2013-11-19 21:28:29,612::task::495::TaskManager.Task::(_debug) Task fc13eb4c-38f4-48fb-bcdb-8f0d3cbafcd1: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911163::DEBUG::2013-11-19 21:28:29,613::task::495::TaskManager.Task::(_debug) Task fc13eb4c-38f4-48fb-bcdb-8f0d3cbafcd1: ref 1 aborting False >Thread-15911163::WARNING::2013-11-19 21:28:29,620::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911163::DEBUG::2013-11-19 21:28:29,639::task::495::TaskManager.Task::(_debug) Task fc13eb4c-38f4-48fb-bcdb-8f0d3cbafcd1: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911163::DEBUG::2013-11-19 21:28:29,646::task::495::TaskManager.Task::(_debug) Task fc13eb4c-38f4-48fb-bcdb-8f0d3cbafcd1: moving from state preparing -> state finished >Thread-15911163::DEBUG::2013-11-19 21:28:29,652::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911163::DEBUG::2013-11-19 21:28:29,652::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911163::DEBUG::2013-11-19 21:28:29,653::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911163::DEBUG::2013-11-19 21:28:29,654::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911163::DEBUG::2013-11-19 21:28:29,654::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911163::DEBUG::2013-11-19 21:28:29,655::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911163::DEBUG::2013-11-19 21:28:29,655::task::495::TaskManager.Task::(_debug) Task fc13eb4c-38f4-48fb-bcdb-8f0d3cbafcd1: ref 0 aborting False >Thread-15911163::INFO::2013-11-19 21:28:29,661::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:28:30,303::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:30,850::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0736112 s, 13.9 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:28:31,986::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:28:32,866::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:33,396::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0870685 s, 11.8 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:28:35,411::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:35,948::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0740842 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:28:37,963::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:38,487::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0745398 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15911168::INFO::2013-11-19 21:28:39,430::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911168::DEBUG::2013-11-19 21:28:39,435::task::495::TaskManager.Task::(_debug) Task b2c5ddee-812b-4824-98e6-9a896575d49d: moving from state init -> state preparing >Thread-15911168::DEBUG::2013-11-19 21:28:39,441::task::495::TaskManager.Task::(_debug) Task b2c5ddee-812b-4824-98e6-9a896575d49d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000410079956055', 'lastCheck': 1384896517.0772691, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00151991844177', 'lastCheck': 1384896516.6382909, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896511.987448, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000568866729736', 'lastCheck': 1384896512.5396371, 'valid': True, 'code': 0}} >Thread-15911168::DEBUG::2013-11-19 21:28:39,444::task::495::TaskManager.Task::(_debug) Task b2c5ddee-812b-4824-98e6-9a896575d49d: moving from state preparing -> state finished >Thread-15911168::DEBUG::2013-11-19 21:28:39,450::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911168::DEBUG::2013-11-19 21:28:39,451::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911168::DEBUG::2013-11-19 21:28:39,451::task::495::TaskManager.Task::(_debug) Task b2c5ddee-812b-4824-98e6-9a896575d49d: ref 0 aborting False >Thread-15911168::INFO::2013-11-19 21:28:39,452::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000410079956055', 'lastCheck': 1384896517.0772691, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00151991844177', 'lastCheck': 1384896516.6382909, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896511.987448, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000568866729736', 'lastCheck': 1384896512.5396371, 'valid': True, 'code': 0}} >Thread-15911170::DEBUG::2013-11-19 21:28:39,799::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911170::INFO::2013-11-19 21:28:39,800::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911170::DEBUG::2013-11-19 21:28:39,801::task::495::TaskManager.Task::(_debug) Task 1db20086-86fd-4812-818d-ede90b828d10: moving from state init -> state preparing >Thread-15911170::DEBUG::2013-11-19 21:28:39,807::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911170::DEBUG::2013-11-19 21:28:39,811::task::495::TaskManager.Task::(_debug) Task 1db20086-86fd-4812-818d-ede90b828d10: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911170::DEBUG::2013-11-19 21:28:39,818::task::495::TaskManager.Task::(_debug) Task 1db20086-86fd-4812-818d-ede90b828d10: moving from state preparing -> state finished >Thread-15911170::DEBUG::2013-11-19 21:28:39,818::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911170::DEBUG::2013-11-19 21:28:39,822::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911170::DEBUG::2013-11-19 21:28:39,823::task::495::TaskManager.Task::(_debug) Task 1db20086-86fd-4812-818d-ede90b828d10: ref 0 aborting False >Thread-15911170::INFO::2013-11-19 21:28:39,825::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911171::DEBUG::2013-11-19 21:28:39,851::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911171::INFO::2013-11-19 21:28:39,852::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911171::DEBUG::2013-11-19 21:28:39,856::task::495::TaskManager.Task::(_debug) Task e3752623-146b-45a1-906a-5784aa8a2f43: moving from state init -> state preparing >Thread-15911171::DEBUG::2013-11-19 21:28:39,869::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1cceb5c8-b95f-4136-8518-462874e5942e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911171::DEBUG::2013-11-19 21:28:39,875::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911171::DEBUG::2013-11-19 21:28:39,876::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911171::DEBUG::2013-11-19 21:28:39,877::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1cceb5c8-b95f-4136-8518-462874e5942e`::Granted request >Thread-15911171::DEBUG::2013-11-19 21:28:39,888::task::495::TaskManager.Task::(_debug) Task e3752623-146b-45a1-906a-5784aa8a2f43: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911171::DEBUG::2013-11-19 21:28:39,889::task::495::TaskManager.Task::(_debug) Task e3752623-146b-45a1-906a-5784aa8a2f43: ref 1 aborting False >Thread-15911171::WARNING::2013-11-19 21:28:39,899::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911171::DEBUG::2013-11-19 21:28:39,910::task::495::TaskManager.Task::(_debug) Task e3752623-146b-45a1-906a-5784aa8a2f43: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911171::DEBUG::2013-11-19 21:28:39,913::task::495::TaskManager.Task::(_debug) Task e3752623-146b-45a1-906a-5784aa8a2f43: moving from state preparing -> state finished >Thread-15911171::DEBUG::2013-11-19 21:28:39,914::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911171::DEBUG::2013-11-19 21:28:39,917::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911171::DEBUG::2013-11-19 21:28:39,917::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911171::DEBUG::2013-11-19 21:28:39,918::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911171::DEBUG::2013-11-19 21:28:39,918::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911171::DEBUG::2013-11-19 21:28:39,919::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911171::DEBUG::2013-11-19 21:28:39,920::task::495::TaskManager.Task::(_debug) Task e3752623-146b-45a1-906a-5784aa8a2f43: ref 0 aborting False >Thread-15911171::INFO::2013-11-19 21:28:39,920::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:28:40,502::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:41,014::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0732723 s, 14.0 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:28:42,177::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:28:43,030::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:43,583::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0739955 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:28:45,599::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:46,127::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0727015 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:28:48,150::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:48,692::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0745462 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15911176::INFO::2013-11-19 21:28:49,632::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911176::DEBUG::2013-11-19 21:28:49,633::task::495::TaskManager.Task::(_debug) Task 6e4def5b-4ce5-4e83-a53c-d278d19ac7cf: moving from state init -> state preparing >Thread-15911176::DEBUG::2013-11-19 21:28:49,634::task::495::TaskManager.Task::(_debug) Task 6e4def5b-4ce5-4e83-a53c-d278d19ac7cf: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00042200088501', 'lastCheck': 1384896527.27424, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00242805480957', 'lastCheck': 1384896526.8190501, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896522.1777599, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000473976135254', 'lastCheck': 1384896522.700588, 'valid': True, 'code': 0}} >Thread-15911176::DEBUG::2013-11-19 21:28:49,641::task::495::TaskManager.Task::(_debug) Task 6e4def5b-4ce5-4e83-a53c-d278d19ac7cf: moving from state preparing -> state finished >Thread-15911176::DEBUG::2013-11-19 21:28:49,642::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911176::DEBUG::2013-11-19 21:28:49,643::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911176::DEBUG::2013-11-19 21:28:49,643::task::495::TaskManager.Task::(_debug) Task 6e4def5b-4ce5-4e83-a53c-d278d19ac7cf: ref 0 aborting False >Thread-15911176::INFO::2013-11-19 21:28:49,644::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00042200088501', 'lastCheck': 1384896527.27424, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00242805480957', 'lastCheck': 1384896526.8190501, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896522.1777599, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000473976135254', 'lastCheck': 1384896522.700588, 'valid': True, 'code': 0}} >Thread-15911178::DEBUG::2013-11-19 21:28:50,057::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911178::INFO::2013-11-19 21:28:50,058::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911178::DEBUG::2013-11-19 21:28:50,059::task::495::TaskManager.Task::(_debug) Task 84b2db63-119a-4b4a-b386-285178004e2a: moving from state init -> state preparing >Thread-15911178::DEBUG::2013-11-19 21:28:50,059::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911178::DEBUG::2013-11-19 21:28:50,060::task::495::TaskManager.Task::(_debug) Task 84b2db63-119a-4b4a-b386-285178004e2a: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911178::DEBUG::2013-11-19 21:28:50,061::task::495::TaskManager.Task::(_debug) Task 84b2db63-119a-4b4a-b386-285178004e2a: moving from state preparing -> state finished >Thread-15911178::DEBUG::2013-11-19 21:28:50,061::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911178::DEBUG::2013-11-19 21:28:50,062::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911178::DEBUG::2013-11-19 21:28:50,062::task::495::TaskManager.Task::(_debug) Task 84b2db63-119a-4b4a-b386-285178004e2a: ref 0 aborting False >Thread-15911178::INFO::2013-11-19 21:28:50,063::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911179::DEBUG::2013-11-19 21:28:50,092::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911179::INFO::2013-11-19 21:28:50,093::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911179::DEBUG::2013-11-19 21:28:50,094::task::495::TaskManager.Task::(_debug) Task 5b9d8ec2-0640-41b9-a9e8-26d1db3ba8bd: moving from state init -> state preparing >Thread-15911179::DEBUG::2013-11-19 21:28:50,095::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6030f192-0342-405e-a85f-609c3b41125f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911179::DEBUG::2013-11-19 21:28:50,096::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911179::DEBUG::2013-11-19 21:28:50,097::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911179::DEBUG::2013-11-19 21:28:50,097::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6030f192-0342-405e-a85f-609c3b41125f`::Granted request >Thread-15911179::DEBUG::2013-11-19 21:28:50,098::task::495::TaskManager.Task::(_debug) Task 5b9d8ec2-0640-41b9-a9e8-26d1db3ba8bd: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911179::DEBUG::2013-11-19 21:28:50,099::task::495::TaskManager.Task::(_debug) Task 5b9d8ec2-0640-41b9-a9e8-26d1db3ba8bd: ref 1 aborting False >Thread-15911179::WARNING::2013-11-19 21:28:50,119::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911179::DEBUG::2013-11-19 21:28:50,135::task::495::TaskManager.Task::(_debug) Task 5b9d8ec2-0640-41b9-a9e8-26d1db3ba8bd: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911179::DEBUG::2013-11-19 21:28:50,137::task::495::TaskManager.Task::(_debug) Task 5b9d8ec2-0640-41b9-a9e8-26d1db3ba8bd: moving from state preparing -> state finished >Thread-15911179::DEBUG::2013-11-19 21:28:50,138::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911179::DEBUG::2013-11-19 21:28:50,139::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911179::DEBUG::2013-11-19 21:28:50,139::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911179::DEBUG::2013-11-19 21:28:50,140::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911179::DEBUG::2013-11-19 21:28:50,140::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911179::DEBUG::2013-11-19 21:28:50,141::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911179::DEBUG::2013-11-19 21:28:50,142::task::495::TaskManager.Task::(_debug) Task 5b9d8ec2-0640-41b9-a9e8-26d1db3ba8bd: ref 0 aborting False >Thread-15911179::INFO::2013-11-19 21:28:50,142::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:28:50,708::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:51,215::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.078533 s, 13.0 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:28:52,385::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:28:53,231::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:53,760::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.072534 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:28:55,779::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:56,328::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0741464 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:28:58,344::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:28:58,866::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.073965 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15911184::INFO::2013-11-19 21:29:00,098::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911184::DEBUG::2013-11-19 21:29:00,100::task::495::TaskManager.Task::(_debug) Task 17bbdf95-6cae-4d79-bbc7-039c05254b85: moving from state init -> state preparing >Thread-15911184::DEBUG::2013-11-19 21:29:00,100::task::495::TaskManager.Task::(_debug) Task 17bbdf95-6cae-4d79-bbc7-039c05254b85: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000375032424927', 'lastCheck': 1384896537.4444809, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154209136963', 'lastCheck': 1384896537.0167899, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896532.3859661, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000632047653198', 'lastCheck': 1384896532.9112451, 'valid': True, 'code': 0}} >Thread-15911184::DEBUG::2013-11-19 21:29:00,106::task::495::TaskManager.Task::(_debug) Task 17bbdf95-6cae-4d79-bbc7-039c05254b85: moving from state preparing -> state finished >Thread-15911184::DEBUG::2013-11-19 21:29:00,111::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911184::DEBUG::2013-11-19 21:29:00,112::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911184::DEBUG::2013-11-19 21:29:00,112::task::495::TaskManager.Task::(_debug) Task 17bbdf95-6cae-4d79-bbc7-039c05254b85: ref 0 aborting False >Thread-15911184::INFO::2013-11-19 21:29:00,113::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000375032424927', 'lastCheck': 1384896537.4444809, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154209136963', 'lastCheck': 1384896537.0167899, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896532.3859661, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000632047653198', 'lastCheck': 1384896532.9112451, 'valid': True, 'code': 0}} >Thread-15911186::DEBUG::2013-11-19 21:29:00,264::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911186::INFO::2013-11-19 21:29:00,265::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911186::DEBUG::2013-11-19 21:29:00,266::task::495::TaskManager.Task::(_debug) Task 50bbbbfc-b217-4797-9b9d-97aea2a38f7c: moving from state init -> state preparing >Thread-15911186::DEBUG::2013-11-19 21:29:00,266::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911186::DEBUG::2013-11-19 21:29:00,272::task::495::TaskManager.Task::(_debug) Task 50bbbbfc-b217-4797-9b9d-97aea2a38f7c: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911186::DEBUG::2013-11-19 21:29:00,272::task::495::TaskManager.Task::(_debug) Task 50bbbbfc-b217-4797-9b9d-97aea2a38f7c: moving from state preparing -> state finished >Thread-15911186::DEBUG::2013-11-19 21:29:00,273::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911186::DEBUG::2013-11-19 21:29:00,273::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911186::DEBUG::2013-11-19 21:29:00,274::task::495::TaskManager.Task::(_debug) Task 50bbbbfc-b217-4797-9b9d-97aea2a38f7c: ref 0 aborting False >Thread-15911186::INFO::2013-11-19 21:29:00,275::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911187::DEBUG::2013-11-19 21:29:00,296::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911187::INFO::2013-11-19 21:29:00,297::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911187::DEBUG::2013-11-19 21:29:00,298::task::495::TaskManager.Task::(_debug) Task bbd37f0b-32ea-45f6-9044-5965ca1aaa1e: moving from state init -> state preparing >Thread-15911187::DEBUG::2013-11-19 21:29:00,302::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e23cfff4-c5b0-4719-8d0b-7190e2bc59dc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911187::DEBUG::2013-11-19 21:29:00,303::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911187::DEBUG::2013-11-19 21:29:00,304::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911187::DEBUG::2013-11-19 21:29:00,304::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e23cfff4-c5b0-4719-8d0b-7190e2bc59dc`::Granted request >Thread-15911187::DEBUG::2013-11-19 21:29:00,312::task::495::TaskManager.Task::(_debug) Task bbd37f0b-32ea-45f6-9044-5965ca1aaa1e: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911187::DEBUG::2013-11-19 21:29:00,313::task::495::TaskManager.Task::(_debug) Task bbd37f0b-32ea-45f6-9044-5965ca1aaa1e: ref 1 aborting False >Thread-15911187::WARNING::2013-11-19 21:29:00,320::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911187::DEBUG::2013-11-19 21:29:00,321::task::495::TaskManager.Task::(_debug) Task bbd37f0b-32ea-45f6-9044-5965ca1aaa1e: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911187::DEBUG::2013-11-19 21:29:00,328::task::495::TaskManager.Task::(_debug) Task bbd37f0b-32ea-45f6-9044-5965ca1aaa1e: moving from state preparing -> state finished >Thread-15911187::DEBUG::2013-11-19 21:29:00,328::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911187::DEBUG::2013-11-19 21:29:00,333::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911187::DEBUG::2013-11-19 21:29:00,334::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911187::DEBUG::2013-11-19 21:29:00,336::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911187::DEBUG::2013-11-19 21:29:00,337::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911187::DEBUG::2013-11-19 21:29:00,337::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911187::DEBUG::2013-11-19 21:29:00,338::task::495::TaskManager.Task::(_debug) Task bbd37f0b-32ea-45f6-9044-5965ca1aaa1e: ref 0 aborting False >Thread-15911187::INFO::2013-11-19 21:29:00,338::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:29:00,880::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:01,431::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.113469 s, 9.0 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:29:02,555::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:29:03,446::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:03,942::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0504075 s, 20.3 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:29:05,952::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:06,497::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0834511 s, 12.3 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:29:08,513::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:09,060::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0742441 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15911192::INFO::2013-11-19 21:29:10,301::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911192::DEBUG::2013-11-19 21:29:10,310::task::495::TaskManager.Task::(_debug) Task c870f476-ede0-4b64-89ee-78415e6ae56b: moving from state init -> state preparing >Thread-15911192::DEBUG::2013-11-19 21:29:10,321::task::495::TaskManager.Task::(_debug) Task c870f476-ede0-4b64-89ee-78415e6ae56b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000420093536377', 'lastCheck': 1384896547.6862071, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00173401832581', 'lastCheck': 1384896547.1792181, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896542.556654, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00122213363647', 'lastCheck': 1384896543.0787611, 'valid': True, 'code': 0}} >Thread-15911192::DEBUG::2013-11-19 21:29:10,330::task::495::TaskManager.Task::(_debug) Task c870f476-ede0-4b64-89ee-78415e6ae56b: moving from state preparing -> state finished >Thread-15911192::DEBUG::2013-11-19 21:29:10,331::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911192::DEBUG::2013-11-19 21:29:10,336::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911192::DEBUG::2013-11-19 21:29:10,337::task::495::TaskManager.Task::(_debug) Task c870f476-ede0-4b64-89ee-78415e6ae56b: ref 0 aborting False >Thread-15911192::INFO::2013-11-19 21:29:10,338::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000420093536377', 'lastCheck': 1384896547.6862071, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00173401832581', 'lastCheck': 1384896547.1792181, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896542.556654, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00122213363647', 'lastCheck': 1384896543.0787611, 'valid': True, 'code': 0}} >Thread-15911194::DEBUG::2013-11-19 21:29:10,467::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911194::INFO::2013-11-19 21:29:10,468::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911194::DEBUG::2013-11-19 21:29:10,469::task::495::TaskManager.Task::(_debug) Task 2faa87e3-7948-40b8-bad3-b8c871590e18: moving from state init -> state preparing >Thread-15911194::DEBUG::2013-11-19 21:29:10,470::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911194::DEBUG::2013-11-19 21:29:10,470::task::495::TaskManager.Task::(_debug) Task 2faa87e3-7948-40b8-bad3-b8c871590e18: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911194::DEBUG::2013-11-19 21:29:10,471::task::495::TaskManager.Task::(_debug) Task 2faa87e3-7948-40b8-bad3-b8c871590e18: moving from state preparing -> state finished >Thread-15911194::DEBUG::2013-11-19 21:29:10,471::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911194::DEBUG::2013-11-19 21:29:10,472::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911194::DEBUG::2013-11-19 21:29:10,472::task::495::TaskManager.Task::(_debug) Task 2faa87e3-7948-40b8-bad3-b8c871590e18: ref 0 aborting False >Thread-15911194::INFO::2013-11-19 21:29:10,473::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911195::DEBUG::2013-11-19 21:29:10,502::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911195::INFO::2013-11-19 21:29:10,503::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911195::DEBUG::2013-11-19 21:29:10,503::task::495::TaskManager.Task::(_debug) Task 9455834e-1402-434e-bb55-0c4de5d479eb: moving from state init -> state preparing >Thread-15911195::DEBUG::2013-11-19 21:29:10,508::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fbe21839-a72a-4de5-8a10-e4df67e61427`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911195::DEBUG::2013-11-19 21:29:10,509::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911195::DEBUG::2013-11-19 21:29:10,510::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911195::DEBUG::2013-11-19 21:29:10,510::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fbe21839-a72a-4de5-8a10-e4df67e61427`::Granted request >Thread-15911195::DEBUG::2013-11-19 21:29:10,515::task::495::TaskManager.Task::(_debug) Task 9455834e-1402-434e-bb55-0c4de5d479eb: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911195::DEBUG::2013-11-19 21:29:10,524::task::495::TaskManager.Task::(_debug) Task 9455834e-1402-434e-bb55-0c4de5d479eb: ref 1 aborting False >Thread-15911195::WARNING::2013-11-19 21:29:10,535::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911195::DEBUG::2013-11-19 21:29:10,542::task::495::TaskManager.Task::(_debug) Task 9455834e-1402-434e-bb55-0c4de5d479eb: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911195::DEBUG::2013-11-19 21:29:10,543::task::495::TaskManager.Task::(_debug) Task 9455834e-1402-434e-bb55-0c4de5d479eb: moving from state preparing -> state finished >Thread-15911195::DEBUG::2013-11-19 21:29:10,543::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911195::DEBUG::2013-11-19 21:29:10,544::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911195::DEBUG::2013-11-19 21:29:10,544::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911195::DEBUG::2013-11-19 21:29:10,545::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911195::DEBUG::2013-11-19 21:29:10,546::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911195::DEBUG::2013-11-19 21:29:10,546::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911195::DEBUG::2013-11-19 21:29:10,547::task::495::TaskManager.Task::(_debug) Task 9455834e-1402-434e-bb55-0c4de5d479eb: ref 0 aborting False >Thread-15911195::INFO::2013-11-19 21:29:10,551::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:29:11,075::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:11,605::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0744361 s, 13.8 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:29:12,750::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:29:13,621::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:14,144::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0737296 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:29:16,159::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:16,713::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0741607 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:29:18,729::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:19,259::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0732013 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15911200::INFO::2013-11-19 21:29:20,513::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911200::DEBUG::2013-11-19 21:29:20,513::task::495::TaskManager.Task::(_debug) Task 243c7b4d-497d-4671-96aa-8086923f68d8: moving from state init -> state preparing >Thread-15911200::DEBUG::2013-11-19 21:29:20,514::task::495::TaskManager.Task::(_debug) Task 243c7b4d-497d-4671-96aa-8086923f68d8: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000432968139648', 'lastCheck': 1384896557.831037, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150394439697', 'lastCheck': 1384896557.398869, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896552.751441, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000511884689331', 'lastCheck': 1384896553.289361, 'valid': True, 'code': 0}} >Thread-15911200::DEBUG::2013-11-19 21:29:20,521::task::495::TaskManager.Task::(_debug) Task 243c7b4d-497d-4671-96aa-8086923f68d8: moving from state preparing -> state finished >Thread-15911200::DEBUG::2013-11-19 21:29:20,522::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911200::DEBUG::2013-11-19 21:29:20,522::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911200::DEBUG::2013-11-19 21:29:20,523::task::495::TaskManager.Task::(_debug) Task 243c7b4d-497d-4671-96aa-8086923f68d8: ref 0 aborting False >Thread-15911200::INFO::2013-11-19 21:29:20,524::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000432968139648', 'lastCheck': 1384896557.831037, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150394439697', 'lastCheck': 1384896557.398869, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896552.751441, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000511884689331', 'lastCheck': 1384896553.289361, 'valid': True, 'code': 0}} >Thread-15911202::DEBUG::2013-11-19 21:29:20,648::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911202::INFO::2013-11-19 21:29:20,649::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911202::DEBUG::2013-11-19 21:29:20,649::task::495::TaskManager.Task::(_debug) Task 8b995671-e105-4ba6-aa69-2c03d896064e: moving from state init -> state preparing >Thread-15911202::DEBUG::2013-11-19 21:29:20,650::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911202::DEBUG::2013-11-19 21:29:20,651::task::495::TaskManager.Task::(_debug) Task 8b995671-e105-4ba6-aa69-2c03d896064e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911202::DEBUG::2013-11-19 21:29:20,651::task::495::TaskManager.Task::(_debug) Task 8b995671-e105-4ba6-aa69-2c03d896064e: moving from state preparing -> state finished >Thread-15911202::DEBUG::2013-11-19 21:29:20,652::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911202::DEBUG::2013-11-19 21:29:20,652::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911202::DEBUG::2013-11-19 21:29:20,653::task::495::TaskManager.Task::(_debug) Task 8b995671-e105-4ba6-aa69-2c03d896064e: ref 0 aborting False >Thread-15911202::INFO::2013-11-19 21:29:20,654::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911203::DEBUG::2013-11-19 21:29:20,671::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911203::INFO::2013-11-19 21:29:20,671::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911203::DEBUG::2013-11-19 21:29:20,672::task::495::TaskManager.Task::(_debug) Task 05a52452-6fc4-4232-95e0-7f4221ab9baf: moving from state init -> state preparing >Thread-15911203::DEBUG::2013-11-19 21:29:20,676::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`0769aada-d4f7-465d-b487-047ed99e1206`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911203::DEBUG::2013-11-19 21:29:20,677::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911203::DEBUG::2013-11-19 21:29:20,678::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911203::DEBUG::2013-11-19 21:29:20,678::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`0769aada-d4f7-465d-b487-047ed99e1206`::Granted request >Thread-15911203::DEBUG::2013-11-19 21:29:20,682::task::495::TaskManager.Task::(_debug) Task 05a52452-6fc4-4232-95e0-7f4221ab9baf: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911203::DEBUG::2013-11-19 21:29:20,682::task::495::TaskManager.Task::(_debug) Task 05a52452-6fc4-4232-95e0-7f4221ab9baf: ref 1 aborting False >Thread-15911203::WARNING::2013-11-19 21:29:20,688::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911203::DEBUG::2013-11-19 21:29:20,692::task::495::TaskManager.Task::(_debug) Task 05a52452-6fc4-4232-95e0-7f4221ab9baf: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911203::DEBUG::2013-11-19 21:29:20,693::task::495::TaskManager.Task::(_debug) Task 05a52452-6fc4-4232-95e0-7f4221ab9baf: moving from state preparing -> state finished >Thread-15911203::DEBUG::2013-11-19 21:29:20,694::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911203::DEBUG::2013-11-19 21:29:20,694::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911203::DEBUG::2013-11-19 21:29:20,700::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911203::DEBUG::2013-11-19 21:29:20,700::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911203::DEBUG::2013-11-19 21:29:20,701::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911203::DEBUG::2013-11-19 21:29:20,701::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911203::DEBUG::2013-11-19 21:29:20,702::task::495::TaskManager.Task::(_debug) Task 05a52452-6fc4-4232-95e0-7f4221ab9baf: ref 0 aborting False >Thread-15911203::INFO::2013-11-19 21:29:20,703::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:29:21,275::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:21,822::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0737275 s, 13.9 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:29:22,946::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:29:23,837::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:24,369::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0742101 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:29:26,385::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:26,915::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0732197 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:29:28,930::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:29,473::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.073984 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15911208::INFO::2013-11-19 21:29:30,706::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911208::DEBUG::2013-11-19 21:29:30,707::task::495::TaskManager.Task::(_debug) Task 4fef6051-efeb-4955-9a6a-808a3dd50d17: moving from state init -> state preparing >Thread-15911208::DEBUG::2013-11-19 21:29:30,708::task::495::TaskManager.Task::(_debug) Task 4fef6051-efeb-4955-9a6a-808a3dd50d17: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000375986099243', 'lastCheck': 1384896568.057802, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00149703025818', 'lastCheck': 1384896567.612828, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896562.947335, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000591039657593', 'lastCheck': 1384896563.514246, 'valid': True, 'code': 0}} >Thread-15911208::DEBUG::2013-11-19 21:29:30,720::task::495::TaskManager.Task::(_debug) Task 4fef6051-efeb-4955-9a6a-808a3dd50d17: moving from state preparing -> state finished >Thread-15911208::DEBUG::2013-11-19 21:29:30,721::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911208::DEBUG::2013-11-19 21:29:30,721::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911208::DEBUG::2013-11-19 21:29:30,722::task::495::TaskManager.Task::(_debug) Task 4fef6051-efeb-4955-9a6a-808a3dd50d17: ref 0 aborting False >Thread-15911208::INFO::2013-11-19 21:29:30,723::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000375986099243', 'lastCheck': 1384896568.057802, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00149703025818', 'lastCheck': 1384896567.612828, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896562.947335, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000591039657593', 'lastCheck': 1384896563.514246, 'valid': True, 'code': 0}} >Thread-15911210::DEBUG::2013-11-19 21:29:30,818::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911210::INFO::2013-11-19 21:29:30,819::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911210::DEBUG::2013-11-19 21:29:30,820::task::495::TaskManager.Task::(_debug) Task ec6b77b9-995a-4132-8b5f-63b9c202c290: moving from state init -> state preparing >Thread-15911210::DEBUG::2013-11-19 21:29:30,821::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911210::DEBUG::2013-11-19 21:29:30,821::task::495::TaskManager.Task::(_debug) Task ec6b77b9-995a-4132-8b5f-63b9c202c290: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911210::DEBUG::2013-11-19 21:29:30,826::task::495::TaskManager.Task::(_debug) Task ec6b77b9-995a-4132-8b5f-63b9c202c290: moving from state preparing -> state finished >Thread-15911210::DEBUG::2013-11-19 21:29:30,826::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911210::DEBUG::2013-11-19 21:29:30,827::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911210::DEBUG::2013-11-19 21:29:30,827::task::495::TaskManager.Task::(_debug) Task ec6b77b9-995a-4132-8b5f-63b9c202c290: ref 0 aborting False >Thread-15911210::INFO::2013-11-19 21:29:30,828::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911211::DEBUG::2013-11-19 21:29:30,836::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911211::INFO::2013-11-19 21:29:30,837::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911211::DEBUG::2013-11-19 21:29:30,837::task::495::TaskManager.Task::(_debug) Task 46aeb565-bf01-4954-9fcf-b753dc54889e: moving from state init -> state preparing >Thread-15911211::DEBUG::2013-11-19 21:29:30,842::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`914e6be4-e0db-49bd-843a-2e0128a87ba7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911211::DEBUG::2013-11-19 21:29:30,843::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911211::DEBUG::2013-11-19 21:29:30,843::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911211::DEBUG::2013-11-19 21:29:30,844::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`914e6be4-e0db-49bd-843a-2e0128a87ba7`::Granted request >Thread-15911211::DEBUG::2013-11-19 21:29:30,850::task::495::TaskManager.Task::(_debug) Task 46aeb565-bf01-4954-9fcf-b753dc54889e: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911211::DEBUG::2013-11-19 21:29:30,851::task::495::TaskManager.Task::(_debug) Task 46aeb565-bf01-4954-9fcf-b753dc54889e: ref 1 aborting False >Thread-15911211::WARNING::2013-11-19 21:29:30,867::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911211::DEBUG::2013-11-19 21:29:30,878::task::495::TaskManager.Task::(_debug) Task 46aeb565-bf01-4954-9fcf-b753dc54889e: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911211::DEBUG::2013-11-19 21:29:30,883::task::495::TaskManager.Task::(_debug) Task 46aeb565-bf01-4954-9fcf-b753dc54889e: moving from state preparing -> state finished >Thread-15911211::DEBUG::2013-11-19 21:29:30,884::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911211::DEBUG::2013-11-19 21:29:30,889::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911211::DEBUG::2013-11-19 21:29:30,890::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911211::DEBUG::2013-11-19 21:29:30,891::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911211::DEBUG::2013-11-19 21:29:30,891::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911211::DEBUG::2013-11-19 21:29:30,892::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911211::DEBUG::2013-11-19 21:29:30,892::task::495::TaskManager.Task::(_debug) Task 46aeb565-bf01-4954-9fcf-b753dc54889e: ref 0 aborting False >Thread-15911211::INFO::2013-11-19 21:29:30,893::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:29:31,488::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:32,007::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.073101 s, 14.0 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:29:33,163::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:29:34,022::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:34,577::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0810506 s, 12.6 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:29:36,592::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:37,124::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0733157 s, 14.0 MB/s\n'; <rc> = 0 >Thread-42838::DEBUG::2013-11-19 21:29:38,256::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 21:29:38,949::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:29:39,148::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:39,693::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0737704 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15911216::INFO::2013-11-19 21:29:40,906::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911216::DEBUG::2013-11-19 21:29:40,907::task::495::TaskManager.Task::(_debug) Task bb5bb847-f7eb-47da-8d7f-c541e071ac60: moving from state init -> state preparing >Thread-15911216::DEBUG::2013-11-19 21:29:40,907::task::495::TaskManager.Task::(_debug) Task bb5bb847-f7eb-47da-8d7f-c541e071ac60: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000382900238037', 'lastCheck': 1384896578.954037, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00160193443298', 'lastCheck': 1384896577.8213, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896573.1647429, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00143504142761', 'lastCheck': 1384896573.6980071, 'valid': True, 'code': 0}} >Thread-15911216::DEBUG::2013-11-19 21:29:40,919::task::495::TaskManager.Task::(_debug) Task bb5bb847-f7eb-47da-8d7f-c541e071ac60: moving from state preparing -> state finished >Thread-15911216::DEBUG::2013-11-19 21:29:40,920::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911216::DEBUG::2013-11-19 21:29:40,920::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911216::DEBUG::2013-11-19 21:29:40,921::task::495::TaskManager.Task::(_debug) Task bb5bb847-f7eb-47da-8d7f-c541e071ac60: ref 0 aborting False >Thread-15911216::INFO::2013-11-19 21:29:40,922::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000382900238037', 'lastCheck': 1384896578.954037, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00160193443298', 'lastCheck': 1384896577.8213, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896573.1647429, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00143504142761', 'lastCheck': 1384896573.6980071, 'valid': True, 'code': 0}} >Thread-15911218::DEBUG::2013-11-19 21:29:40,990::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911218::INFO::2013-11-19 21:29:40,991::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911218::DEBUG::2013-11-19 21:29:40,992::task::495::TaskManager.Task::(_debug) Task 551a0333-816a-42db-aaa5-492441828195: moving from state init -> state preparing >Thread-15911218::DEBUG::2013-11-19 21:29:40,992::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911218::DEBUG::2013-11-19 21:29:40,993::task::495::TaskManager.Task::(_debug) Task 551a0333-816a-42db-aaa5-492441828195: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911218::DEBUG::2013-11-19 21:29:40,994::task::495::TaskManager.Task::(_debug) Task 551a0333-816a-42db-aaa5-492441828195: moving from state preparing -> state finished >Thread-15911218::DEBUG::2013-11-19 21:29:40,994::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911218::DEBUG::2013-11-19 21:29:40,995::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911218::DEBUG::2013-11-19 21:29:40,995::task::495::TaskManager.Task::(_debug) Task 551a0333-816a-42db-aaa5-492441828195: ref 0 aborting False >Thread-15911218::INFO::2013-11-19 21:29:40,996::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911219::DEBUG::2013-11-19 21:29:41,005::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911219::INFO::2013-11-19 21:29:41,006::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911219::DEBUG::2013-11-19 21:29:41,007::task::495::TaskManager.Task::(_debug) Task fed045f3-2aa8-4897-a23f-2eb35bfdd660: moving from state init -> state preparing >Thread-15911219::DEBUG::2013-11-19 21:29:41,008::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`48b4dd5f-3154-4561-a99a-fe0dc0e74914`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911219::DEBUG::2013-11-19 21:29:41,013::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911219::DEBUG::2013-11-19 21:29:41,014::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911219::DEBUG::2013-11-19 21:29:41,015::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`48b4dd5f-3154-4561-a99a-fe0dc0e74914`::Granted request >Thread-15911219::DEBUG::2013-11-19 21:29:41,022::task::495::TaskManager.Task::(_debug) Task fed045f3-2aa8-4897-a23f-2eb35bfdd660: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911219::DEBUG::2013-11-19 21:29:41,027::task::495::TaskManager.Task::(_debug) Task fed045f3-2aa8-4897-a23f-2eb35bfdd660: ref 1 aborting False >Thread-15911219::WARNING::2013-11-19 21:29:41,029::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911219::DEBUG::2013-11-19 21:29:41,044::task::495::TaskManager.Task::(_debug) Task fed045f3-2aa8-4897-a23f-2eb35bfdd660: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911219::DEBUG::2013-11-19 21:29:41,050::task::495::TaskManager.Task::(_debug) Task fed045f3-2aa8-4897-a23f-2eb35bfdd660: moving from state preparing -> state finished >Thread-15911219::DEBUG::2013-11-19 21:29:41,051::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911219::DEBUG::2013-11-19 21:29:41,051::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911219::DEBUG::2013-11-19 21:29:41,052::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911219::DEBUG::2013-11-19 21:29:41,053::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911219::DEBUG::2013-11-19 21:29:41,053::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911219::DEBUG::2013-11-19 21:29:41,054::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911219::DEBUG::2013-11-19 21:29:41,054::task::495::TaskManager.Task::(_debug) Task fed045f3-2aa8-4897-a23f-2eb35bfdd660: ref 0 aborting False >Thread-15911219::INFO::2013-11-19 21:29:41,055::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:29:41,708::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:42,245::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0745469 s, 13.7 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:29:43,384::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:29:44,261::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:44,792::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0734955 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:29:46,808::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:47,343::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0744038 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:29:49,359::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:49,882::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0730842 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15911225::INFO::2013-11-19 21:29:51,161::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911224::DEBUG::2013-11-19 21:29:51,181::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911225::DEBUG::2013-11-19 21:29:51,182::task::495::TaskManager.Task::(_debug) Task 5b2c4d23-b753-4469-9ebc-cb303e68dde5: moving from state init -> state preparing >Thread-15911224::INFO::2013-11-19 21:29:51,187::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911225::DEBUG::2013-11-19 21:29:51,190::task::495::TaskManager.Task::(_debug) Task 5b2c4d23-b753-4469-9ebc-cb303e68dde5: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000433921813965', 'lastCheck': 1384896589.0285299, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00238299369812', 'lastCheck': 1384896588.03268, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896583.384994, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00048303604126', 'lastCheck': 1384896583.92977, 'valid': True, 'code': 0}} >Thread-15911224::DEBUG::2013-11-19 21:29:51,191::task::495::TaskManager.Task::(_debug) Task 82fe740a-f930-4f24-b1a1-2a7c9d94d3de: moving from state init -> state preparing >Thread-15911225::DEBUG::2013-11-19 21:29:51,192::task::495::TaskManager.Task::(_debug) Task 5b2c4d23-b753-4469-9ebc-cb303e68dde5: moving from state preparing -> state finished >Thread-15911224::DEBUG::2013-11-19 21:29:51,193::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911225::DEBUG::2013-11-19 21:29:51,195::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911224::DEBUG::2013-11-19 21:29:51,196::task::495::TaskManager.Task::(_debug) Task 82fe740a-f930-4f24-b1a1-2a7c9d94d3de: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911225::DEBUG::2013-11-19 21:29:51,203::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911224::DEBUG::2013-11-19 21:29:51,204::task::495::TaskManager.Task::(_debug) Task 82fe740a-f930-4f24-b1a1-2a7c9d94d3de: moving from state preparing -> state finished >Thread-15911225::DEBUG::2013-11-19 21:29:51,208::task::495::TaskManager.Task::(_debug) Task 5b2c4d23-b753-4469-9ebc-cb303e68dde5: ref 0 aborting False >Thread-15911224::DEBUG::2013-11-19 21:29:51,208::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911225::INFO::2013-11-19 21:29:51,209::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000433921813965', 'lastCheck': 1384896589.0285299, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00238299369812', 'lastCheck': 1384896588.03268, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896583.384994, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00048303604126', 'lastCheck': 1384896583.92977, 'valid': True, 'code': 0}} >Thread-15911224::DEBUG::2013-11-19 21:29:51,210::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911224::DEBUG::2013-11-19 21:29:51,215::task::495::TaskManager.Task::(_debug) Task 82fe740a-f930-4f24-b1a1-2a7c9d94d3de: ref 0 aborting False >Thread-15911224::INFO::2013-11-19 21:29:51,224::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911226::DEBUG::2013-11-19 21:29:51,243::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911226::INFO::2013-11-19 21:29:51,245::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911226::DEBUG::2013-11-19 21:29:51,245::task::495::TaskManager.Task::(_debug) Task 4d136391-383a-4cde-9508-5755f3a7e225: moving from state init -> state preparing >Thread-15911226::DEBUG::2013-11-19 21:29:51,252::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1d42ae31-8d97-4dee-8793-781dfdff419b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911226::DEBUG::2013-11-19 21:29:51,253::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911226::DEBUG::2013-11-19 21:29:51,254::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911226::DEBUG::2013-11-19 21:29:51,254::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1d42ae31-8d97-4dee-8793-781dfdff419b`::Granted request >Thread-15911226::DEBUG::2013-11-19 21:29:51,260::task::495::TaskManager.Task::(_debug) Task 4d136391-383a-4cde-9508-5755f3a7e225: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911226::DEBUG::2013-11-19 21:29:51,264::task::495::TaskManager.Task::(_debug) Task 4d136391-383a-4cde-9508-5755f3a7e225: ref 1 aborting False >Thread-15911226::WARNING::2013-11-19 21:29:51,266::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911226::DEBUG::2013-11-19 21:29:51,276::task::495::TaskManager.Task::(_debug) Task 4d136391-383a-4cde-9508-5755f3a7e225: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911226::DEBUG::2013-11-19 21:29:51,283::task::495::TaskManager.Task::(_debug) Task 4d136391-383a-4cde-9508-5755f3a7e225: moving from state preparing -> state finished >Thread-15911226::DEBUG::2013-11-19 21:29:51,283::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911226::DEBUG::2013-11-19 21:29:51,284::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911226::DEBUG::2013-11-19 21:29:51,284::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911226::DEBUG::2013-11-19 21:29:51,285::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911226::DEBUG::2013-11-19 21:29:51,285::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911226::DEBUG::2013-11-19 21:29:51,286::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911226::DEBUG::2013-11-19 21:29:51,287::task::495::TaskManager.Task::(_debug) Task 4d136391-383a-4cde-9508-5755f3a7e225: ref 0 aborting False >Thread-15911226::INFO::2013-11-19 21:29:51,287::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:29:51,897::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:52,436::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.073784 s, 13.9 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:29:53,576::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:29:54,451::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:54,988::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.07357 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:29:57,003::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:29:57,527::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0732544 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:29:59,543::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:00,083::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0738056 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15911232::DEBUG::2013-11-19 21:30:01,383::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911232::INFO::2013-11-19 21:30:01,384::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911232::DEBUG::2013-11-19 21:30:01,385::task::495::TaskManager.Task::(_debug) Task 65d430e8-8ff4-4085-bc2a-517a2098a680: moving from state init -> state preparing >Thread-15911232::DEBUG::2013-11-19 21:30:01,386::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911232::DEBUG::2013-11-19 21:30:01,387::task::495::TaskManager.Task::(_debug) Task 65d430e8-8ff4-4085-bc2a-517a2098a680: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911232::DEBUG::2013-11-19 21:30:01,387::task::495::TaskManager.Task::(_debug) Task 65d430e8-8ff4-4085-bc2a-517a2098a680: moving from state preparing -> state finished >Thread-15911232::DEBUG::2013-11-19 21:30:01,388::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911232::DEBUG::2013-11-19 21:30:01,388::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911232::DEBUG::2013-11-19 21:30:01,389::task::495::TaskManager.Task::(_debug) Task 65d430e8-8ff4-4085-bc2a-517a2098a680: ref 0 aborting False >Thread-15911232::INFO::2013-11-19 21:30:01,389::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911234::INFO::2013-11-19 21:30:01,400::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911233::DEBUG::2013-11-19 21:30:01,411::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911234::DEBUG::2013-11-19 21:30:01,412::task::495::TaskManager.Task::(_debug) Task 4e6b7e1d-b865-4b06-b340-7baf10488b03: moving from state init -> state preparing >Thread-15911233::INFO::2013-11-19 21:30:01,413::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911234::DEBUG::2013-11-19 21:30:01,417::task::495::TaskManager.Task::(_debug) Task 4e6b7e1d-b865-4b06-b340-7baf10488b03: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000426054000854', 'lastCheck': 1384896599.2087009, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154185295105', 'lastCheck': 1384896598.210664, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896593.577342, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000489950180054', 'lastCheck': 1384896594.1274059, 'valid': True, 'code': 0}} >Thread-15911233::DEBUG::2013-11-19 21:30:01,418::task::495::TaskManager.Task::(_debug) Task 6cd5d3f8-1470-47c9-8d67-b79e22b4541b: moving from state init -> state preparing >Thread-15911234::DEBUG::2013-11-19 21:30:01,424::task::495::TaskManager.Task::(_debug) Task 4e6b7e1d-b865-4b06-b340-7baf10488b03: moving from state preparing -> state finished >Thread-15911233::DEBUG::2013-11-19 21:30:01,425::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`56490edc-e929-461c-a167-646fa7b729c2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911234::DEBUG::2013-11-19 21:30:01,426::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911233::DEBUG::2013-11-19 21:30:01,427::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911234::DEBUG::2013-11-19 21:30:01,430::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911234::DEBUG::2013-11-19 21:30:01,431::task::495::TaskManager.Task::(_debug) Task 4e6b7e1d-b865-4b06-b340-7baf10488b03: ref 0 aborting False >Thread-15911233::DEBUG::2013-11-19 21:30:01,432::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911234::INFO::2013-11-19 21:30:01,432::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000426054000854', 'lastCheck': 1384896599.2087009, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154185295105', 'lastCheck': 1384896598.210664, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896593.577342, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000489950180054', 'lastCheck': 1384896594.1274059, 'valid': True, 'code': 0}} >Thread-15911233::DEBUG::2013-11-19 21:30:01,433::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`56490edc-e929-461c-a167-646fa7b729c2`::Granted request >Thread-15911233::DEBUG::2013-11-19 21:30:01,439::task::495::TaskManager.Task::(_debug) Task 6cd5d3f8-1470-47c9-8d67-b79e22b4541b: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911233::DEBUG::2013-11-19 21:30:01,446::task::495::TaskManager.Task::(_debug) Task 6cd5d3f8-1470-47c9-8d67-b79e22b4541b: ref 1 aborting False >Thread-15911233::WARNING::2013-11-19 21:30:01,449::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911233::DEBUG::2013-11-19 21:30:01,453::task::495::TaskManager.Task::(_debug) Task 6cd5d3f8-1470-47c9-8d67-b79e22b4541b: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911233::DEBUG::2013-11-19 21:30:01,455::task::495::TaskManager.Task::(_debug) Task 6cd5d3f8-1470-47c9-8d67-b79e22b4541b: moving from state preparing -> state finished >Thread-15911233::DEBUG::2013-11-19 21:30:01,455::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911233::DEBUG::2013-11-19 21:30:01,456::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911233::DEBUG::2013-11-19 21:30:01,463::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911233::DEBUG::2013-11-19 21:30:01,464::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911233::DEBUG::2013-11-19 21:30:01,464::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911233::DEBUG::2013-11-19 21:30:01,465::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911233::DEBUG::2013-11-19 21:30:01,465::task::495::TaskManager.Task::(_debug) Task 6cd5d3f8-1470-47c9-8d67-b79e22b4541b: ref 0 aborting False >Thread-15911233::INFO::2013-11-19 21:30:01,466::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:30:02,099::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:02,620::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0739173 s, 13.9 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:30:03,771::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:30:04,635::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:05,159::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0729323 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:30:07,175::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:07,741::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.108079 s, 9.5 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:30:09,755::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:10,275::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0733282 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15911240::DEBUG::2013-11-19 21:30:11,591::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911240::INFO::2013-11-19 21:30:11,592::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911240::DEBUG::2013-11-19 21:30:11,593::task::495::TaskManager.Task::(_debug) Task be565fa4-5847-4507-8101-d2a2078a1d4a: moving from state init -> state preparing >Thread-15911240::DEBUG::2013-11-19 21:30:11,603::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911240::DEBUG::2013-11-19 21:30:11,603::task::495::TaskManager.Task::(_debug) Task be565fa4-5847-4507-8101-d2a2078a1d4a: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911240::DEBUG::2013-11-19 21:30:11,604::task::495::TaskManager.Task::(_debug) Task be565fa4-5847-4507-8101-d2a2078a1d4a: moving from state preparing -> state finished >Thread-15911240::DEBUG::2013-11-19 21:30:11,605::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911240::DEBUG::2013-11-19 21:30:11,605::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911240::DEBUG::2013-11-19 21:30:11,606::task::495::TaskManager.Task::(_debug) Task be565fa4-5847-4507-8101-d2a2078a1d4a: ref 0 aborting False >Thread-15911240::INFO::2013-11-19 21:30:11,606::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911241::DEBUG::2013-11-19 21:30:11,622::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911241::INFO::2013-11-19 21:30:11,623::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911241::DEBUG::2013-11-19 21:30:11,624::task::495::TaskManager.Task::(_debug) Task ddcd7a5b-3db5-4f0a-b3d1-6a0ad628fa0a: moving from state init -> state preparing >Thread-15911241::DEBUG::2013-11-19 21:30:11,628::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`14b5c650-e8f7-43ed-81cc-f2da01d0b5c1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911242::INFO::2013-11-19 21:30:11,630::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911241::DEBUG::2013-11-19 21:30:11,631::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911242::DEBUG::2013-11-19 21:30:11,640::task::495::TaskManager.Task::(_debug) Task 1909219e-2299-499b-9d54-451f9e61a39b: moving from state init -> state preparing >Thread-15911242::DEBUG::2013-11-19 21:30:11,641::task::495::TaskManager.Task::(_debug) Task 1909219e-2299-499b-9d54-451f9e61a39b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000590085983276', 'lastCheck': 1384896609.3994031, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00155782699585', 'lastCheck': 1384896608.397249, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896603.771982, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000539064407349', 'lastCheck': 1384896604.305979, 'valid': True, 'code': 0}} >Thread-15911241::DEBUG::2013-11-19 21:30:11,642::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911242::DEBUG::2013-11-19 21:30:11,643::task::495::TaskManager.Task::(_debug) Task 1909219e-2299-499b-9d54-451f9e61a39b: moving from state preparing -> state finished >Thread-15911241::DEBUG::2013-11-19 21:30:11,644::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`14b5c650-e8f7-43ed-81cc-f2da01d0b5c1`::Granted request >Thread-15911242::DEBUG::2013-11-19 21:30:11,648::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911241::DEBUG::2013-11-19 21:30:11,654::task::495::TaskManager.Task::(_debug) Task ddcd7a5b-3db5-4f0a-b3d1-6a0ad628fa0a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911242::DEBUG::2013-11-19 21:30:11,654::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911241::DEBUG::2013-11-19 21:30:11,659::task::495::TaskManager.Task::(_debug) Task ddcd7a5b-3db5-4f0a-b3d1-6a0ad628fa0a: ref 1 aborting False >Thread-15911242::DEBUG::2013-11-19 21:30:11,662::task::495::TaskManager.Task::(_debug) Task 1909219e-2299-499b-9d54-451f9e61a39b: ref 0 aborting False >Thread-15911241::WARNING::2013-11-19 21:30:11,669::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911242::INFO::2013-11-19 21:30:11,671::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000590085983276', 'lastCheck': 1384896609.3994031, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00155782699585', 'lastCheck': 1384896608.397249, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896603.771982, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000539064407349', 'lastCheck': 1384896604.305979, 'valid': True, 'code': 0}} >Thread-15911241::DEBUG::2013-11-19 21:30:11,680::task::495::TaskManager.Task::(_debug) Task ddcd7a5b-3db5-4f0a-b3d1-6a0ad628fa0a: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911241::DEBUG::2013-11-19 21:30:11,693::task::495::TaskManager.Task::(_debug) Task ddcd7a5b-3db5-4f0a-b3d1-6a0ad628fa0a: moving from state preparing -> state finished >Thread-15911241::DEBUG::2013-11-19 21:30:11,702::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911241::DEBUG::2013-11-19 21:30:11,703::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911241::DEBUG::2013-11-19 21:30:11,704::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911241::DEBUG::2013-11-19 21:30:11,705::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911241::DEBUG::2013-11-19 21:30:11,705::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911241::DEBUG::2013-11-19 21:30:11,706::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911241::DEBUG::2013-11-19 21:30:11,707::task::495::TaskManager.Task::(_debug) Task ddcd7a5b-3db5-4f0a-b3d1-6a0ad628fa0a: ref 0 aborting False >Thread-15911241::INFO::2013-11-19 21:30:11,708::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:30:12,290::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:12,776::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.072332 s, 14.2 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:30:13,980::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:30:14,791::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:15,327::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0747574 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:30:17,343::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:17,766::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0449244 s, 22.8 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:30:19,776::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:20,313::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0716356 s, 14.3 MB/s\n'; <rc> = 0 >Thread-15911248::DEBUG::2013-11-19 21:30:21,844::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911248::INFO::2013-11-19 21:30:21,845::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911248::DEBUG::2013-11-19 21:30:21,853::task::495::TaskManager.Task::(_debug) Task 986df438-0433-4f65-8e9a-7dc70dff5195: moving from state init -> state preparing >Thread-15911248::DEBUG::2013-11-19 21:30:21,858::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911248::DEBUG::2013-11-19 21:30:21,862::task::495::TaskManager.Task::(_debug) Task 986df438-0433-4f65-8e9a-7dc70dff5195: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911248::DEBUG::2013-11-19 21:30:21,863::task::495::TaskManager.Task::(_debug) Task 986df438-0433-4f65-8e9a-7dc70dff5195: moving from state preparing -> state finished >Thread-15911248::DEBUG::2013-11-19 21:30:21,867::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911248::DEBUG::2013-11-19 21:30:21,868::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911248::DEBUG::2013-11-19 21:30:21,868::task::495::TaskManager.Task::(_debug) Task 986df438-0433-4f65-8e9a-7dc70dff5195: ref 0 aborting False >Thread-15911248::INFO::2013-11-19 21:30:21,869::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911249::DEBUG::2013-11-19 21:30:21,876::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911249::INFO::2013-11-19 21:30:21,877::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911249::DEBUG::2013-11-19 21:30:21,878::task::495::TaskManager.Task::(_debug) Task f6118835-431d-4d2e-b3b5-7a633ce62a07: moving from state init -> state preparing >Thread-15911249::DEBUG::2013-11-19 21:30:21,879::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2737d3bb-a430-4e4a-99e9-aa31755e835a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911249::DEBUG::2013-11-19 21:30:21,884::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911249::DEBUG::2013-11-19 21:30:21,884::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911249::DEBUG::2013-11-19 21:30:21,885::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2737d3bb-a430-4e4a-99e9-aa31755e835a`::Granted request >Thread-15911249::DEBUG::2013-11-19 21:30:21,886::task::495::TaskManager.Task::(_debug) Task f6118835-431d-4d2e-b3b5-7a633ce62a07: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911249::DEBUG::2013-11-19 21:30:21,887::task::495::TaskManager.Task::(_debug) Task f6118835-431d-4d2e-b3b5-7a633ce62a07: ref 1 aborting False >Thread-15911249::WARNING::2013-11-19 21:30:21,889::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911249::DEBUG::2013-11-19 21:30:21,893::task::495::TaskManager.Task::(_debug) Task f6118835-431d-4d2e-b3b5-7a633ce62a07: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911249::DEBUG::2013-11-19 21:30:21,898::task::495::TaskManager.Task::(_debug) Task f6118835-431d-4d2e-b3b5-7a633ce62a07: moving from state preparing -> state finished >Thread-15911249::DEBUG::2013-11-19 21:30:21,907::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911249::DEBUG::2013-11-19 21:30:21,913::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911249::DEBUG::2013-11-19 21:30:21,917::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911249::DEBUG::2013-11-19 21:30:21,917::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911249::DEBUG::2013-11-19 21:30:21,921::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911249::DEBUG::2013-11-19 21:30:21,922::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911249::DEBUG::2013-11-19 21:30:21,927::task::495::TaskManager.Task::(_debug) Task f6118835-431d-4d2e-b3b5-7a633ce62a07: ref 0 aborting False >Thread-15911249::INFO::2013-11-19 21:30:21,933::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911250::INFO::2013-11-19 21:30:22,021::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911250::DEBUG::2013-11-19 21:30:22,021::task::495::TaskManager.Task::(_debug) Task 45a3321e-a585-4c7e-ae61-e53bb6fa61a9: moving from state init -> state preparing >Thread-15911250::DEBUG::2013-11-19 21:30:22,022::task::495::TaskManager.Task::(_debug) Task 45a3321e-a585-4c7e-ae61-e53bb6fa61a9: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000437021255493', 'lastCheck': 1384896619.554714, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00155997276306', 'lastCheck': 1384896618.556078, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896613.9808481, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000546932220459', 'lastCheck': 1384896614.5314109, 'valid': True, 'code': 0}} >Thread-15911250::DEBUG::2013-11-19 21:30:22,023::task::495::TaskManager.Task::(_debug) Task 45a3321e-a585-4c7e-ae61-e53bb6fa61a9: moving from state preparing -> state finished >Thread-15911250::DEBUG::2013-11-19 21:30:22,028::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911250::DEBUG::2013-11-19 21:30:22,031::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911250::DEBUG::2013-11-19 21:30:22,037::task::495::TaskManager.Task::(_debug) Task 45a3321e-a585-4c7e-ae61-e53bb6fa61a9: ref 0 aborting False >Thread-15911250::INFO::2013-11-19 21:30:22,041::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000437021255493', 'lastCheck': 1384896619.554714, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00155997276306', 'lastCheck': 1384896618.556078, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896613.9808481, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000546932220459', 'lastCheck': 1384896614.5314109, 'valid': True, 'code': 0}} >Dummy-15911115::DEBUG::2013-11-19 21:30:22,328::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:22,863::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0716847 s, 14.3 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:30:24,022::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:30:24,877::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:25,389::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0706514 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:30:27,404::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:27,953::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0740763 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:30:29,969::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:30,498::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0738606 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15911256::DEBUG::2013-11-19 21:30:32,054::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911256::INFO::2013-11-19 21:30:32,064::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911256::DEBUG::2013-11-19 21:30:32,065::task::495::TaskManager.Task::(_debug) Task 68828aa5-6635-4490-997a-b773e1cc716e: moving from state init -> state preparing >Thread-15911256::DEBUG::2013-11-19 21:30:32,070::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911256::DEBUG::2013-11-19 21:30:32,074::task::495::TaskManager.Task::(_debug) Task 68828aa5-6635-4490-997a-b773e1cc716e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911256::DEBUG::2013-11-19 21:30:32,077::task::495::TaskManager.Task::(_debug) Task 68828aa5-6635-4490-997a-b773e1cc716e: moving from state preparing -> state finished >Thread-15911256::DEBUG::2013-11-19 21:30:32,089::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911256::DEBUG::2013-11-19 21:30:32,094::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911256::DEBUG::2013-11-19 21:30:32,095::task::495::TaskManager.Task::(_debug) Task 68828aa5-6635-4490-997a-b773e1cc716e: ref 0 aborting False >Thread-15911256::INFO::2013-11-19 21:30:32,096::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911257::DEBUG::2013-11-19 21:30:32,101::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911257::INFO::2013-11-19 21:30:32,102::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911257::DEBUG::2013-11-19 21:30:32,103::task::495::TaskManager.Task::(_debug) Task 1d52166f-58fd-4800-af92-b41a4fbdb019: moving from state init -> state preparing >Thread-15911257::DEBUG::2013-11-19 21:30:32,104::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`06360c5e-2a3b-4c55-87e0-21508008b549`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911257::DEBUG::2013-11-19 21:30:32,104::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911257::DEBUG::2013-11-19 21:30:32,105::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911257::DEBUG::2013-11-19 21:30:32,106::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`06360c5e-2a3b-4c55-87e0-21508008b549`::Granted request >Thread-15911257::DEBUG::2013-11-19 21:30:32,107::task::495::TaskManager.Task::(_debug) Task 1d52166f-58fd-4800-af92-b41a4fbdb019: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911257::DEBUG::2013-11-19 21:30:32,108::task::495::TaskManager.Task::(_debug) Task 1d52166f-58fd-4800-af92-b41a4fbdb019: ref 1 aborting False >Thread-15911257::WARNING::2013-11-19 21:30:32,114::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911257::DEBUG::2013-11-19 21:30:32,122::task::495::TaskManager.Task::(_debug) Task 1d52166f-58fd-4800-af92-b41a4fbdb019: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911257::DEBUG::2013-11-19 21:30:32,126::task::495::TaskManager.Task::(_debug) Task 1d52166f-58fd-4800-af92-b41a4fbdb019: moving from state preparing -> state finished >Thread-15911257::DEBUG::2013-11-19 21:30:32,127::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911257::DEBUG::2013-11-19 21:30:32,127::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911257::DEBUG::2013-11-19 21:30:32,128::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911257::DEBUG::2013-11-19 21:30:32,129::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911257::DEBUG::2013-11-19 21:30:32,129::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911257::DEBUG::2013-11-19 21:30:32,130::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911257::DEBUG::2013-11-19 21:30:32,131::task::495::TaskManager.Task::(_debug) Task 1d52166f-58fd-4800-af92-b41a4fbdb019: ref 0 aborting False >Thread-15911257::INFO::2013-11-19 21:30:32,131::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911258::INFO::2013-11-19 21:30:32,194::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911258::DEBUG::2013-11-19 21:30:32,195::task::495::TaskManager.Task::(_debug) Task 429e3c1a-2344-4c2c-97ef-4659ac6d5abe: moving from state init -> state preparing >Thread-15911258::DEBUG::2013-11-19 21:30:32,202::task::495::TaskManager.Task::(_debug) Task 429e3c1a-2344-4c2c-97ef-4659ac6d5abe: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000518083572388', 'lastCheck': 1384896629.640168, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154185295105', 'lastCheck': 1384896628.641227, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896624.0231359, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00044584274292', 'lastCheck': 1384896624.5577459, 'valid': True, 'code': 0}} >Thread-15911258::DEBUG::2013-11-19 21:30:32,206::task::495::TaskManager.Task::(_debug) Task 429e3c1a-2344-4c2c-97ef-4659ac6d5abe: moving from state preparing -> state finished >Thread-15911258::DEBUG::2013-11-19 21:30:32,209::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911258::DEBUG::2013-11-19 21:30:32,210::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911258::DEBUG::2013-11-19 21:30:32,211::task::495::TaskManager.Task::(_debug) Task 429e3c1a-2344-4c2c-97ef-4659ac6d5abe: ref 0 aborting False >Thread-15911258::INFO::2013-11-19 21:30:32,214::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000518083572388', 'lastCheck': 1384896629.640168, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154185295105', 'lastCheck': 1384896628.641227, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896624.0231359, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00044584274292', 'lastCheck': 1384896624.5577459, 'valid': True, 'code': 0}} >Dummy-15911115::DEBUG::2013-11-19 21:30:32,513::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:33,056::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0730939 s, 14.0 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:30:34,189::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:30:35,071::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:35,604::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0737681 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:30:37,620::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:38,130::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.070309 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:30:40,146::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:40,714::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0955083 s, 10.7 MB/s\n'; <rc> = 0 >Thread-15911264::DEBUG::2013-11-19 21:30:42,258::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911264::INFO::2013-11-19 21:30:42,259::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911264::DEBUG::2013-11-19 21:30:42,260::task::495::TaskManager.Task::(_debug) Task 9c090c21-1137-4cb7-bd40-56846cd1e5d4: moving from state init -> state preparing >Thread-15911264::DEBUG::2013-11-19 21:30:42,261::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911264::DEBUG::2013-11-19 21:30:42,267::task::495::TaskManager.Task::(_debug) Task 9c090c21-1137-4cb7-bd40-56846cd1e5d4: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911264::DEBUG::2013-11-19 21:30:42,273::task::495::TaskManager.Task::(_debug) Task 9c090c21-1137-4cb7-bd40-56846cd1e5d4: moving from state preparing -> state finished >Thread-15911264::DEBUG::2013-11-19 21:30:42,274::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911264::DEBUG::2013-11-19 21:30:42,281::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911264::DEBUG::2013-11-19 21:30:42,288::task::495::TaskManager.Task::(_debug) Task 9c090c21-1137-4cb7-bd40-56846cd1e5d4: ref 0 aborting False >Thread-15911264::INFO::2013-11-19 21:30:42,288::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911265::DEBUG::2013-11-19 21:30:42,294::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911265::INFO::2013-11-19 21:30:42,295::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911265::DEBUG::2013-11-19 21:30:42,296::task::495::TaskManager.Task::(_debug) Task 3f39b349-ef89-4425-b719-b89c1a5e6ec4: moving from state init -> state preparing >Thread-15911265::DEBUG::2013-11-19 21:30:42,297::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`709580a8-34c0-404e-8cde-23f779b76bc4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911265::DEBUG::2013-11-19 21:30:42,298::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911265::DEBUG::2013-11-19 21:30:42,299::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911265::DEBUG::2013-11-19 21:30:42,300::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`709580a8-34c0-404e-8cde-23f779b76bc4`::Granted request >Thread-15911265::DEBUG::2013-11-19 21:30:42,301::task::495::TaskManager.Task::(_debug) Task 3f39b349-ef89-4425-b719-b89c1a5e6ec4: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911265::DEBUG::2013-11-19 21:30:42,301::task::495::TaskManager.Task::(_debug) Task 3f39b349-ef89-4425-b719-b89c1a5e6ec4: ref 1 aborting False >Thread-15911265::WARNING::2013-11-19 21:30:42,304::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911265::DEBUG::2013-11-19 21:30:42,305::task::495::TaskManager.Task::(_debug) Task 3f39b349-ef89-4425-b719-b89c1a5e6ec4: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911265::DEBUG::2013-11-19 21:30:42,306::task::495::TaskManager.Task::(_debug) Task 3f39b349-ef89-4425-b719-b89c1a5e6ec4: moving from state preparing -> state finished >Thread-15911265::DEBUG::2013-11-19 21:30:42,306::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911265::DEBUG::2013-11-19 21:30:42,307::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911265::DEBUG::2013-11-19 21:30:42,307::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911265::DEBUG::2013-11-19 21:30:42,308::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911265::DEBUG::2013-11-19 21:30:42,309::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911265::DEBUG::2013-11-19 21:30:42,309::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911265::DEBUG::2013-11-19 21:30:42,314::task::495::TaskManager.Task::(_debug) Task 3f39b349-ef89-4425-b719-b89c1a5e6ec4: ref 0 aborting False >Thread-15911265::INFO::2013-11-19 21:30:42,317::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911266::INFO::2013-11-19 21:30:42,402::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911266::DEBUG::2013-11-19 21:30:42,403::task::495::TaskManager.Task::(_debug) Task 3b99929b-b481-4d73-b10c-e939ff9ea6ef: moving from state init -> state preparing >Thread-15911266::DEBUG::2013-11-19 21:30:42,406::task::495::TaskManager.Task::(_debug) Task 3b99929b-b481-4d73-b10c-e939ff9ea6ef: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000479221343994', 'lastCheck': 1384896639.827662, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146698951721', 'lastCheck': 1384896638.8318191, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896634.1899741, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00066089630127', 'lastCheck': 1384896634.737155, 'valid': True, 'code': 0}} >Thread-15911266::DEBUG::2013-11-19 21:30:42,407::task::495::TaskManager.Task::(_debug) Task 3b99929b-b481-4d73-b10c-e939ff9ea6ef: moving from state preparing -> state finished >Thread-15911266::DEBUG::2013-11-19 21:30:42,412::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911266::DEBUG::2013-11-19 21:30:42,417::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911266::DEBUG::2013-11-19 21:30:42,422::task::495::TaskManager.Task::(_debug) Task 3b99929b-b481-4d73-b10c-e939ff9ea6ef: ref 0 aborting False >Thread-15911266::INFO::2013-11-19 21:30:42,423::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000479221343994', 'lastCheck': 1384896639.827662, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146698951721', 'lastCheck': 1384896638.8318191, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896634.1899741, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00066089630127', 'lastCheck': 1384896634.737155, 'valid': True, 'code': 0}} >Dummy-15911115::DEBUG::2013-11-19 21:30:42,730::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:43,260::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0788588 s, 13.0 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:30:44,381::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:30:45,276::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:45,814::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0710354 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:30:47,830::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:48,354::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0713813 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:30:50,370::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:50,888::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0711102 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15911272::DEBUG::2013-11-19 21:30:52,469::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911272::INFO::2013-11-19 21:30:52,471::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911272::DEBUG::2013-11-19 21:30:52,479::task::495::TaskManager.Task::(_debug) Task e1bad405-3e36-44b9-b727-b6e0b1d07e50: moving from state init -> state preparing >Thread-15911272::DEBUG::2013-11-19 21:30:52,485::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911272::DEBUG::2013-11-19 21:30:52,488::task::495::TaskManager.Task::(_debug) Task e1bad405-3e36-44b9-b727-b6e0b1d07e50: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911272::DEBUG::2013-11-19 21:30:52,493::task::495::TaskManager.Task::(_debug) Task e1bad405-3e36-44b9-b727-b6e0b1d07e50: moving from state preparing -> state finished >Thread-15911272::DEBUG::2013-11-19 21:30:52,503::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911272::DEBUG::2013-11-19 21:30:52,506::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911272::DEBUG::2013-11-19 21:30:52,512::task::495::TaskManager.Task::(_debug) Task e1bad405-3e36-44b9-b727-b6e0b1d07e50: ref 0 aborting False >Thread-15911272::INFO::2013-11-19 21:30:52,514::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911273::DEBUG::2013-11-19 21:30:52,520::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911273::INFO::2013-11-19 21:30:52,521::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911273::DEBUG::2013-11-19 21:30:52,521::task::495::TaskManager.Task::(_debug) Task 5957c26e-571b-4e71-a3bc-824e081e6228: moving from state init -> state preparing >Thread-15911273::DEBUG::2013-11-19 21:30:52,528::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`cdbf29da-a91f-448b-a13a-1b392f2ff3ec`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911273::DEBUG::2013-11-19 21:30:52,534::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911273::DEBUG::2013-11-19 21:30:52,536::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911273::DEBUG::2013-11-19 21:30:52,537::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`cdbf29da-a91f-448b-a13a-1b392f2ff3ec`::Granted request >Thread-15911273::DEBUG::2013-11-19 21:30:52,538::task::495::TaskManager.Task::(_debug) Task 5957c26e-571b-4e71-a3bc-824e081e6228: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911273::DEBUG::2013-11-19 21:30:52,544::task::495::TaskManager.Task::(_debug) Task 5957c26e-571b-4e71-a3bc-824e081e6228: ref 1 aborting False >Thread-15911273::WARNING::2013-11-19 21:30:52,546::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911273::DEBUG::2013-11-19 21:30:52,547::task::495::TaskManager.Task::(_debug) Task 5957c26e-571b-4e71-a3bc-824e081e6228: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911273::DEBUG::2013-11-19 21:30:52,548::task::495::TaskManager.Task::(_debug) Task 5957c26e-571b-4e71-a3bc-824e081e6228: moving from state preparing -> state finished >Thread-15911273::DEBUG::2013-11-19 21:30:52,549::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911273::DEBUG::2013-11-19 21:30:52,549::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911273::DEBUG::2013-11-19 21:30:52,552::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911273::DEBUG::2013-11-19 21:30:52,557::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911273::DEBUG::2013-11-19 21:30:52,558::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911273::DEBUG::2013-11-19 21:30:52,561::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911273::DEBUG::2013-11-19 21:30:52,561::task::495::TaskManager.Task::(_debug) Task 5957c26e-571b-4e71-a3bc-824e081e6228: ref 0 aborting False >Thread-15911273::INFO::2013-11-19 21:30:52,565::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911274::INFO::2013-11-19 21:30:52,585::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911274::DEBUG::2013-11-19 21:30:52,586::task::495::TaskManager.Task::(_debug) Task 5cd1e240-25fd-4a2f-a025-3ab9df75f5a6: moving from state init -> state preparing >Thread-15911274::DEBUG::2013-11-19 21:30:52,587::task::495::TaskManager.Task::(_debug) Task 5cd1e240-25fd-4a2f-a025-3ab9df75f5a6: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000415086746216', 'lastCheck': 1384896650.044364, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00239205360413', 'lastCheck': 1384896649.0466111, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896644.382061, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000519990921021', 'lastCheck': 1384896644.939728, 'valid': True, 'code': 0}} >Thread-15911274::DEBUG::2013-11-19 21:30:52,588::task::495::TaskManager.Task::(_debug) Task 5cd1e240-25fd-4a2f-a025-3ab9df75f5a6: moving from state preparing -> state finished >Thread-15911274::DEBUG::2013-11-19 21:30:52,592::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911274::DEBUG::2013-11-19 21:30:52,592::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911274::DEBUG::2013-11-19 21:30:52,593::task::495::TaskManager.Task::(_debug) Task 5cd1e240-25fd-4a2f-a025-3ab9df75f5a6: ref 0 aborting False >Thread-15911274::INFO::2013-11-19 21:30:52,598::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000415086746216', 'lastCheck': 1384896650.044364, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00239205360413', 'lastCheck': 1384896649.0466111, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896644.382061, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000519990921021', 'lastCheck': 1384896644.939728, 'valid': True, 'code': 0}} >Dummy-15911115::DEBUG::2013-11-19 21:30:52,903::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:53,447::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0720865 s, 14.2 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:30:54,579::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:30:55,463::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:55,949::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0454066 s, 22.6 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:30:57,960::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:30:58,497::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0740901 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:31:00,512::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:31:01,044::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0730413 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15911280::DEBUG::2013-11-19 21:31:02,722::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911280::INFO::2013-11-19 21:31:02,728::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911280::DEBUG::2013-11-19 21:31:02,729::task::495::TaskManager.Task::(_debug) Task 5ad184f2-d95b-44a0-a8d9-a29ad354b7eb: moving from state init -> state preparing >Thread-15911280::DEBUG::2013-11-19 21:31:02,730::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911280::DEBUG::2013-11-19 21:31:02,730::task::495::TaskManager.Task::(_debug) Task 5ad184f2-d95b-44a0-a8d9-a29ad354b7eb: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911280::DEBUG::2013-11-19 21:31:02,740::task::495::TaskManager.Task::(_debug) Task 5ad184f2-d95b-44a0-a8d9-a29ad354b7eb: moving from state preparing -> state finished >Thread-15911280::DEBUG::2013-11-19 21:31:02,744::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911280::DEBUG::2013-11-19 21:31:02,754::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911280::DEBUG::2013-11-19 21:31:02,758::task::495::TaskManager.Task::(_debug) Task 5ad184f2-d95b-44a0-a8d9-a29ad354b7eb: ref 0 aborting False >Thread-15911280::INFO::2013-11-19 21:31:02,759::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911281::DEBUG::2013-11-19 21:31:02,767::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911281::INFO::2013-11-19 21:31:02,771::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911281::DEBUG::2013-11-19 21:31:02,771::task::495::TaskManager.Task::(_debug) Task 29e9ee31-458f-4c77-8198-ba5271afa4c9: moving from state init -> state preparing >Thread-15911281::DEBUG::2013-11-19 21:31:02,779::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a536898a-de29-41f5-9b8a-edd0ceac1484`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911281::DEBUG::2013-11-19 21:31:02,781::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911281::DEBUG::2013-11-19 21:31:02,785::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911281::DEBUG::2013-11-19 21:31:02,790::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a536898a-de29-41f5-9b8a-edd0ceac1484`::Granted request >Thread-15911282::INFO::2013-11-19 21:31:02,810::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911281::DEBUG::2013-11-19 21:31:02,811::task::495::TaskManager.Task::(_debug) Task 29e9ee31-458f-4c77-8198-ba5271afa4c9: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911282::DEBUG::2013-11-19 21:31:02,811::task::495::TaskManager.Task::(_debug) Task d4653dad-07ca-4f23-b689-4fadf02a6941: moving from state init -> state preparing >Thread-15911281::DEBUG::2013-11-19 21:31:02,816::task::495::TaskManager.Task::(_debug) Task 29e9ee31-458f-4c77-8198-ba5271afa4c9: ref 1 aborting False >Thread-15911282::DEBUG::2013-11-19 21:31:02,817::task::495::TaskManager.Task::(_debug) Task d4653dad-07ca-4f23-b689-4fadf02a6941: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000400066375732', 'lastCheck': 1384896660.1900761, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00151205062866', 'lastCheck': 1384896659.1837609, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896654.5803289, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000839948654175', 'lastCheck': 1384896655.135042, 'valid': True, 'code': 0}} >Thread-15911282::DEBUG::2013-11-19 21:31:02,831::task::495::TaskManager.Task::(_debug) Task d4653dad-07ca-4f23-b689-4fadf02a6941: moving from state preparing -> state finished >Thread-15911281::WARNING::2013-11-19 21:31:02,832::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911282::DEBUG::2013-11-19 21:31:02,833::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911281::DEBUG::2013-11-19 21:31:02,834::task::495::TaskManager.Task::(_debug) Task 29e9ee31-458f-4c77-8198-ba5271afa4c9: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911282::DEBUG::2013-11-19 21:31:02,835::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911281::DEBUG::2013-11-19 21:31:02,836::task::495::TaskManager.Task::(_debug) Task 29e9ee31-458f-4c77-8198-ba5271afa4c9: moving from state preparing -> state finished >Thread-15911282::DEBUG::2013-11-19 21:31:02,836::task::495::TaskManager.Task::(_debug) Task d4653dad-07ca-4f23-b689-4fadf02a6941: ref 0 aborting False >Thread-15911281::DEBUG::2013-11-19 21:31:02,841::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911282::INFO::2013-11-19 21:31:02,848::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000400066375732', 'lastCheck': 1384896660.1900761, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00151205062866', 'lastCheck': 1384896659.1837609, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896654.5803289, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000839948654175', 'lastCheck': 1384896655.135042, 'valid': True, 'code': 0}} >Thread-15911281::DEBUG::2013-11-19 21:31:02,848::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911281::DEBUG::2013-11-19 21:31:02,863::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911281::DEBUG::2013-11-19 21:31:02,864::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911281::DEBUG::2013-11-19 21:31:02,867::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911281::DEBUG::2013-11-19 21:31:02,872::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911281::DEBUG::2013-11-19 21:31:02,877::task::495::TaskManager.Task::(_debug) Task 29e9ee31-458f-4c77-8198-ba5271afa4c9: ref 0 aborting False >Thread-15911281::INFO::2013-11-19 21:31:02,878::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:31:03,059::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:31:03,597::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0736866 s, 13.9 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:31:04,752::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:31:05,612::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:31:06,127::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0727431 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:31:08,141::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:31:08,663::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0735321 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:31:10,679::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:31:11,224::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0738136 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15911288::DEBUG::2013-11-19 21:31:13,037::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911288::INFO::2013-11-19 21:31:13,047::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911288::DEBUG::2013-11-19 21:31:13,048::task::495::TaskManager.Task::(_debug) Task e7f4c79d-7a77-4746-866c-aebdd23adb0d: moving from state init -> state preparing >Thread-15911288::DEBUG::2013-11-19 21:31:13,049::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911288::DEBUG::2013-11-19 21:31:13,050::task::495::TaskManager.Task::(_debug) Task e7f4c79d-7a77-4746-866c-aebdd23adb0d: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911288::DEBUG::2013-11-19 21:31:13,051::task::495::TaskManager.Task::(_debug) Task e7f4c79d-7a77-4746-866c-aebdd23adb0d: moving from state preparing -> state finished >Thread-15911288::DEBUG::2013-11-19 21:31:13,054::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911289::INFO::2013-11-19 21:31:13,063::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911288::DEBUG::2013-11-19 21:31:13,064::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911289::DEBUG::2013-11-19 21:31:13,064::task::495::TaskManager.Task::(_debug) Task 3d4b33a5-4356-48c5-8c8f-24324f7b4077: moving from state init -> state preparing >Thread-15911288::DEBUG::2013-11-19 21:31:13,069::task::495::TaskManager.Task::(_debug) Task e7f4c79d-7a77-4746-866c-aebdd23adb0d: ref 0 aborting False >Thread-15911289::DEBUG::2013-11-19 21:31:13,070::task::495::TaskManager.Task::(_debug) Task 3d4b33a5-4356-48c5-8c8f-24324f7b4077: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000416994094849', 'lastCheck': 1384896670.3685949, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00151395797729', 'lastCheck': 1384896669.3689001, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896664.7532351, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000532150268555', 'lastCheck': 1384896665.2833481, 'valid': True, 'code': 0}} >Thread-15911288::INFO::2013-11-19 21:31:13,074::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911289::DEBUG::2013-11-19 21:31:13,085::task::495::TaskManager.Task::(_debug) Task 3d4b33a5-4356-48c5-8c8f-24324f7b4077: moving from state preparing -> state finished >Thread-15911289::DEBUG::2013-11-19 21:31:13,097::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911289::DEBUG::2013-11-19 21:31:13,098::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911289::DEBUG::2013-11-19 21:31:13,098::task::495::TaskManager.Task::(_debug) Task 3d4b33a5-4356-48c5-8c8f-24324f7b4077: ref 0 aborting False >Thread-15911289::INFO::2013-11-19 21:31:13,099::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000416994094849', 'lastCheck': 1384896670.3685949, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00151395797729', 'lastCheck': 1384896669.3689001, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896664.7532351, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000532150268555', 'lastCheck': 1384896665.2833481, 'valid': True, 'code': 0}} >Thread-15911290::DEBUG::2013-11-19 21:31:13,105::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911290::INFO::2013-11-19 21:31:13,123::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911290::DEBUG::2013-11-19 21:31:13,124::task::495::TaskManager.Task::(_debug) Task 5d037206-4dd1-442b-ad79-76735c1158c4: moving from state init -> state preparing >Thread-15911290::DEBUG::2013-11-19 21:31:13,125::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`71ec479d-deae-4c5c-8f40-e3674335f62d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911290::DEBUG::2013-11-19 21:31:13,131::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911290::DEBUG::2013-11-19 21:31:13,136::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911290::DEBUG::2013-11-19 21:31:13,139::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`71ec479d-deae-4c5c-8f40-e3674335f62d`::Granted request >Thread-15911290::DEBUG::2013-11-19 21:31:13,142::task::495::TaskManager.Task::(_debug) Task 5d037206-4dd1-442b-ad79-76735c1158c4: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911290::DEBUG::2013-11-19 21:31:13,146::task::495::TaskManager.Task::(_debug) Task 5d037206-4dd1-442b-ad79-76735c1158c4: ref 1 aborting False >Thread-15911290::WARNING::2013-11-19 21:31:13,152::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911290::DEBUG::2013-11-19 21:31:13,157::task::495::TaskManager.Task::(_debug) Task 5d037206-4dd1-442b-ad79-76735c1158c4: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911290::DEBUG::2013-11-19 21:31:13,161::task::495::TaskManager.Task::(_debug) Task 5d037206-4dd1-442b-ad79-76735c1158c4: moving from state preparing -> state finished >Thread-15911290::DEBUG::2013-11-19 21:31:13,167::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911290::DEBUG::2013-11-19 21:31:13,168::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911290::DEBUG::2013-11-19 21:31:13,169::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911290::DEBUG::2013-11-19 21:31:13,174::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911290::DEBUG::2013-11-19 21:31:13,175::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911290::DEBUG::2013-11-19 21:31:13,175::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911290::DEBUG::2013-11-19 21:31:13,176::task::495::TaskManager.Task::(_debug) Task 5d037206-4dd1-442b-ad79-76735c1158c4: ref 0 aborting False >Thread-15911290::INFO::2013-11-19 21:31:13,177::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:31:13,239::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:31:13,773::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0748828 s, 13.7 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:31:14,927::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:31:15,788::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:31:16,348::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0904018 s, 11.3 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:31:18,363::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:31:18,897::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919825 s, 11.1 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:31:20,912::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:31:21,449::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0739486 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15911296::DEBUG::2013-11-19 21:31:23,289::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911296::INFO::2013-11-19 21:31:23,296::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911296::DEBUG::2013-11-19 21:31:23,299::task::495::TaskManager.Task::(_debug) Task 0a426eb1-364c-42e3-bc28-aa91f626746c: moving from state init -> state preparing >Thread-15911296::DEBUG::2013-11-19 21:31:23,300::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911296::DEBUG::2013-11-19 21:31:23,305::task::495::TaskManager.Task::(_debug) Task 0a426eb1-364c-42e3-bc28-aa91f626746c: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911297::INFO::2013-11-19 21:31:23,308::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911296::DEBUG::2013-11-19 21:31:23,310::task::495::TaskManager.Task::(_debug) Task 0a426eb1-364c-42e3-bc28-aa91f626746c: moving from state preparing -> state finished >Thread-15911297::DEBUG::2013-11-19 21:31:23,311::task::495::TaskManager.Task::(_debug) Task 90d7c265-97be-4a00-ac88-f919976144d9: moving from state init -> state preparing >Thread-15911296::DEBUG::2013-11-19 21:31:23,319::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911297::DEBUG::2013-11-19 21:31:23,320::task::495::TaskManager.Task::(_debug) Task 90d7c265-97be-4a00-ac88-f919976144d9: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000410079956055', 'lastCheck': 1384896680.567976, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147795677185', 'lastCheck': 1384896679.5693519, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896674.9286611, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000479936599731', 'lastCheck': 1384896675.451674, 'valid': True, 'code': 0}} >Thread-15911296::DEBUG::2013-11-19 21:31:23,329::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911297::DEBUG::2013-11-19 21:31:23,330::task::495::TaskManager.Task::(_debug) Task 90d7c265-97be-4a00-ac88-f919976144d9: moving from state preparing -> state finished >Thread-15911296::DEBUG::2013-11-19 21:31:23,331::task::495::TaskManager.Task::(_debug) Task 0a426eb1-364c-42e3-bc28-aa91f626746c: ref 0 aborting False >Thread-15911297::DEBUG::2013-11-19 21:31:23,331::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911296::INFO::2013-11-19 21:31:23,337::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911297::DEBUG::2013-11-19 21:31:23,338::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911297::DEBUG::2013-11-19 21:31:23,342::task::495::TaskManager.Task::(_debug) Task 90d7c265-97be-4a00-ac88-f919976144d9: ref 0 aborting False >Thread-15911297::INFO::2013-11-19 21:31:23,345::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000410079956055', 'lastCheck': 1384896680.567976, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147795677185', 'lastCheck': 1384896679.5693519, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896674.9286611, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000479936599731', 'lastCheck': 1384896675.451674, 'valid': True, 'code': 0}} >Thread-15911298::DEBUG::2013-11-19 21:31:23,368::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911298::INFO::2013-11-19 21:31:23,378::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911298::DEBUG::2013-11-19 21:31:23,397::task::495::TaskManager.Task::(_debug) Task 19488453-461a-477c-bb71-521d21aca2e0: moving from state init -> state preparing >Thread-15911298::DEBUG::2013-11-19 21:31:23,413::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`147a4ef5-1421-4e97-9180-0e4f12fb3e3a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911298::DEBUG::2013-11-19 21:31:23,414::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911298::DEBUG::2013-11-19 21:31:23,418::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911298::DEBUG::2013-11-19 21:31:23,421::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`147a4ef5-1421-4e97-9180-0e4f12fb3e3a`::Granted request >Thread-15911298::DEBUG::2013-11-19 21:31:23,427::task::495::TaskManager.Task::(_debug) Task 19488453-461a-477c-bb71-521d21aca2e0: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911298::DEBUG::2013-11-19 21:31:23,432::task::495::TaskManager.Task::(_debug) Task 19488453-461a-477c-bb71-521d21aca2e0: ref 1 aborting False >Thread-15911298::WARNING::2013-11-19 21:31:23,438::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911298::DEBUG::2013-11-19 21:31:23,439::task::495::TaskManager.Task::(_debug) Task 19488453-461a-477c-bb71-521d21aca2e0: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911298::DEBUG::2013-11-19 21:31:23,441::task::495::TaskManager.Task::(_debug) Task 19488453-461a-477c-bb71-521d21aca2e0: moving from state preparing -> state finished >Thread-15911298::DEBUG::2013-11-19 21:31:23,444::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911298::DEBUG::2013-11-19 21:31:23,445::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911298::DEBUG::2013-11-19 21:31:23,446::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911298::DEBUG::2013-11-19 21:31:23,452::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911298::DEBUG::2013-11-19 21:31:23,452::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911298::DEBUG::2013-11-19 21:31:23,458::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911298::DEBUG::2013-11-19 21:31:23,460::task::495::TaskManager.Task::(_debug) Task 19488453-461a-477c-bb71-521d21aca2e0: ref 0 aborting False >Dummy-15911115::DEBUG::2013-11-19 21:31:23,464::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15911298::INFO::2013-11-19 21:31:23,466::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:31:23,994::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0741411 s, 13.8 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:31:25,137::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:31:26,009::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:31:26,541::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0733732 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:31:28,557::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:31:29,101::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0751608 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:31:31,116::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:31:31,640::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0732934 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15911304::INFO::2013-11-19 21:31:33,520::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911304::DEBUG::2013-11-19 21:31:33,521::task::495::TaskManager.Task::(_debug) Task 8d918c69-fc09-42fb-b03c-1183dc22ede1: moving from state init -> state preparing >Thread-15911304::DEBUG::2013-11-19 21:31:33,532::task::495::TaskManager.Task::(_debug) Task 8d918c69-fc09-42fb-b03c-1183dc22ede1: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00038480758667', 'lastCheck': 1384896690.7822969, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00155901908875', 'lastCheck': 1384896689.7859249, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896685.138437, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000769853591919', 'lastCheck': 1384896685.6800931, 'valid': True, 'code': 0}} >Thread-15911304::DEBUG::2013-11-19 21:31:33,538::task::495::TaskManager.Task::(_debug) Task 8d918c69-fc09-42fb-b03c-1183dc22ede1: moving from state preparing -> state finished >Thread-15911304::DEBUG::2013-11-19 21:31:33,538::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911304::DEBUG::2013-11-19 21:31:33,541::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911304::DEBUG::2013-11-19 21:31:33,547::task::495::TaskManager.Task::(_debug) Task 8d918c69-fc09-42fb-b03c-1183dc22ede1: ref 0 aborting False >Thread-15911304::INFO::2013-11-19 21:31:33,549::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00038480758667', 'lastCheck': 1384896690.7822969, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00155901908875', 'lastCheck': 1384896689.7859249, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896685.138437, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000769853591919', 'lastCheck': 1384896685.6800931, 'valid': True, 'code': 0}} >Dummy-15911115::DEBUG::2013-11-19 21:31:33,655::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15911306::DEBUG::2013-11-19 21:31:33,910::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911306::INFO::2013-11-19 21:31:33,921::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911306::DEBUG::2013-11-19 21:31:33,931::task::495::TaskManager.Task::(_debug) Task 203780d8-f6a8-4384-bdaf-48ca4e57aac4: moving from state init -> state preparing >Thread-15911306::DEBUG::2013-11-19 21:31:33,937::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911306::DEBUG::2013-11-19 21:31:33,948::task::495::TaskManager.Task::(_debug) Task 203780d8-f6a8-4384-bdaf-48ca4e57aac4: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911306::DEBUG::2013-11-19 21:31:33,950::task::495::TaskManager.Task::(_debug) Task 203780d8-f6a8-4384-bdaf-48ca4e57aac4: moving from state preparing -> state finished >Thread-15911306::DEBUG::2013-11-19 21:31:33,951::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911306::DEBUG::2013-11-19 21:31:33,956::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911306::DEBUG::2013-11-19 21:31:33,962::task::495::TaskManager.Task::(_debug) Task 203780d8-f6a8-4384-bdaf-48ca4e57aac4: ref 0 aborting False >Thread-15911306::INFO::2013-11-19 21:31:33,963::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911307::DEBUG::2013-11-19 21:31:33,973::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911307::INFO::2013-11-19 21:31:33,977::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911307::DEBUG::2013-11-19 21:31:33,986::task::495::TaskManager.Task::(_debug) Task bd1c5134-4903-4214-8a5c-f618cecc43f8: moving from state init -> state preparing >Thread-15911307::DEBUG::2013-11-19 21:31:33,988::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ca2f3a68-98bb-423b-8f9c-5dbe7c16f333`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911307::DEBUG::2013-11-19 21:31:33,993::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911307::DEBUG::2013-11-19 21:31:33,999::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911307::DEBUG::2013-11-19 21:31:34,005::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ca2f3a68-98bb-423b-8f9c-5dbe7c16f333`::Granted request >Thread-15911307::DEBUG::2013-11-19 21:31:34,007::task::495::TaskManager.Task::(_debug) Task bd1c5134-4903-4214-8a5c-f618cecc43f8: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911307::DEBUG::2013-11-19 21:31:34,008::task::495::TaskManager.Task::(_debug) Task bd1c5134-4903-4214-8a5c-f618cecc43f8: ref 1 aborting False >Thread-15911307::WARNING::2013-11-19 21:31:34,015::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911307::DEBUG::2013-11-19 21:31:34,016::task::495::TaskManager.Task::(_debug) Task bd1c5134-4903-4214-8a5c-f618cecc43f8: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911307::DEBUG::2013-11-19 21:31:34,017::task::495::TaskManager.Task::(_debug) Task bd1c5134-4903-4214-8a5c-f618cecc43f8: moving from state preparing -> state finished >Thread-15911307::DEBUG::2013-11-19 21:31:34,020::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911307::DEBUG::2013-11-19 21:31:34,026::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911307::DEBUG::2013-11-19 21:31:34,030::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911307::DEBUG::2013-11-19 21:31:34,033::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911307::DEBUG::2013-11-19 21:31:34,037::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911307::DEBUG::2013-11-19 21:31:34,041::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911307::DEBUG::2013-11-19 21:31:34,045::task::495::TaskManager.Task::(_debug) Task bd1c5134-4903-4214-8a5c-f618cecc43f8: ref 0 aborting False >Thread-15911307::INFO::2013-11-19 21:31:34,050::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:31:34,200::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0744408 s, 13.8 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:31:35,341::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:31:36,215::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:31:36,744::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0736149 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:31:38,760::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:31:39,290::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0725406 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:31:41,306::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:31:41,853::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0898653 s, 11.4 MB/s\n'; <rc> = 0 >Thread-15911312::INFO::2013-11-19 21:31:43,752::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911312::DEBUG::2013-11-19 21:31:43,753::task::495::TaskManager.Task::(_debug) Task aea4d7b0-7f0e-4151-8e83-94581bd5e5e6: moving from state init -> state preparing >Thread-15911312::DEBUG::2013-11-19 21:31:43,760::task::495::TaskManager.Task::(_debug) Task aea4d7b0-7f0e-4151-8e83-94581bd5e5e6: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000498056411743', 'lastCheck': 1384896700.9948461, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154805183411', 'lastCheck': 1384896699.995724, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896695.34273, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000533103942871', 'lastCheck': 1384896695.8908579, 'valid': True, 'code': 0}} >Thread-15911312::DEBUG::2013-11-19 21:31:43,761::task::495::TaskManager.Task::(_debug) Task aea4d7b0-7f0e-4151-8e83-94581bd5e5e6: moving from state preparing -> state finished >Thread-15911312::DEBUG::2013-11-19 21:31:43,764::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911312::DEBUG::2013-11-19 21:31:43,767::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911312::DEBUG::2013-11-19 21:31:43,777::task::495::TaskManager.Task::(_debug) Task aea4d7b0-7f0e-4151-8e83-94581bd5e5e6: ref 0 aborting False >Thread-15911312::INFO::2013-11-19 21:31:43,781::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000498056411743', 'lastCheck': 1384896700.9948461, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154805183411', 'lastCheck': 1384896699.995724, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896695.34273, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000533103942871', 'lastCheck': 1384896695.8908579, 'valid': True, 'code': 0}} >Dummy-15911115::DEBUG::2013-11-19 21:31:43,868::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15911314::DEBUG::2013-11-19 21:31:44,168::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911314::INFO::2013-11-19 21:31:44,175::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911314::DEBUG::2013-11-19 21:31:44,186::task::495::TaskManager.Task::(_debug) Task f2d666ba-ae83-4717-bd07-cd0e3e8a0e82: moving from state init -> state preparing >Thread-15911314::DEBUG::2013-11-19 21:31:44,193::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911314::DEBUG::2013-11-19 21:31:44,203::task::495::TaskManager.Task::(_debug) Task f2d666ba-ae83-4717-bd07-cd0e3e8a0e82: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911314::DEBUG::2013-11-19 21:31:44,209::task::495::TaskManager.Task::(_debug) Task f2d666ba-ae83-4717-bd07-cd0e3e8a0e82: moving from state preparing -> state finished >Thread-15911314::DEBUG::2013-11-19 21:31:44,213::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911314::DEBUG::2013-11-19 21:31:44,218::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911314::DEBUG::2013-11-19 21:31:44,218::task::495::TaskManager.Task::(_debug) Task f2d666ba-ae83-4717-bd07-cd0e3e8a0e82: ref 0 aborting False >Thread-15911314::INFO::2013-11-19 21:31:44,220::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911315::DEBUG::2013-11-19 21:31:44,230::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911315::INFO::2013-11-19 21:31:44,236::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911315::DEBUG::2013-11-19 21:31:44,241::task::495::TaskManager.Task::(_debug) Task bcf480b9-87ac-40c3-9d5a-b6c229e12a55: moving from state init -> state preparing >Thread-15911315::DEBUG::2013-11-19 21:31:44,243::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a551880b-8bd6-4c24-92e0-e1cab09ef4f9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911315::DEBUG::2013-11-19 21:31:44,248::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911315::DEBUG::2013-11-19 21:31:44,255::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911315::DEBUG::2013-11-19 21:31:44,259::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a551880b-8bd6-4c24-92e0-e1cab09ef4f9`::Granted request >Thread-15911315::DEBUG::2013-11-19 21:31:44,260::task::495::TaskManager.Task::(_debug) Task bcf480b9-87ac-40c3-9d5a-b6c229e12a55: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911315::DEBUG::2013-11-19 21:31:44,263::task::495::TaskManager.Task::(_debug) Task bcf480b9-87ac-40c3-9d5a-b6c229e12a55: ref 1 aborting False >Thread-15911315::WARNING::2013-11-19 21:31:44,265::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911315::DEBUG::2013-11-19 21:31:44,266::task::495::TaskManager.Task::(_debug) Task bcf480b9-87ac-40c3-9d5a-b6c229e12a55: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911315::DEBUG::2013-11-19 21:31:44,267::task::495::TaskManager.Task::(_debug) Task bcf480b9-87ac-40c3-9d5a-b6c229e12a55: moving from state preparing -> state finished >Thread-15911315::DEBUG::2013-11-19 21:31:44,271::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911315::DEBUG::2013-11-19 21:31:44,275::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911315::DEBUG::2013-11-19 21:31:44,280::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911315::DEBUG::2013-11-19 21:31:44,282::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911315::DEBUG::2013-11-19 21:31:44,285::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911315::DEBUG::2013-11-19 21:31:44,285::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911315::DEBUG::2013-11-19 21:31:44,287::task::495::TaskManager.Task::(_debug) Task bcf480b9-87ac-40c3-9d5a-b6c229e12a55: ref 0 aborting False >Thread-15911315::INFO::2013-11-19 21:31:44,288::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:31:44,361::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0562281 s, 18.2 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:31:45,526::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:31:46,371::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:31:46,892::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0747638 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:31:48,908::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:31:49,470::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0976026 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:31:51,485::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:31:52,023::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0937152 s, 10.9 MB/s\n'; <rc> = 0 >Thread-15911320::INFO::2013-11-19 21:31:53,952::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911320::DEBUG::2013-11-19 21:31:53,957::task::495::TaskManager.Task::(_debug) Task 0462a362-af71-4510-851f-a20c00d33896: moving from state init -> state preparing >Thread-15911320::DEBUG::2013-11-19 21:31:53,962::task::495::TaskManager.Task::(_debug) Task 0462a362-af71-4510-851f-a20c00d33896: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000231027603149', 'lastCheck': 1384896711.144629, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00188899040222', 'lastCheck': 1384896710.150712, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896705.527746, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000523090362549', 'lastCheck': 1384896706.089349, 'valid': True, 'code': 0}} >Thread-15911320::DEBUG::2013-11-19 21:31:53,972::task::495::TaskManager.Task::(_debug) Task 0462a362-af71-4510-851f-a20c00d33896: moving from state preparing -> state finished >Thread-15911320::DEBUG::2013-11-19 21:31:53,976::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911320::DEBUG::2013-11-19 21:31:53,979::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911320::DEBUG::2013-11-19 21:31:53,979::task::495::TaskManager.Task::(_debug) Task 0462a362-af71-4510-851f-a20c00d33896: ref 0 aborting False >Thread-15911320::INFO::2013-11-19 21:31:53,980::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000231027603149', 'lastCheck': 1384896711.144629, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00188899040222', 'lastCheck': 1384896710.150712, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896705.527746, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000523090362549', 'lastCheck': 1384896706.089349, 'valid': True, 'code': 0}} >Dummy-15911115::DEBUG::2013-11-19 21:31:54,038::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15911322::DEBUG::2013-11-19 21:31:54,400::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911322::INFO::2013-11-19 21:31:54,406::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911322::DEBUG::2013-11-19 21:31:54,407::task::495::TaskManager.Task::(_debug) Task 8e4cab5b-3068-482f-b635-d0632d6974b1: moving from state init -> state preparing >Thread-15911322::DEBUG::2013-11-19 21:31:54,411::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911322::DEBUG::2013-11-19 21:31:54,415::task::495::TaskManager.Task::(_debug) Task 8e4cab5b-3068-482f-b635-d0632d6974b1: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911322::DEBUG::2013-11-19 21:31:54,422::task::495::TaskManager.Task::(_debug) Task 8e4cab5b-3068-482f-b635-d0632d6974b1: moving from state preparing -> state finished >Thread-15911322::DEBUG::2013-11-19 21:31:54,428::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911322::DEBUG::2013-11-19 21:31:54,434::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911322::DEBUG::2013-11-19 21:31:54,439::task::495::TaskManager.Task::(_debug) Task 8e4cab5b-3068-482f-b635-d0632d6974b1: ref 0 aborting False >Thread-15911322::INFO::2013-11-19 21:31:54,444::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911323::DEBUG::2013-11-19 21:31:54,456::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911323::INFO::2013-11-19 21:31:54,458::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911323::DEBUG::2013-11-19 21:31:54,462::task::495::TaskManager.Task::(_debug) Task 43513123-8c22-4d68-a96d-8442ede0103f: moving from state init -> state preparing >Thread-15911323::DEBUG::2013-11-19 21:31:54,468::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e312e8e4-c5d5-4227-ac17-65882aa64857`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911323::DEBUG::2013-11-19 21:31:54,475::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911323::DEBUG::2013-11-19 21:31:54,485::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911323::DEBUG::2013-11-19 21:31:54,491::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e312e8e4-c5d5-4227-ac17-65882aa64857`::Granted request >Thread-15911323::DEBUG::2013-11-19 21:31:54,496::task::495::TaskManager.Task::(_debug) Task 43513123-8c22-4d68-a96d-8442ede0103f: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911323::DEBUG::2013-11-19 21:31:54,500::task::495::TaskManager.Task::(_debug) Task 43513123-8c22-4d68-a96d-8442ede0103f: ref 1 aborting False >Thread-15911323::WARNING::2013-11-19 21:31:54,505::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911323::DEBUG::2013-11-19 21:31:54,510::task::495::TaskManager.Task::(_debug) Task 43513123-8c22-4d68-a96d-8442ede0103f: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911323::DEBUG::2013-11-19 21:31:54,513::task::495::TaskManager.Task::(_debug) Task 43513123-8c22-4d68-a96d-8442ede0103f: moving from state preparing -> state finished >Thread-15911323::DEBUG::2013-11-19 21:31:54,521::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911323::DEBUG::2013-11-19 21:31:54,527::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911323::DEBUG::2013-11-19 21:31:54,528::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911323::DEBUG::2013-11-19 21:31:54,533::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911323::DEBUG::2013-11-19 21:31:54,534::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911323::DEBUG::2013-11-19 21:31:54,535::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911323::DEBUG::2013-11-19 21:31:54,539::task::495::TaskManager.Task::(_debug) Task 43513123-8c22-4d68-a96d-8442ede0103f: ref 0 aborting False >Thread-15911323::INFO::2013-11-19 21:31:54,544::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:31:54,598::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092226 s, 11.1 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:31:55,691::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:31:56,615::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:31:57,127::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0709177 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:31:59,141::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:31:59,685::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0709683 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:32:01,701::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:32:02,265::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.109372 s, 9.4 MB/s\n'; <rc> = 0 >Thread-15911328::INFO::2013-11-19 21:32:04,152::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911328::DEBUG::2013-11-19 21:32:04,153::task::495::TaskManager.Task::(_debug) Task 6b21e019-3a01-4da4-819b-f04b66e61113: moving from state init -> state preparing >Thread-15911328::DEBUG::2013-11-19 21:32:04,159::task::495::TaskManager.Task::(_debug) Task 6b21e019-3a01-4da4-819b-f04b66e61113: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000411033630371', 'lastCheck': 1384896721.3771479, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00153112411499', 'lastCheck': 1384896720.377507, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896715.69279, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000561952590942', 'lastCheck': 1384896716.2595651, 'valid': True, 'code': 0}} >Thread-15911328::DEBUG::2013-11-19 21:32:04,161::task::495::TaskManager.Task::(_debug) Task 6b21e019-3a01-4da4-819b-f04b66e61113: moving from state preparing -> state finished >Thread-15911328::DEBUG::2013-11-19 21:32:04,161::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911328::DEBUG::2013-11-19 21:32:04,167::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911328::DEBUG::2013-11-19 21:32:04,167::task::495::TaskManager.Task::(_debug) Task 6b21e019-3a01-4da4-819b-f04b66e61113: ref 0 aborting False >Thread-15911328::INFO::2013-11-19 21:32:04,173::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000411033630371', 'lastCheck': 1384896721.3771479, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00153112411499', 'lastCheck': 1384896720.377507, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896715.69279, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000561952590942', 'lastCheck': 1384896716.2595651, 'valid': True, 'code': 0}} >Dummy-15911115::DEBUG::2013-11-19 21:32:04,280::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15911330::DEBUG::2013-11-19 21:32:04,657::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911330::INFO::2013-11-19 21:32:04,661::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911330::DEBUG::2013-11-19 21:32:04,665::task::495::TaskManager.Task::(_debug) Task 068bae4c-73f8-4637-a18c-b8826878a90d: moving from state init -> state preparing >Thread-15911330::DEBUG::2013-11-19 21:32:04,672::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911330::DEBUG::2013-11-19 21:32:04,673::task::495::TaskManager.Task::(_debug) Task 068bae4c-73f8-4637-a18c-b8826878a90d: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911330::DEBUG::2013-11-19 21:32:04,674::task::495::TaskManager.Task::(_debug) Task 068bae4c-73f8-4637-a18c-b8826878a90d: moving from state preparing -> state finished >Thread-15911330::DEBUG::2013-11-19 21:32:04,679::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911330::DEBUG::2013-11-19 21:32:04,680::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911330::DEBUG::2013-11-19 21:32:04,680::task::495::TaskManager.Task::(_debug) Task 068bae4c-73f8-4637-a18c-b8826878a90d: ref 0 aborting False >Thread-15911330::INFO::2013-11-19 21:32:04,681::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911331::DEBUG::2013-11-19 21:32:04,689::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911331::INFO::2013-11-19 21:32:04,691::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911331::DEBUG::2013-11-19 21:32:04,697::task::495::TaskManager.Task::(_debug) Task 621a2743-8d68-40a1-9f95-b17c476f11a9: moving from state init -> state preparing >Thread-15911331::DEBUG::2013-11-19 21:32:04,704::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b6cfa35a-874d-4e6b-a6fe-5cc25e58bb9b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911331::DEBUG::2013-11-19 21:32:04,705::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911331::DEBUG::2013-11-19 21:32:04,706::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911331::DEBUG::2013-11-19 21:32:04,711::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b6cfa35a-874d-4e6b-a6fe-5cc25e58bb9b`::Granted request >Thread-15911331::DEBUG::2013-11-19 21:32:04,718::task::495::TaskManager.Task::(_debug) Task 621a2743-8d68-40a1-9f95-b17c476f11a9: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911331::DEBUG::2013-11-19 21:32:04,724::task::495::TaskManager.Task::(_debug) Task 621a2743-8d68-40a1-9f95-b17c476f11a9: ref 1 aborting False >Thread-15911331::WARNING::2013-11-19 21:32:04,731::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911331::DEBUG::2013-11-19 21:32:04,737::task::495::TaskManager.Task::(_debug) Task 621a2743-8d68-40a1-9f95-b17c476f11a9: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911331::DEBUG::2013-11-19 21:32:04,738::task::495::TaskManager.Task::(_debug) Task 621a2743-8d68-40a1-9f95-b17c476f11a9: moving from state preparing -> state finished >Thread-15911331::DEBUG::2013-11-19 21:32:04,743::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911331::DEBUG::2013-11-19 21:32:04,747::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911331::DEBUG::2013-11-19 21:32:04,752::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911331::DEBUG::2013-11-19 21:32:04,756::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911331::DEBUG::2013-11-19 21:32:04,763::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911331::DEBUG::2013-11-19 21:32:04,763::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911331::DEBUG::2013-11-19 21:32:04,767::task::495::TaskManager.Task::(_debug) Task 621a2743-8d68-40a1-9f95-b17c476f11a9: ref 0 aborting False >Thread-15911331::INFO::2013-11-19 21:32:04,773::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:32:04,867::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0948396 s, 10.8 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:32:05,918::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:32:06,887::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:32:07,420::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0730991 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:32:09,436::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:32:09,987::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.076453 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:32:12,003::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:32:12,520::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0738053 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15911336::INFO::2013-11-19 21:32:14,299::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911336::DEBUG::2013-11-19 21:32:14,307::task::495::TaskManager.Task::(_debug) Task d59b038e-5e2c-4ad1-8214-b781deeb0c2d: moving from state init -> state preparing >Thread-15911336::DEBUG::2013-11-19 21:32:14,310::task::495::TaskManager.Task::(_debug) Task d59b038e-5e2c-4ad1-8214-b781deeb0c2d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000458955764771', 'lastCheck': 1384896731.6659329, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00158905982971', 'lastCheck': 1384896730.6680319, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896725.9189889, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00053596496582', 'lastCheck': 1384896726.51808, 'valid': True, 'code': 0}} >Thread-15911336::DEBUG::2013-11-19 21:32:14,311::task::495::TaskManager.Task::(_debug) Task d59b038e-5e2c-4ad1-8214-b781deeb0c2d: moving from state preparing -> state finished >Thread-15911336::DEBUG::2013-11-19 21:32:14,318::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911336::DEBUG::2013-11-19 21:32:14,320::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911336::DEBUG::2013-11-19 21:32:14,328::task::495::TaskManager.Task::(_debug) Task d59b038e-5e2c-4ad1-8214-b781deeb0c2d: ref 0 aborting False >Thread-15911336::INFO::2013-11-19 21:32:14,332::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000458955764771', 'lastCheck': 1384896731.6659329, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00158905982971', 'lastCheck': 1384896730.6680319, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896725.9189889, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00053596496582', 'lastCheck': 1384896726.51808, 'valid': True, 'code': 0}} >Dummy-15911115::DEBUG::2013-11-19 21:32:14,535::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15911338::DEBUG::2013-11-19 21:32:14,902::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911338::INFO::2013-11-19 21:32:14,904::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911338::DEBUG::2013-11-19 21:32:14,908::task::495::TaskManager.Task::(_debug) Task 26b709da-742a-4ba9-b6a0-1470bc4baa98: moving from state init -> state preparing >Thread-15911338::DEBUG::2013-11-19 21:32:14,909::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911338::DEBUG::2013-11-19 21:32:14,910::task::495::TaskManager.Task::(_debug) Task 26b709da-742a-4ba9-b6a0-1470bc4baa98: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911338::DEBUG::2013-11-19 21:32:14,911::task::495::TaskManager.Task::(_debug) Task 26b709da-742a-4ba9-b6a0-1470bc4baa98: moving from state preparing -> state finished >Thread-15911338::DEBUG::2013-11-19 21:32:14,916::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911338::DEBUG::2013-11-19 21:32:14,916::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911338::DEBUG::2013-11-19 21:32:14,921::task::495::TaskManager.Task::(_debug) Task 26b709da-742a-4ba9-b6a0-1470bc4baa98: ref 0 aborting False >Thread-15911338::INFO::2013-11-19 21:32:14,926::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911339::DEBUG::2013-11-19 21:32:14,934::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911339::INFO::2013-11-19 21:32:14,937::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911339::DEBUG::2013-11-19 21:32:14,943::task::495::TaskManager.Task::(_debug) Task 5a499037-7a78-4695-b67d-a333b122d336: moving from state init -> state preparing >Thread-15911339::DEBUG::2013-11-19 21:32:14,947::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6c0dd485-0f9d-401f-a6a4-a4238345bad2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911339::DEBUG::2013-11-19 21:32:14,955::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911339::DEBUG::2013-11-19 21:32:14,962::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911339::DEBUG::2013-11-19 21:32:14,968::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6c0dd485-0f9d-401f-a6a4-a4238345bad2`::Granted request >Thread-15911339::DEBUG::2013-11-19 21:32:14,970::task::495::TaskManager.Task::(_debug) Task 5a499037-7a78-4695-b67d-a333b122d336: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911339::DEBUG::2013-11-19 21:32:14,981::task::495::TaskManager.Task::(_debug) Task 5a499037-7a78-4695-b67d-a333b122d336: ref 1 aborting False >Thread-15911339::WARNING::2013-11-19 21:32:14,986::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911339::DEBUG::2013-11-19 21:32:14,991::task::495::TaskManager.Task::(_debug) Task 5a499037-7a78-4695-b67d-a333b122d336: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911339::DEBUG::2013-11-19 21:32:14,999::task::495::TaskManager.Task::(_debug) Task 5a499037-7a78-4695-b67d-a333b122d336: moving from state preparing -> state finished >Thread-15911339::DEBUG::2013-11-19 21:32:15,001::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911339::DEBUG::2013-11-19 21:32:15,002::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911339::DEBUG::2013-11-19 21:32:15,007::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911339::DEBUG::2013-11-19 21:32:15,012::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911339::DEBUG::2013-11-19 21:32:15,018::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911339::DEBUG::2013-11-19 21:32:15,022::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911339::DEBUG::2013-11-19 21:32:15,026::task::495::TaskManager.Task::(_debug) Task 5a499037-7a78-4695-b67d-a333b122d336: ref 0 aborting False >Thread-15911339::INFO::2013-11-19 21:32:15,031::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:32:15,092::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0893432 s, 11.5 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:32:16,214::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 21:32:16,757::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:32:17,113::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-42840::DEBUG::2013-11-19 21:32:17,466::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:32:17,659::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0736515 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:32:19,675::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:32:20,236::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100983 s, 10.1 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:32:22,252::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:32:22,770::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0703223 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15911344::INFO::2013-11-19 21:32:24,500::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911344::DEBUG::2013-11-19 21:32:24,500::task::495::TaskManager.Task::(_debug) Task 5f20bfb2-20e6-4ff3-9182-3e4f0964fc70: moving from state init -> state preparing >Thread-15911344::DEBUG::2013-11-19 21:32:24,501::task::495::TaskManager.Task::(_debug) Task 5f20bfb2-20e6-4ff3-9182-3e4f0964fc70: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00041389465332', 'lastCheck': 1384896741.8957131, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0015389919281', 'lastCheck': 1384896740.896699, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896736.215724, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000415086746216', 'lastCheck': 1384896737.46929, 'valid': True, 'code': 0}} >Thread-15911344::DEBUG::2013-11-19 21:32:24,511::task::495::TaskManager.Task::(_debug) Task 5f20bfb2-20e6-4ff3-9182-3e4f0964fc70: moving from state preparing -> state finished >Thread-15911344::DEBUG::2013-11-19 21:32:24,513::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911344::DEBUG::2013-11-19 21:32:24,520::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911344::DEBUG::2013-11-19 21:32:24,522::task::495::TaskManager.Task::(_debug) Task 5f20bfb2-20e6-4ff3-9182-3e4f0964fc70: ref 0 aborting False >Thread-15911344::INFO::2013-11-19 21:32:24,525::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00041389465332', 'lastCheck': 1384896741.8957131, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0015389919281', 'lastCheck': 1384896740.896699, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896736.215724, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000415086746216', 'lastCheck': 1384896737.46929, 'valid': True, 'code': 0}} >Dummy-15911115::DEBUG::2013-11-19 21:32:24,786::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15911346::DEBUG::2013-11-19 21:32:25,168::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911346::INFO::2013-11-19 21:32:25,172::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911346::DEBUG::2013-11-19 21:32:25,173::task::495::TaskManager.Task::(_debug) Task e41c018f-9c08-4cb8-9f50-7835dc3edacc: moving from state init -> state preparing >Thread-15911346::DEBUG::2013-11-19 21:32:25,179::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911346::DEBUG::2013-11-19 21:32:25,184::task::495::TaskManager.Task::(_debug) Task e41c018f-9c08-4cb8-9f50-7835dc3edacc: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911346::DEBUG::2013-11-19 21:32:25,189::task::495::TaskManager.Task::(_debug) Task e41c018f-9c08-4cb8-9f50-7835dc3edacc: moving from state preparing -> state finished >Thread-15911346::DEBUG::2013-11-19 21:32:25,191::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911346::DEBUG::2013-11-19 21:32:25,195::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911346::DEBUG::2013-11-19 21:32:25,197::task::495::TaskManager.Task::(_debug) Task e41c018f-9c08-4cb8-9f50-7835dc3edacc: ref 0 aborting False >Thread-15911346::INFO::2013-11-19 21:32:25,200::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911347::DEBUG::2013-11-19 21:32:25,209::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911347::INFO::2013-11-19 21:32:25,216::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911347::DEBUG::2013-11-19 21:32:25,216::task::495::TaskManager.Task::(_debug) Task 4055a242-d0f7-4c8b-8032-c34c411e1274: moving from state init -> state preparing >Thread-15911347::DEBUG::2013-11-19 21:32:25,223::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`72d39545-1747-4894-abe1-2aadba9be500`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911347::DEBUG::2013-11-19 21:32:25,225::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911347::DEBUG::2013-11-19 21:32:25,232::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911347::DEBUG::2013-11-19 21:32:25,235::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`72d39545-1747-4894-abe1-2aadba9be500`::Granted request >Thread-15911347::DEBUG::2013-11-19 21:32:25,240::task::495::TaskManager.Task::(_debug) Task 4055a242-d0f7-4c8b-8032-c34c411e1274: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911347::DEBUG::2013-11-19 21:32:25,241::task::495::TaskManager.Task::(_debug) Task 4055a242-d0f7-4c8b-8032-c34c411e1274: ref 1 aborting False >Thread-15911347::WARNING::2013-11-19 21:32:25,249::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911347::DEBUG::2013-11-19 21:32:25,256::task::495::TaskManager.Task::(_debug) Task 4055a242-d0f7-4c8b-8032-c34c411e1274: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911347::DEBUG::2013-11-19 21:32:25,259::task::495::TaskManager.Task::(_debug) Task 4055a242-d0f7-4c8b-8032-c34c411e1274: moving from state preparing -> state finished >Thread-15911347::DEBUG::2013-11-19 21:32:25,264::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911347::DEBUG::2013-11-19 21:32:25,271::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911347::DEBUG::2013-11-19 21:32:25,274::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911347::DEBUG::2013-11-19 21:32:25,277::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911347::DEBUG::2013-11-19 21:32:25,282::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911347::DEBUG::2013-11-19 21:32:25,286::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911347::DEBUG::2013-11-19 21:32:25,287::task::495::TaskManager.Task::(_debug) Task 4055a242-d0f7-4c8b-8032-c34c411e1274: ref 0 aborting False >Thread-15911347::INFO::2013-11-19 21:32:25,297::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911115::DEBUG::2013-11-19 21:32:25,349::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0938593 s, 10.9 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:32:26,458::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:32:27,364::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:32:27,882::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0727621 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:32:29,897::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:32:30,435::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0733308 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:32:32,450::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:32:32,977::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0731734 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15911352::INFO::2013-11-19 21:32:34,702::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911352::DEBUG::2013-11-19 21:32:34,707::task::495::TaskManager.Task::(_debug) Task ab65de2c-f0b9-44f2-89e8-9cf87fe014d3: moving from state init -> state preparing >Thread-15911352::DEBUG::2013-11-19 21:32:34,713::task::495::TaskManager.Task::(_debug) Task ab65de2c-f0b9-44f2-89e8-9cf87fe014d3: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000406980514526', 'lastCheck': 1384896752.1252999, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0022599697113', 'lastCheck': 1384896751.129323, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896746.459507, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00099515914917', 'lastCheck': 1384896747.573915, 'valid': True, 'code': 0}} >Thread-15911352::DEBUG::2013-11-19 21:32:34,714::task::495::TaskManager.Task::(_debug) Task ab65de2c-f0b9-44f2-89e8-9cf87fe014d3: moving from state preparing -> state finished >Thread-15911352::DEBUG::2013-11-19 21:32:34,718::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911352::DEBUG::2013-11-19 21:32:34,720::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911352::DEBUG::2013-11-19 21:32:34,724::task::495::TaskManager.Task::(_debug) Task ab65de2c-f0b9-44f2-89e8-9cf87fe014d3: ref 0 aborting False >Thread-15911352::INFO::2013-11-19 21:32:34,725::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000406980514526', 'lastCheck': 1384896752.1252999, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0022599697113', 'lastCheck': 1384896751.129323, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896746.459507, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00099515914917', 'lastCheck': 1384896747.573915, 'valid': True, 'code': 0}} >Dummy-15911115::DEBUG::2013-11-19 21:32:34,992::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15911354::DEBUG::2013-11-19 21:32:35,415::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911354::INFO::2013-11-19 21:32:35,420::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911354::DEBUG::2013-11-19 21:32:35,421::task::495::TaskManager.Task::(_debug) Task eba7c324-dca9-4b0a-9c9f-36d040f680d5: moving from state init -> state preparing >Thread-15911354::DEBUG::2013-11-19 21:32:35,429::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911354::DEBUG::2013-11-19 21:32:35,435::task::495::TaskManager.Task::(_debug) Task eba7c324-dca9-4b0a-9c9f-36d040f680d5: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911354::DEBUG::2013-11-19 21:32:35,440::task::495::TaskManager.Task::(_debug) Task eba7c324-dca9-4b0a-9c9f-36d040f680d5: moving from state preparing -> state finished >Thread-15911354::DEBUG::2013-11-19 21:32:35,450::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911354::DEBUG::2013-11-19 21:32:35,458::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911354::DEBUG::2013-11-19 21:32:35,463::task::495::TaskManager.Task::(_debug) Task eba7c324-dca9-4b0a-9c9f-36d040f680d5: ref 0 aborting False >Thread-15911354::INFO::2013-11-19 21:32:35,463::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911355::DEBUG::2013-11-19 21:32:35,474::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911355::INFO::2013-11-19 21:32:35,484::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911355::DEBUG::2013-11-19 21:32:35,486::task::495::TaskManager.Task::(_debug) Task f89d0027-f096-40ff-b4d2-6b9baa3c0fd9: moving from state init -> state preparing >Thread-15911355::DEBUG::2013-11-19 21:32:35,489::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d80abfec-8735-43dc-9775-6ea84bcf0b46`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911355::DEBUG::2013-11-19 21:32:35,490::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911355::DEBUG::2013-11-19 21:32:35,496::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911355::DEBUG::2013-11-19 21:32:35,500::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d80abfec-8735-43dc-9775-6ea84bcf0b46`::Granted request >Thread-15911355::DEBUG::2013-11-19 21:32:35,503::task::495::TaskManager.Task::(_debug) Task f89d0027-f096-40ff-b4d2-6b9baa3c0fd9: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911355::DEBUG::2013-11-19 21:32:35,506::task::495::TaskManager.Task::(_debug) Task f89d0027-f096-40ff-b4d2-6b9baa3c0fd9: ref 1 aborting False >Thread-15911355::WARNING::2013-11-19 21:32:35,513::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911355::DEBUG::2013-11-19 21:32:35,519::task::495::TaskManager.Task::(_debug) Task f89d0027-f096-40ff-b4d2-6b9baa3c0fd9: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911355::DEBUG::2013-11-19 21:32:35,525::task::495::TaskManager.Task::(_debug) Task f89d0027-f096-40ff-b4d2-6b9baa3c0fd9: moving from state preparing -> state finished >Thread-15911355::DEBUG::2013-11-19 21:32:35,532::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911355::DEBUG::2013-11-19 21:32:35,533::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911355::DEBUG::2013-11-19 21:32:35,535::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911355::DEBUG::2013-11-19 21:32:35,539::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911355::DEBUG::2013-11-19 21:32:35,544::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Dummy-15911115::DEBUG::2013-11-19 21:32:35,545::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.087863 s, 11.7 MB/s\n'; <rc> = 0 >Thread-15911355::DEBUG::2013-11-19 21:32:35,545::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911355::DEBUG::2013-11-19 21:32:35,561::task::495::TaskManager.Task::(_debug) Task f89d0027-f096-40ff-b4d2-6b9baa3c0fd9: ref 0 aborting False >Thread-15911355::INFO::2013-11-19 21:32:35,567::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-1024767::ERROR::2013-11-19 21:32:36,667::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:32:37,563::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:32:38,101::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0751734 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:32:40,117::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:32:40,663::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0940199 s, 10.9 MB/s\n'; <rc> = 0 >Dummy-15911115::DEBUG::2013-11-19 21:32:42,678::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:32:43,225::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0746352 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15911360::INFO::2013-11-19 21:32:44,920::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911360::DEBUG::2013-11-19 21:32:44,926::task::495::TaskManager.Task::(_debug) Task ba1e2c87-f876-4d8d-b6f6-a0842c5b9cd2: moving from state init -> state preparing >Thread-15911360::DEBUG::2013-11-19 21:32:44,929::task::495::TaskManager.Task::(_debug) Task ba1e2c87-f876-4d8d-b6f6-a0842c5b9cd2: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000515222549438', 'lastCheck': 1384896762.3370249, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00155186653137', 'lastCheck': 1384896761.345922, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896756.668355, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00338411331177', 'lastCheck': 1384896757.789011, 'valid': True, 'code': 0}} >Thread-15911360::DEBUG::2013-11-19 21:32:44,929::task::495::TaskManager.Task::(_debug) Task ba1e2c87-f876-4d8d-b6f6-a0842c5b9cd2: moving from state preparing -> state finished >Thread-15911360::DEBUG::2013-11-19 21:32:44,930::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911360::DEBUG::2013-11-19 21:32:44,931::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911360::DEBUG::2013-11-19 21:32:44,934::task::495::TaskManager.Task::(_debug) Task ba1e2c87-f876-4d8d-b6f6-a0842c5b9cd2: ref 0 aborting False >Thread-15911360::INFO::2013-11-19 21:32:44,943::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000515222549438', 'lastCheck': 1384896762.3370249, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00155186653137', 'lastCheck': 1384896761.345922, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896756.668355, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00338411331177', 'lastCheck': 1384896757.789011, 'valid': True, 'code': 0}} >Dummy-15911115::DEBUG::2013-11-19 21:32:45,240::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15911362::DEBUG::2013-11-19 21:32:45,690::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911362::INFO::2013-11-19 21:32:45,698::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911362::DEBUG::2013-11-19 21:32:45,699::task::495::TaskManager.Task::(_debug) Task 16eef668-0538-409a-ad13-ab28123776e8: moving from state init -> state preparing >Thread-15911362::DEBUG::2013-11-19 21:32:45,706::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911362::DEBUG::2013-11-19 21:32:45,709::task::495::TaskManager.Task::(_debug) Task 16eef668-0538-409a-ad13-ab28123776e8: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911362::DEBUG::2013-11-19 21:32:45,715::task::495::TaskManager.Task::(_debug) Task 16eef668-0538-409a-ad13-ab28123776e8: moving from state preparing -> state finished >Thread-15911362::DEBUG::2013-11-19 21:32:45,726::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911362::DEBUG::2013-11-19 21:32:45,732::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911362::DEBUG::2013-11-19 21:32:45,733::task::495::TaskManager.Task::(_debug) Task 16eef668-0538-409a-ad13-ab28123776e8: ref 0 aborting False >Thread-15911362::INFO::2013-11-19 21:32:45,737::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911363::DEBUG::2013-11-19 21:32:45,755::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911363::INFO::2013-11-19 21:32:45,760::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911363::DEBUG::2013-11-19 21:32:45,761::task::495::TaskManager.Task::(_debug) Task 591dd447-ba13-4da3-83fc-f5d24aae10e5: moving from state init -> state preparing >Thread-15911363::DEBUG::2013-11-19 21:32:45,764::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6e2f4d37-9d34-4c61-a005-e9f77ef402eb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911363::DEBUG::2013-11-19 21:32:45,765::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911363::DEBUG::2013-11-19 21:32:45,771::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911363::DEBUG::2013-11-19 21:32:45,772::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6e2f4d37-9d34-4c61-a005-e9f77ef402eb`::Granted request >Thread-15911363::DEBUG::2013-11-19 21:32:45,774::task::495::TaskManager.Task::(_debug) Task 591dd447-ba13-4da3-83fc-f5d24aae10e5: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911363::DEBUG::2013-11-19 21:32:45,780::task::495::TaskManager.Task::(_debug) Task 591dd447-ba13-4da3-83fc-f5d24aae10e5: ref 1 aborting False >Dummy-15911115::DEBUG::2013-11-19 21:32:45,786::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0804732 s, 12.7 MB/s\n'; <rc> = 0 >Thread-15911363::WARNING::2013-11-19 21:32:45,789::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911363::DEBUG::2013-11-19 21:32:45,804::task::495::TaskManager.Task::(_debug) Task 591dd447-ba13-4da3-83fc-f5d24aae10e5: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911363::DEBUG::2013-11-19 21:32:45,810::task::495::TaskManager.Task::(_debug) Task 591dd447-ba13-4da3-83fc-f5d24aae10e5: moving from state preparing -> state finished >Thread-15911363::DEBUG::2013-11-19 21:32:45,813::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911363::DEBUG::2013-11-19 21:32:45,816::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911363::DEBUG::2013-11-19 21:32:45,816::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911363::DEBUG::2013-11-19 21:32:45,820::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911363::DEBUG::2013-11-19 21:32:45,824::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911363::DEBUG::2013-11-19 21:32:45,829::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911363::DEBUG::2013-11-19 21:32:45,832::task::495::TaskManager.Task::(_debug) Task 591dd447-ba13-4da3-83fc-f5d24aae10e5: ref 0 aborting False >Thread-15911363::INFO::2013-11-19 21:32:45,837::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 282}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-1024767::ERROR::2013-11-19 21:32:46,912::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911115::DEBUG::2013-11-19 21:32:47,806::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15911365::DEBUG::2013-11-19 21:32:48,240::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911365::INFO::2013-11-19 21:32:48,247::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksStatuses, args: () >Thread-15911365::DEBUG::2013-11-19 21:32:48,250::task::495::TaskManager.Task::(_debug) Task eaf94ff5-229a-4e8e-a5b0-f3f0eeba6293: moving from state init -> state preparing >Thread-15911365::DEBUG::2013-11-19 21:32:48,255::taskManager::92::TaskManager::(getAllTasksStatuses) Entry. >Thread-15911365::DEBUG::2013-11-19 21:32:48,258::taskManager::97::TaskManager::(getAllTasksStatuses) Return: {} >Thread-15911365::DEBUG::2013-11-19 21:32:48,261::task::495::TaskManager.Task::(_debug) Task eaf94ff5-229a-4e8e-a5b0-f3f0eeba6293: finished: {'allTasksStatus': {}} >Thread-15911365::DEBUG::2013-11-19 21:32:48,265::task::495::TaskManager.Task::(_debug) Task eaf94ff5-229a-4e8e-a5b0-f3f0eeba6293: moving from state preparing -> state finished >Thread-15911365::DEBUG::2013-11-19 21:32:48,266::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911365::DEBUG::2013-11-19 21:32:48,269::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911365::DEBUG::2013-11-19 21:32:48,277::task::495::TaskManager.Task::(_debug) Task eaf94ff5-229a-4e8e-a5b0-f3f0eeba6293: ref 0 aborting False >Thread-15911365::INFO::2013-11-19 21:32:48,282::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksStatuses, Return response: {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}} >Thread-15911366::DEBUG::2013-11-19 21:32:48,325::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911366::INFO::2013-11-19 21:32:48,334::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: deactivateStorageDomain, args: ( sdUUID=5d7b5178-3790-472c-a13b-b481eb27a61e spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 msdUUID=00000000-0000-0000-0000-000000000000 masterVersion=1) >Thread-15911366::DEBUG::2013-11-19 21:32:48,341::task::495::TaskManager.Task::(_debug) Task 03c2cd07-d237-43f3-9bb6-2337dd99ef48: moving from state init -> state preparing >Thread-15911366::DEBUG::2013-11-19 21:32:48,344::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a6176029-37b3-4249-9978-33dabf666afc`::Request was made in '/usr/share/vdsm/storage/spm.py' line '1112' at 'public_deactivateStorageDomain' >Thread-15911366::DEBUG::2013-11-19 21:32:48,348::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >Thread-15911366::DEBUG::2013-11-19 21:32:48,354::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >Thread-15911366::DEBUG::2013-11-19 21:32:48,360::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a6176029-37b3-4249-9978-33dabf666afc`::Granted request >Thread-15911366::DEBUG::2013-11-19 21:32:48,386::task::495::TaskManager.Task::(_debug) Task 03c2cd07-d237-43f3-9bb6-2337dd99ef48: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (exclusive) >Thread-15911366::DEBUG::2013-11-19 21:32:48,391::task::495::TaskManager.Task::(_debug) Task 03c2cd07-d237-43f3-9bb6-2337dd99ef48: ref 1 aborting False >Thread-15911366::DEBUG::2013-11-19 21:32:48,396::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`1188c516-4f4b-42af-bb27-4e0902c22262`::Request was made in '/usr/share/vdsm/storage/spm.py' line '1113' at 'public_deactivateStorageDomain' >Thread-15911366::DEBUG::2013-11-19 21:32:48,401::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' for lock type 'exclusive' >Thread-15911366::DEBUG::2013-11-19 21:32:48,404::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' is free. Now locking as 'exclusive' (1 active user) >Dummy-15911115::DEBUG::2013-11-19 21:32:48,406::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0890451 s, 11.5 MB/s\n'; <rc> = 0 >Thread-15911366::DEBUG::2013-11-19 21:32:48,409::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`1188c516-4f4b-42af-bb27-4e0902c22262`::Granted request >Thread-15911366::DEBUG::2013-11-19 21:32:48,426::task::495::TaskManager.Task::(_debug) Task 03c2cd07-d237-43f3-9bb6-2337dd99ef48: _resourcesAcquired: Storage.5d7b5178-3790-472c-a13b-b481eb27a61e (exclusive) >Thread-15911366::DEBUG::2013-11-19 21:32:48,431::task::495::TaskManager.Task::(_debug) Task 03c2cd07-d237-43f3-9bb6-2337dd99ef48: ref 1 aborting False >Thread-15911366::INFO::2013-11-19 21:32:48,437::sp::775::Storage.StoragePool::(deactivateSD) sdUUID=5d7b5178-3790-472c-a13b-b481eb27a61e spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 new_msdUUID=00000000-0000-0000-0000-000000000000 >Thread-15911366::INFO::2013-11-19 21:32:48,443::fileSD::176::Storage.StorageDomain::(validate) sdUUID=5d7b5178-3790-472c-a13b-b481eb27a61e >Thread-15911366::DEBUG::2013-11-19 21:32:48,452::persistentDict::204::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=[] >Thread-15911366::WARNING::2013-11-19 21:32:48,455::persistentDict::226::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-15911366::ERROR::2013-11-19 21:32:48,458::task::868::TaskManager.Task::(_setError) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 876, in _run > File "/usr/share/vdsm/storage/spm.py", line 115, in run > File "/usr/share/vdsm/storage/spm.py", line 1115, in public_deactivateStorageDomain > File "/usr/share/vdsm/storage/sp.py", line 798, in deactivateSD > File "/usr/share/vdsm/storage/sd.py", line 666, in isBackup > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911366::DEBUG::2013-11-19 21:32:48,466::task::495::TaskManager.Task::(_debug) Task 03c2cd07-d237-43f3-9bb6-2337dd99ef48: Task._run: 03c2cd07-d237-43f3-9bb6-2337dd99ef48 ('5d7b5178-3790-472c-a13b-b481eb27a61e', 'c4e5889a-348d-11e2-83fe-9f72369eab30', '00000000-0000-0000-0000-000000000000', 1) {} failed - stopping task >Thread-15911366::DEBUG::2013-11-19 21:32:48,469::task::495::TaskManager.Task::(_debug) Task 03c2cd07-d237-43f3-9bb6-2337dd99ef48: stopping in state preparing (force False) >Thread-15911366::DEBUG::2013-11-19 21:32:48,474::task::495::TaskManager.Task::(_debug) Task 03c2cd07-d237-43f3-9bb6-2337dd99ef48: ref 1 aborting True >Thread-15911366::INFO::2013-11-19 21:32:48,477::task::1171::TaskManager.Task::(prepare) aborting: Task is aborted: "'CLASS'" - code 100 >Thread-15911366::DEBUG::2013-11-19 21:32:48,483::task::495::TaskManager.Task::(_debug) Task 03c2cd07-d237-43f3-9bb6-2337dd99ef48: Prepare: aborted: 'CLASS' >Thread-15911366::DEBUG::2013-11-19 21:32:48,484::task::495::TaskManager.Task::(_debug) Task 03c2cd07-d237-43f3-9bb6-2337dd99ef48: ref 0 aborting True >Thread-15911366::DEBUG::2013-11-19 21:32:48,486::task::495::TaskManager.Task::(_debug) Task 03c2cd07-d237-43f3-9bb6-2337dd99ef48: Task._doAbort: force False >Thread-15911366::DEBUG::2013-11-19 21:32:48,486::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911366::DEBUG::2013-11-19 21:32:48,490::task::495::TaskManager.Task::(_debug) Task 03c2cd07-d237-43f3-9bb6-2337dd99ef48: moving from state preparing -> state aborting >Thread-15911366::DEBUG::2013-11-19 21:32:48,492::task::495::TaskManager.Task::(_debug) Task 03c2cd07-d237-43f3-9bb6-2337dd99ef48: _aborting: recover policy none >Thread-15911366::DEBUG::2013-11-19 21:32:48,497::task::495::TaskManager.Task::(_debug) Task 03c2cd07-d237-43f3-9bb6-2337dd99ef48: moving from state aborting -> state failed >Thread-15911366::DEBUG::2013-11-19 21:32:48,502::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>, 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e': < ResourceRef 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e', isValid: 'True' obj: 'None'>} >Thread-15911366::DEBUG::2013-11-19 21:32:48,510::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911366::DEBUG::2013-11-19 21:32:48,516::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911366::DEBUG::2013-11-19 21:32:48,522::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911366::DEBUG::2013-11-19 21:32:48,526::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911366::DEBUG::2013-11-19 21:32:48,530::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911366::DEBUG::2013-11-19 21:32:48,533::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' >Thread-15911366::DEBUG::2013-11-19 21:32:48,533::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' (0 active users) >Thread-15911366::DEBUG::2013-11-19 21:32:48,538::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' is free, finding out if anyone is waiting for it. >Thread-15911366::DEBUG::2013-11-19 21:32:48,542::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e', Clearing records. >Thread-15911366::ERROR::2013-11-19 21:32:48,551::dispatcher::106::Storage.Dispatcher.Protect::(run) 'CLASS' >Thread-15911366::ERROR::2013-11-19 21:32:48,552::dispatcher::107::Storage.Dispatcher.Protect::(run) Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 96, in run > File "/usr/share/vdsm/storage/task.py", line 1178, in prepare >KeyError: 'CLASS' > >Thread-15911367::DEBUG::2013-11-19 21:32:48,566::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911367::INFO::2013-11-19 21:32:48,568::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911367::DEBUG::2013-11-19 21:32:48,569::task::495::TaskManager.Task::(_debug) Task 03e8fcfb-b639-43c7-8a15-a3301ae789e7: moving from state init -> state preparing >Thread-15911367::DEBUG::2013-11-19 21:32:48,570::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=282 spmId=2 >Thread-15911367::DEBUG::2013-11-19 21:32:48,573::task::495::TaskManager.Task::(_debug) Task 03e8fcfb-b639-43c7-8a15-a3301ae789e7: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911367::DEBUG::2013-11-19 21:32:48,580::task::495::TaskManager.Task::(_debug) Task 03e8fcfb-b639-43c7-8a15-a3301ae789e7: moving from state preparing -> state finished >Thread-15911367::DEBUG::2013-11-19 21:32:48,584::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911367::DEBUG::2013-11-19 21:32:48,589::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911367::DEBUG::2013-11-19 21:32:48,594::task::495::TaskManager.Task::(_debug) Task 03e8fcfb-b639-43c7-8a15-a3301ae789e7: ref 0 aborting False >Thread-15911367::INFO::2013-11-19 21:32:48,596::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 282}} >Thread-15911368::DEBUG::2013-11-19 21:32:48,611::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911368::INFO::2013-11-19 21:32:48,612::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksStatuses, args: () >Thread-15911368::DEBUG::2013-11-19 21:32:48,616::task::495::TaskManager.Task::(_debug) Task f9e3cf38-af8c-4cfa-b45d-0b592e8eb94c: moving from state init -> state preparing >Thread-15911368::DEBUG::2013-11-19 21:32:48,620::taskManager::92::TaskManager::(getAllTasksStatuses) Entry. >Thread-15911368::DEBUG::2013-11-19 21:32:48,627::taskManager::97::TaskManager::(getAllTasksStatuses) Return: {} >Thread-15911368::DEBUG::2013-11-19 21:32:48,628::task::495::TaskManager.Task::(_debug) Task f9e3cf38-af8c-4cfa-b45d-0b592e8eb94c: finished: {'allTasksStatus': {}} >Thread-15911368::DEBUG::2013-11-19 21:32:48,632::task::495::TaskManager.Task::(_debug) Task f9e3cf38-af8c-4cfa-b45d-0b592e8eb94c: moving from state preparing -> state finished >Thread-15911368::DEBUG::2013-11-19 21:32:48,637::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911368::DEBUG::2013-11-19 21:32:48,643::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911368::DEBUG::2013-11-19 21:32:48,652::task::495::TaskManager.Task::(_debug) Task f9e3cf38-af8c-4cfa-b45d-0b592e8eb94c: ref 0 aborting False >Thread-15911368::INFO::2013-11-19 21:32:48,652::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksStatuses, Return response: {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}} >Thread-15911369::DEBUG::2013-11-19 21:32:48,666::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911369::INFO::2013-11-19 21:32:48,672::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: spmStop, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911369::DEBUG::2013-11-19 21:32:48,677::task::495::TaskManager.Task::(_debug) Task fa928307-b43e-4461-a204-968268ffd7bc: moving from state init -> state preparing >Thread-15911369::DEBUG::2013-11-19 21:32:48,681::spm::547::Storage.SPM::(public_spmStop) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 >Thread-15911369::DEBUG::2013-11-19 21:32:48,686::taskManager::105::TaskManager::(getAllTasks) Entry. >Thread-15911369::DEBUG::2013-11-19 21:32:48,686::taskManager::110::TaskManager::(getAllTasks) Return: {} >Thread-15911369::DEBUG::2013-11-19 21:32:48,693::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`14cf12b2-c543-4f20-864d-94e57c02488f`::Request was made in '/usr/share/vdsm/storage/spm.py' line '573' at 'public_spmStop' >Thread-15911369::DEBUG::2013-11-19 21:32:48,702::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >Thread-15911369::DEBUG::2013-11-19 21:32:48,705::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >Thread-15911369::DEBUG::2013-11-19 21:32:48,705::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`14cf12b2-c543-4f20-864d-94e57c02488f`::Granted request >Thread-15911369::DEBUG::2013-11-19 21:32:48,709::task::495::TaskManager.Task::(_debug) Task fa928307-b43e-4461-a204-968268ffd7bc: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (exclusive) >Thread-15911369::DEBUG::2013-11-19 21:32:48,713::task::495::TaskManager.Task::(_debug) Task fa928307-b43e-4461-a204-968268ffd7bc: ref 1 aborting False >Thread-15911369::DEBUG::2013-11-19 21:32:48,717::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`354ddf24-151c-458e-9af8-0698e5e152c2`::Request was made in '/usr/share/vdsm/storage/spm.py' line '584' at '_stop' >Thread-15911369::DEBUG::2013-11-19 21:32:48,717::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >Thread-15911369::DEBUG::2013-11-19 21:32:48,722::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >Thread-15911369::DEBUG::2013-11-19 21:32:48,727::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`354ddf24-151c-458e-9af8-0698e5e152c2`::Granted request >Thread-15911369::DEBUG::2013-11-19 21:32:48,730::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`270247ba-3217-4bcb-a230-cbb83730fba8`::Request was made in '/usr/share/vdsm/storage/spm.py' line '601' at '_stop' >Thread-15911369::DEBUG::2013-11-19 21:32:48,735::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' for lock type 'exclusive' >Thread-15911369::DEBUG::2013-11-19 21:32:48,741::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' is free. Now locking as 'exclusive' (1 active user) >Thread-15911369::DEBUG::2013-11-19 21:32:48,747::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`270247ba-3217-4bcb-a230-cbb83730fba8`::Granted request >Thread-15911369::DEBUG::2013-11-19 21:32:48,755::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' >Thread-15911369::DEBUG::2013-11-19 21:32:48,758::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' (0 active users) >Thread-15911369::DEBUG::2013-11-19 21:32:48,764::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' is free, finding out if anyone is waiting for it. >Thread-15911369::DEBUG::2013-11-19 21:32:48,765::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e', Clearing records. >Thread-15911369::DEBUG::2013-11-19 21:32:48,777::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911369::DEBUG::2013-11-19 21:32:48,780::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911369::DEBUG::2013-11-19 21:32:48,784::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911369::DEBUG::2013-11-19 21:32:48,789::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911369::DEBUG::2013-11-19 21:32:48,793::taskManager::118::TaskManager::(unloadTasks) Entry. >Thread-15911369::DEBUG::2013-11-19 21:32:48,796::taskManager::122::TaskManager::(unloadTasks) Return >Thread-15911369::DEBUG::2013-11-19 21:32:48,810::spm::195::Storage.SPM::(__cleanupMasterMount) unmounting /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master >Dummy-15911115::DEBUG::2013-11-19 21:32:50,431::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15911369::DEBUG::2013-11-19 21:32:51,412::fileUtils::112::Storage.Misc.excCmd::(umount) '/usr/bin/sudo -n /bin/umount -f /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Dummy-15911115::DEBUG::2013-11-19 21:32:51,759::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.846124 s, 1.2 MB/s\n'; <rc> = 0 >Thread-15911369::DEBUG::2013-11-19 21:32:51,853::fileUtils::112::Storage.Misc.excCmd::(umount) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911369::DEBUG::2013-11-19 21:32:51,870::spm::290::Storage.SPM::(__cleanupSPM) cleaning up SPM: c4e5889a-348d-11e2-83fe-9f72369eab30 >Thread-15911369::DEBUG::2013-11-19 21:32:51,875::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >Thread-15911369::DEBUG::2013-11-19 21:32:51,876::persistentDict::146::Storage.PersistentDict::(transaction) Flushing changes >Thread-15911369::DEBUG::2013-11-19 21:32:51,885::persistentDict::265::Storage.PersistentDict::(flush) about to write lines (VGTagMetadataRW)=[u'CLASS=Data', u'DESCRIPTION=OS', u'IOOPTIMEOUTSEC=10', u'LEASERETRIES=3', u'LEASETIMESEC=60', u'LOCKPOLICY=', u'LOCKRENEWALINTERVALSEC=5', u'LOGBLKSIZE=512', u'MASTER_VERSION=1', u'PHYBLKSIZE=512', u'POOL_DESCRIPTION=Default', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_ID=-1', u'POOL_SPM_LVER=282', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'ROLE=Master', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'TYPE=FCP', u'VERSION=2', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', '_SHA_CKSUM=f2989d9b2e8ee39a10c4cf9b6999adb068743a1b'] >Thread-15911369::DEBUG::2013-11-19 21:32:51,892::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15911369::DEBUG::2013-11-19 21:32:51,893::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15911369::DEBUG::2013-11-19 21:32:51,898::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15911369::DEBUG::2013-11-19 21:32:51,903::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15911369::DEBUG::2013-11-19 21:32:51,909::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15911369::DEBUG::2013-11-19 21:32:51,915::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15911369::DEBUG::2013-11-19 21:32:52,565::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911369::DEBUG::2013-11-19 21:32:52,574::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15911369::DEBUG::2013-11-19 21:32:52,584::blockSD::145::storage.Metadata.VGTagMetadataRW::(writelines) Updating metadata adding=MDT_POOL_SPM_ID=-1, MDT__SHA_CKSUM=f2989d9b2e8ee39a10c4cf9b6999adb068743a1b removing=MDT__SHA_CKSUM=de56f40c15838d78cba198e8c045a60fe83f87c3, MDT_POOL_SPM_ID=2 >Thread-15911369::DEBUG::2013-11-19 21:32:52,585::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --deltag MDT__SHA_CKSUM=de56f40c15838d78cba198e8c045a60fe83f87c3 --deltag MDT_POOL_SPM_ID=2 --addtag MDT_POOL_SPM_ID=-1 --addtag MDT__SHA_CKSUM=f2989d9b2e8ee39a10c4cf9b6999adb068743a1b 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15911369::DEBUG::2013-11-19 21:32:53,267::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911369::DEBUG::2013-11-19 21:32:53,272::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15911369::DEBUG::2013-11-19 21:32:53,278::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15911369::DEBUG::2013-11-19 21:32:53,281::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >Thread-15911369::INFO::2013-11-19 21:32:53,282::safelease::83::ClusterLock::(release) Releasing cluster lock for domain 6b892067-ac7e-4039-a7be-5b7387525b16 >Thread-15911369::DEBUG::2013-11-19 21:32:53,287::safelease::84::Storage.Misc.excCmd::(release) '/usr/libexec/vdsm/spmstop.sh 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd /usr/libexec/vdsm/) >Dummy-15911115::INFO::2013-11-19 21:32:53,774::storage_mailbox::673::Storage.MailBox.SpmMailMonitor::(run) SPM_MailMonitor - Incoming mail monitoring thread stopped >Thread-15911369::DEBUG::2013-11-19 21:32:54,781::safelease::84::Storage.Misc.excCmd::(release) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911369::DEBUG::2013-11-19 21:32:54,794::safelease::87::ClusterLock::(release) Cluster lock released successfully >Thread-15911369::DEBUG::2013-11-19 21:32:54,795::task::495::TaskManager.Task::(_debug) Task fa928307-b43e-4461-a204-968268ffd7bc: finished: None >Thread-15911369::DEBUG::2013-11-19 21:32:54,797::task::495::TaskManager.Task::(_debug) Task fa928307-b43e-4461-a204-968268ffd7bc: moving from state preparing -> state finished >Thread-15911369::DEBUG::2013-11-19 21:32:54,798::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911369::DEBUG::2013-11-19 21:32:54,802::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911369::DEBUG::2013-11-19 21:32:54,803::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911369::DEBUG::2013-11-19 21:32:54,809::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911369::DEBUG::2013-11-19 21:32:54,814::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911369::DEBUG::2013-11-19 21:32:54,817::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911369::DEBUG::2013-11-19 21:32:54,822::task::495::TaskManager.Task::(_debug) Task fa928307-b43e-4461-a204-968268ffd7bc: ref 0 aborting False >Thread-15911369::INFO::2013-11-19 21:32:54,828::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: spmStop, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15911373::INFO::2013-11-19 21:32:55,167::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911373::DEBUG::2013-11-19 21:32:55,171::task::495::TaskManager.Task::(_debug) Task 342b82f4-0839-4a93-9ef9-88a6574ebf7f: moving from state init -> state preparing >Thread-15911373::DEBUG::2013-11-19 21:32:55,172::task::495::TaskManager.Task::(_debug) Task 342b82f4-0839-4a93-9ef9-88a6574ebf7f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000354051589966', 'lastCheck': 1384896773.285223, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00261998176575', 'lastCheck': 1384896771.677285, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896766.9137299, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00110697746277', 'lastCheck': 1384896768.029747, 'valid': True, 'code': 0}} >Thread-15911373::DEBUG::2013-11-19 21:32:55,177::task::495::TaskManager.Task::(_debug) Task 342b82f4-0839-4a93-9ef9-88a6574ebf7f: moving from state preparing -> state finished >Thread-15911373::DEBUG::2013-11-19 21:32:55,178::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911373::DEBUG::2013-11-19 21:32:55,179::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911373::DEBUG::2013-11-19 21:32:55,179::task::495::TaskManager.Task::(_debug) Task 342b82f4-0839-4a93-9ef9-88a6574ebf7f: ref 0 aborting False >Thread-15911373::INFO::2013-11-19 21:32:55,186::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000354051589966', 'lastCheck': 1384896773.285223, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00261998176575', 'lastCheck': 1384896771.677285, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896766.9137299, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00110697746277', 'lastCheck': 1384896768.029747, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:32:57,151::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 21:33:03,296::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-42838::DEBUG::2013-11-19 21:33:03,297::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 21:33:04,010::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-42838::DEBUG::2013-11-19 21:33:04,014::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15911379::INFO::2013-11-19 21:33:05,530::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911379::DEBUG::2013-11-19 21:33:05,541::task::495::TaskManager.Task::(_debug) Task 51ab0f8b-4f17-458c-9255-dce94fc44939: moving from state init -> state preparing >Thread-15911379::DEBUG::2013-11-19 21:33:05,546::task::495::TaskManager.Task::(_debug) Task 51ab0f8b-4f17-458c-9255-dce94fc44939: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000343084335327', 'lastCheck': 1384896784.017015, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00156593322754', 'lastCheck': 1384896781.816519, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896777.1525199, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000456809997559', 'lastCheck': 1384896778.1475229, 'valid': True, 'code': 0}} >Thread-15911379::DEBUG::2013-11-19 21:33:05,551::task::495::TaskManager.Task::(_debug) Task 51ab0f8b-4f17-458c-9255-dce94fc44939: moving from state preparing -> state finished >Thread-15911379::DEBUG::2013-11-19 21:33:05,553::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911379::DEBUG::2013-11-19 21:33:05,555::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911379::DEBUG::2013-11-19 21:33:05,560::task::495::TaskManager.Task::(_debug) Task 51ab0f8b-4f17-458c-9255-dce94fc44939: ref 0 aborting False >Thread-15911379::INFO::2013-11-19 21:33:05,563::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000343084335327', 'lastCheck': 1384896784.017015, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00156593322754', 'lastCheck': 1384896781.816519, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896777.1525199, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000456809997559', 'lastCheck': 1384896778.1475229, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:33:07,165::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911385::INFO::2013-11-19 21:33:15,710::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911385::DEBUG::2013-11-19 21:33:15,716::task::495::TaskManager.Task::(_debug) Task 1275dd79-bad0-435e-ac2e-344d1511461a: moving from state init -> state preparing >Thread-15911385::DEBUG::2013-11-19 21:33:15,719::task::495::TaskManager.Task::(_debug) Task 1275dd79-bad0-435e-ac2e-344d1511461a: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000350952148438', 'lastCheck': 1384896794.0297301, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152111053467', 'lastCheck': 1384896791.832377, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896787.165859, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000514030456543', 'lastCheck': 1384896788.1593089, 'valid': True, 'code': 0}} >Thread-15911385::DEBUG::2013-11-19 21:33:15,723::task::495::TaskManager.Task::(_debug) Task 1275dd79-bad0-435e-ac2e-344d1511461a: moving from state preparing -> state finished >Thread-15911385::DEBUG::2013-11-19 21:33:15,726::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911385::DEBUG::2013-11-19 21:33:15,732::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911385::DEBUG::2013-11-19 21:33:15,732::task::495::TaskManager.Task::(_debug) Task 1275dd79-bad0-435e-ac2e-344d1511461a: ref 0 aborting False >Thread-15911385::INFO::2013-11-19 21:33:15,733::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000350952148438', 'lastCheck': 1384896794.0297301, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152111053467', 'lastCheck': 1384896791.832377, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896787.165859, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000514030456543', 'lastCheck': 1384896788.1593089, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:33:17,178::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911391::INFO::2013-11-19 21:33:25,902::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911391::DEBUG::2013-11-19 21:33:25,903::task::495::TaskManager.Task::(_debug) Task b0cccb68-1763-4cc3-91de-c638d6b6e87c: moving from state init -> state preparing >Thread-15911391::DEBUG::2013-11-19 21:33:25,914::task::495::TaskManager.Task::(_debug) Task b0cccb68-1763-4cc3-91de-c638d6b6e87c: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000314950942993', 'lastCheck': 1384896804.042403, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152802467346', 'lastCheck': 1384896801.848088, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896797.1790421, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000376224517822', 'lastCheck': 1384896798.170836, 'valid': True, 'code': 0}} >Thread-15911391::DEBUG::2013-11-19 21:33:25,917::task::495::TaskManager.Task::(_debug) Task b0cccb68-1763-4cc3-91de-c638d6b6e87c: moving from state preparing -> state finished >Thread-15911391::DEBUG::2013-11-19 21:33:25,921::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911391::DEBUG::2013-11-19 21:33:25,921::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911391::DEBUG::2013-11-19 21:33:25,924::task::495::TaskManager.Task::(_debug) Task b0cccb68-1763-4cc3-91de-c638d6b6e87c: ref 0 aborting False >Thread-15911391::INFO::2013-11-19 21:33:25,925::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000314950942993', 'lastCheck': 1384896804.042403, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152802467346', 'lastCheck': 1384896801.848088, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896797.1790421, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000376224517822', 'lastCheck': 1384896798.170836, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:33:27,191::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911395::DEBUG::2013-11-19 21:33:30,845::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911395::INFO::2013-11-19 21:33:30,851::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911395::DEBUG::2013-11-19 21:33:30,852::task::495::TaskManager.Task::(_debug) Task 2d064909-e822-488a-94cf-29edada0c506: moving from state init -> state preparing >Thread-15911395::DEBUG::2013-11-19 21:33:30,856::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=Free spmLver=282 spmId=-1 >Thread-15911395::DEBUG::2013-11-19 21:33:30,857::task::495::TaskManager.Task::(_debug) Task 2d064909-e822-488a-94cf-29edada0c506: finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 282}} >Thread-15911395::DEBUG::2013-11-19 21:33:30,859::task::495::TaskManager.Task::(_debug) Task 2d064909-e822-488a-94cf-29edada0c506: moving from state preparing -> state finished >Thread-15911395::DEBUG::2013-11-19 21:33:30,859::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911395::DEBUG::2013-11-19 21:33:30,860::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911395::DEBUG::2013-11-19 21:33:30,860::task::495::TaskManager.Task::(_debug) Task 2d064909-e822-488a-94cf-29edada0c506: ref 0 aborting False >Thread-15911395::INFO::2013-11-19 21:33:30,861::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 282}} >Thread-15911396::DEBUG::2013-11-19 21:33:30,889::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911396::INFO::2013-11-19 21:33:30,897::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: spmStart, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 prevID=-1 prevLVER=282 recoveryMode=0 scsiFencing=false maxHostID=250 domVersion=2) >Thread-15911396::DEBUG::2013-11-19 21:33:30,900::task::495::TaskManager.Task::(_debug) Task d3f111cf-7b31-41dc-bcdf-c918a6697f62: moving from state init -> state preparing >Thread-15911396::DEBUG::2013-11-19 21:33:30,905::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`471856ca-84e1-48c9-a828-58db130cc847`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '330' at 'public_spmStart' >Thread-15911396::DEBUG::2013-11-19 21:33:30,905::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >Thread-15911396::DEBUG::2013-11-19 21:33:30,906::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >Thread-15911396::DEBUG::2013-11-19 21:33:30,909::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`471856ca-84e1-48c9-a828-58db130cc847`::Granted request >Thread-15911396::DEBUG::2013-11-19 21:33:30,910::task::495::TaskManager.Task::(_debug) Task d3f111cf-7b31-41dc-bcdf-c918a6697f62: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (exclusive) >Thread-15911396::DEBUG::2013-11-19 21:33:30,910::task::495::TaskManager.Task::(_debug) Task d3f111cf-7b31-41dc-bcdf-c918a6697f62: ref 1 aborting False >Thread-15911396::DEBUG::2013-11-19 21:33:30,911::task::495::TaskManager.Task::(_debug) Task d3f111cf-7b31-41dc-bcdf-c918a6697f62: Prepare: 1 jobs exist, move to acquiring >Thread-15911396::DEBUG::2013-11-19 21:33:30,914::task::495::TaskManager.Task::(_debug) Task d3f111cf-7b31-41dc-bcdf-c918a6697f62: moving from state preparing -> state acquiring >Thread-15911396::DEBUG::2013-11-19 21:33:30,914::task::495::TaskManager.Task::(_debug) Task d3f111cf-7b31-41dc-bcdf-c918a6697f62: moving from state acquiring -> state queued >Thread-15911396::DEBUG::2013-11-19 21:33:30,917::taskManager::37::TaskManager::(_queueTask) queueing task: d3f111cf-7b31-41dc-bcdf-c918a6697f62 >Thread-15911396::DEBUG::2013-11-19 21:33:30,919::taskManager::43::TaskManager::(_queueTask) task queued: d3f111cf-7b31-41dc-bcdf-c918a6697f62 >af42b7a2-b5f4-4f0d-b67c-ff9c021074aa::DEBUG::2013-11-19 21:33:30,919::threadPool::47::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-15911396::DEBUG::2013-11-19 21:33:30,923::task::1183::TaskManager.Task::(prepare) returning >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:30,923::threadPool::192::Misc.ThreadPool.WorkerThread::(run) Task: d3f111cf-7b31-41dc-bcdf-c918a6697f62 running: <bound method Task.commit of <storage.task.Task instance at 0x7f4878072170>> with: None >Thread-15911396::DEBUG::2013-11-19 21:33:30,928::task::495::TaskManager.Task::(_debug) Task d3f111cf-7b31-41dc-bcdf-c918a6697f62: ref 0 aborting False >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:30,929::task::1195::TaskManager.Task::(commit) Commiting task: d3f111cf-7b31-41dc-bcdf-c918a6697f62 >Thread-15911396::INFO::2013-11-19 21:33:30,932::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: spmStart, Return response: {'status': {'message': 'OK', 'code': 0}, 'uuid': 'd3f111cf-7b31-41dc-bcdf-c918a6697f62'} >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:30,933::task::495::TaskManager.Task::(_debug) Task d3f111cf-7b31-41dc-bcdf-c918a6697f62: moving from state queued -> state running >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:30,934::task::907::TaskManager.Task::(_runJobs) Task.run: running job 0: spmStart: <bound method SPM.start of <storage.spm.SPM instance at 0x1bf9f38>> (args: ('c4e5889a-348d-11e2-83fe-9f72369eab30', -1, '282', 0, 'false', 250, 2) kwargs: {}) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:30,935::task::300::TaskManager.Task::(run) Job.run: running spmStart: <bound method SPM.start of <storage.spm.SPM instance at 0x1bf9f38>> (args: ('c4e5889a-348d-11e2-83fe-9f72369eab30', -1, '282', 0, 'false', 250, 2) kwargs: {}) callback None >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:30,935::spm::433::Storage.SPM::(start) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:30,936::misc::1010::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:30,936::misc::1012::SamplingMethod::(__call__) Got in to sampling method >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:30,937::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:30,938::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:30,938::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:30,939::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:30,940::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:30,941::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:31,628::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:31,632::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:31,633::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=283', u'_SHA_CKSUM=a0430374bee4cdb3438125b8b1df9b26332bc740', u'POOL_SPM_ID=-1'] >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:31,634::misc::1020::SamplingMethod::(__call__) Returning last result >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:31,635::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:31,636::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:31,636::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:31,637::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:31,638::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:31,639::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15911397::DEBUG::2013-11-19 21:33:31,941::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911397::INFO::2013-11-19 21:33:31,946::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=d3f111cf-7b31-41dc-bcdf-c918a6697f62) >Thread-15911397::DEBUG::2013-11-19 21:33:31,949::task::495::TaskManager.Task::(_debug) Task 7c6ed653-5afd-417e-804a-31b28d2935df: moving from state init -> state preparing >Thread-15911397::DEBUG::2013-11-19 21:33:31,952::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: d3f111cf-7b31-41dc-bcdf-c918a6697f62 >Thread-15911397::DEBUG::2013-11-19 21:33:31,953::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'd3f111cf-7b31-41dc-bcdf-c918a6697f62'} >Thread-15911397::DEBUG::2013-11-19 21:33:31,958::task::495::TaskManager.Task::(_debug) Task 7c6ed653-5afd-417e-804a-31b28d2935df: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'd3f111cf-7b31-41dc-bcdf-c918a6697f62'}} >Thread-15911397::DEBUG::2013-11-19 21:33:31,959::task::495::TaskManager.Task::(_debug) Task 7c6ed653-5afd-417e-804a-31b28d2935df: moving from state preparing -> state finished >Thread-15911397::DEBUG::2013-11-19 21:33:31,964::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911397::DEBUG::2013-11-19 21:33:31,965::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911397::DEBUG::2013-11-19 21:33:31,966::task::495::TaskManager.Task::(_debug) Task 7c6ed653-5afd-417e-804a-31b28d2935df: ref 0 aborting False >Thread-15911397::INFO::2013-11-19 21:33:31,967::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'd3f111cf-7b31-41dc-bcdf-c918a6697f62'}} >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:32,300::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:32,303::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:32,304::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=283', u'_SHA_CKSUM=a0430374bee4cdb3438125b8b1df9b26332bc740', u'POOL_SPM_ID=-1'] >d3f111cf-7b31-41dc-bcdf-c918a6697f62::INFO::2013-11-19 21:33:32,306::spm::457::Storage.SPM::(start) expected previd:-1 lver:283 got request for previd:-1 lver:282 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:32,307::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:32,307::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:32,308::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:32,309::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:32,309::persistentDict::163::Storage.PersistentDict::(__init__) Created a persistant dict with LvMetadataRW backend >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:32,310::blockSD::164::Storage.Misc.excCmd::(readlines) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:32,714::blockSD::164::Storage.Misc.excCmd::(readlines) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000321387 s, 6.4 MB/s\n'; <rc> = 0 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:32,715::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000321387 s, 6.4 MB/s'], size: 2048 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:32,716::persistentDict::204::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >d3f111cf-7b31-41dc-bcdf-c918a6697f62::WARNING::2013-11-19 21:33:32,716::persistentDict::226::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:32,717::persistentDict::163::Storage.PersistentDict::(__init__) Created a persistant dict with VGTagMetadataRW backend >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:32,718::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:32,719::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:32,719::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:32,720::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:32,720::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:32,722::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15911399::DEBUG::2013-11-19 21:33:32,984::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911399::INFO::2013-11-19 21:33:32,989::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=d3f111cf-7b31-41dc-bcdf-c918a6697f62) >Thread-15911399::DEBUG::2013-11-19 21:33:32,990::task::495::TaskManager.Task::(_debug) Task b0097669-c869-44d3-b698-f1a9ed107b8c: moving from state init -> state preparing >Thread-15911399::DEBUG::2013-11-19 21:33:32,991::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: d3f111cf-7b31-41dc-bcdf-c918a6697f62 >Thread-15911399::DEBUG::2013-11-19 21:33:32,999::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'd3f111cf-7b31-41dc-bcdf-c918a6697f62'} >Thread-15911399::DEBUG::2013-11-19 21:33:33,002::task::495::TaskManager.Task::(_debug) Task b0097669-c869-44d3-b698-f1a9ed107b8c: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'd3f111cf-7b31-41dc-bcdf-c918a6697f62'}} >Thread-15911399::DEBUG::2013-11-19 21:33:33,004::task::495::TaskManager.Task::(_debug) Task b0097669-c869-44d3-b698-f1a9ed107b8c: moving from state preparing -> state finished >Thread-15911399::DEBUG::2013-11-19 21:33:33,004::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911399::DEBUG::2013-11-19 21:33:33,005::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911399::DEBUG::2013-11-19 21:33:33,009::task::495::TaskManager.Task::(_debug) Task b0097669-c869-44d3-b698-f1a9ed107b8c: ref 0 aborting False >Thread-15911399::INFO::2013-11-19 21:33:33,017::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'd3f111cf-7b31-41dc-bcdf-c918a6697f62'}} >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:33,383::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:33,386::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:33,387::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=283', u'_SHA_CKSUM=a0430374bee4cdb3438125b8b1df9b26332bc740', u'POOL_SPM_ID=-1'] >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:33,389::safelease::52::ClusterLock::(acquire) Acquiring cluster lock for domain 6b892067-ac7e-4039-a7be-5b7387525b16 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:33,390::safelease::61::Storage.Misc.excCmd::(acquire) '/usr/bin/sudo -n /usr/bin/setsid /usr/bin/ionice -c1 -n0 /bin/su vdsm -s /bin/sh -c "/usr/libexec/vdsm/spmprotect.sh start 6b892067-ac7e-4039-a7be-5b7387525b16 2 5 /dev/6b892067-ac7e-4039-a7be-5b7387525b16/leases 60000 10000 3"' (cwd /usr/libexec/vdsm/) >Thread-15911400::DEBUG::2013-11-19 21:33:34,029::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911400::INFO::2013-11-19 21:33:34,038::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=d3f111cf-7b31-41dc-bcdf-c918a6697f62) >Thread-15911400::DEBUG::2013-11-19 21:33:34,039::task::495::TaskManager.Task::(_debug) Task 9b767371-4fee-4b59-a382-57fd90d1f87b: moving from state init -> state preparing >Thread-15911400::DEBUG::2013-11-19 21:33:34,044::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: d3f111cf-7b31-41dc-bcdf-c918a6697f62 >Thread-15911400::DEBUG::2013-11-19 21:33:34,046::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'd3f111cf-7b31-41dc-bcdf-c918a6697f62'} >Thread-15911400::DEBUG::2013-11-19 21:33:34,053::task::495::TaskManager.Task::(_debug) Task 9b767371-4fee-4b59-a382-57fd90d1f87b: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'd3f111cf-7b31-41dc-bcdf-c918a6697f62'}} >Thread-15911400::DEBUG::2013-11-19 21:33:34,055::task::495::TaskManager.Task::(_debug) Task 9b767371-4fee-4b59-a382-57fd90d1f87b: moving from state preparing -> state finished >Thread-15911400::DEBUG::2013-11-19 21:33:34,065::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911400::DEBUG::2013-11-19 21:33:34,066::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911400::DEBUG::2013-11-19 21:33:34,070::task::495::TaskManager.Task::(_debug) Task 9b767371-4fee-4b59-a382-57fd90d1f87b: ref 0 aborting False >Thread-15911400::INFO::2013-11-19 21:33:34,076::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'd3f111cf-7b31-41dc-bcdf-c918a6697f62'}} >Thread-15911402::DEBUG::2013-11-19 21:33:35,087::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911402::INFO::2013-11-19 21:33:35,093::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=d3f111cf-7b31-41dc-bcdf-c918a6697f62) >Thread-15911402::DEBUG::2013-11-19 21:33:35,094::task::495::TaskManager.Task::(_debug) Task 19eb3224-05b1-4373-9542-443a57b6b74b: moving from state init -> state preparing >Thread-15911402::DEBUG::2013-11-19 21:33:35,097::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: d3f111cf-7b31-41dc-bcdf-c918a6697f62 >Thread-15911402::DEBUG::2013-11-19 21:33:35,097::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'd3f111cf-7b31-41dc-bcdf-c918a6697f62'} >Thread-15911402::DEBUG::2013-11-19 21:33:35,098::task::495::TaskManager.Task::(_debug) Task 19eb3224-05b1-4373-9542-443a57b6b74b: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'd3f111cf-7b31-41dc-bcdf-c918a6697f62'}} >Thread-15911402::DEBUG::2013-11-19 21:33:35,103::task::495::TaskManager.Task::(_debug) Task 19eb3224-05b1-4373-9542-443a57b6b74b: moving from state preparing -> state finished >Thread-15911402::DEBUG::2013-11-19 21:33:35,108::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911402::DEBUG::2013-11-19 21:33:35,111::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911402::DEBUG::2013-11-19 21:33:35,114::task::495::TaskManager.Task::(_debug) Task 19eb3224-05b1-4373-9542-443a57b6b74b: ref 0 aborting False >Thread-15911402::INFO::2013-11-19 21:33:35,117::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'd3f111cf-7b31-41dc-bcdf-c918a6697f62'}} >Thread-1024767::ERROR::2013-11-19 21:33:37,205::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:33:47,219::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:53,822::safelease::61::Storage.Misc.excCmd::(acquire) SUCCESS: <err> = ''; <rc> = 0 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:53,823::safelease::65::ClusterLock::(acquire) Clustered lock acquired successfully >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:53,823::spm::468::Storage.SPM::(start) spm lock acquired successfully >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:53,824::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:53,825::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:53,825::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:53,826::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:53,827::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:53,828::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 21:33:54,179::lvm::439::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-42838::DEBUG::2013-11-19 21:33:54,181::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:54,502::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:54,508::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=283', u'_SHA_CKSUM=a0430374bee4cdb3438125b8b1df9b26332bc740', u'POOL_SPM_ID=-1'] >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:54,510::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:54,511::persistentDict::146::Storage.PersistentDict::(transaction) Flushing changes >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:54,512::persistentDict::265::Storage.PersistentDict::(flush) about to write lines (VGTagMetadataRW)=[u'CLASS=Data', u'DESCRIPTION=OS', u'IOOPTIMEOUTSEC=10', u'LEASERETRIES=3', u'LEASETIMESEC=60', u'LOCKPOLICY=', u'LOCKRENEWALINTERVALSEC=5', u'LOGBLKSIZE=512', u'MASTER_VERSION=1', u'PHYBLKSIZE=512', u'POOL_DESCRIPTION=Default', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_ID=2', u'POOL_SPM_LVER=284', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'ROLE=Master', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'TYPE=FCP', u'VERSION=2', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', '_SHA_CKSUM=444b3b54067466f11e0599712ab301365a6be2aa'] >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:54,513::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting... >Thread-42838::DEBUG::2013-11-19 21:33:54,786::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-42838::DEBUG::2013-11-19 21:33:54,789::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:54,790::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:54,791::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:54,792::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:54,792::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:54,793::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:54,794::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:55,402::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:55,405::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:55,406::blockSD::145::storage.Metadata.VGTagMetadataRW::(writelines) Updating metadata adding=MDT_POOL_SPM_ID=2, MDT_POOL_SPM_LVER=284, MDT__SHA_CKSUM=444b3b54067466f11e0599712ab301365a6be2aa removing=MDT_POOL_SPM_ID=-1, MDT_POOL_SPM_LVER=283, MDT__SHA_CKSUM=a0430374bee4cdb3438125b8b1df9b26332bc740 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:55,408::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --deltag MDT_POOL_SPM_LVER=283 --deltag MDT__SHA_CKSUM=a0430374bee4cdb3438125b8b1df9b26332bc740 --deltag MDT_POOL_SPM_ID=-1 --addtag MDT_POOL_SPM_ID=2 --addtag MDT__SHA_CKSUM=444b3b54067466f11e0599712ab301365a6be2aa --addtag MDT_POOL_SPM_LVER=284 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,147::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,148::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,149::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,150::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,152::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`54b0ff01-1808-474c-b733-777c94c948e3`::Request was made in '/usr/share/vdsm/storage/spm.py' line '330' at '_upgradePool' >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,152::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,155::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,155::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`54b0ff01-1808-474c-b733-777c94c948e3`::Granted request >d3f111cf-7b31-41dc-bcdf-c918a6697f62::INFO::2013-11-19 21:33:56,156::spm::336::Storage.SPM::(_upgradePool) Trying to upgrade master domain `6b892067-ac7e-4039-a7be-5b7387525b16` >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,157::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ffd71c28-f399-4057-92d8-ecc590861d53`::Request was made in '/usr/share/vdsm/storage/spm.py' line '337' at '_upgradePool' >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,158::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'exclusive' >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,158::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'exclusive' (1 active user) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,159::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ffd71c28-f399-4057-92d8-ecc590861d53`::Granted request >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,160::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,160::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,161::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,162::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,162::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,163::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,759::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,762::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,763::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_ID=2', u'_SHA_CKSUM=444b3b54067466f11e0599712ab301365a6be2aa', u'POOL_SPM_LVER=284'] >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,764::blockSD::625::Storage.StorageDomain::(upgrade) Trying to upgrade domain `6b892067-ac7e-4039-a7be-5b7387525b16` from version 2 to version 2 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,765::blockSD::631::Storage.StorageDomain::(upgrade) No need to upgrade domain `6b892067-ac7e-4039-a7be-5b7387525b16`, leaving unchanged >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,766::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,766::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,767::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,767::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,768::spm::340::Storage.SPM::(_upgradePool) Marking all domains for upgrade >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,769::spm::347::Storage.SPM::(_upgradePool) Registering with state change event >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,769::spm::349::Storage.SPM::(_upgradePool) Running initial domain upgrade threads >Thread-15911403::DEBUG::2013-11-19 21:33:56,770::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 45cdff8b-89c0-4926-a8c7-80354064bfa6 >Thread-15911404::DEBUG::2013-11-19 21:33:56,771::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 772472cb-2429-466d-96cd-00b3301ef78a >Thread-15911405::DEBUG::2013-11-19 21:33:56,772::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 5d7b5178-3790-472c-a13b-b481eb27a61e >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,782::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911403::DEBUG::2013-11-19 21:33:56,783::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`3563c0a1-5cdc-49c0-b62d-2d1a126a690a`::Request was made in '/usr/share/vdsm/storage/spm.py' line '376' at '_upgradePoolDomain' >Thread-15911404::DEBUG::2013-11-19 21:33:56,790::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`8257dd3e-29f6-42cc-b5a0-a52a75c5c917`::Request was made in '/usr/share/vdsm/storage/spm.py' line '376' at '_upgradePoolDomain' >Thread-15911405::ERROR::2013-11-19 21:33:56,790::spm::373::Storage.SPM::(_upgradePoolDomain) Error while preparing domain `5d7b5178-3790-472c-a13b-b481eb27a61e` upgrade >Traceback (most recent call last): > File "/usr/share/vdsm/storage/spm.py", line 369, in _upgradePoolDomain >IndexError: list index out of range >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,791::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911403::DEBUG::2013-11-19 21:33:56,798::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' for lock type 'exclusive' >Thread-15911404::DEBUG::2013-11-19 21:33:56,805::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'exclusive' >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,807::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,812::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911403::DEBUG::2013-11-19 21:33:56,813::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' is free. Now locking as 'exclusive' (1 active user) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:56,814::blockSD::791::Storage.Misc.excCmd::(mountMaster) '/usr/bin/sudo -n /sbin/e2fsck -p /dev/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15911403::DEBUG::2013-11-19 21:33:56,814::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`3563c0a1-5cdc-49c0-b62d-2d1a126a690a`::Granted request >Thread-15911404::DEBUG::2013-11-19 21:33:57,034::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'exclusive' (1 active user) >Thread-15911404::DEBUG::2013-11-19 21:33:57,036::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`8257dd3e-29f6-42cc-b5a0-a52a75c5c917`::Granted request >Thread-15911403::DEBUG::2013-11-19 21:33:57,038::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`232a75f0-99b2-4d73-aff6-f7e9d1a1f5c4`::Request was made in '/usr/share/vdsm/storage/spm.py' line '377' at '_upgradePoolDomain' >Thread-15911404::DEBUG::2013-11-19 21:33:57,040::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`2e031752-9d1a-4dd2-80a3-76c2c699d4b3`::Request was made in '/usr/share/vdsm/storage/spm.py' line '377' at '_upgradePoolDomain' >Thread-15911403::DEBUG::2013-11-19 21:33:57,043::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' for lock type 'exclusive' >Thread-15911404::DEBUG::2013-11-19 21:33:57,044::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'exclusive' >Thread-15911403::DEBUG::2013-11-19 21:33:57,045::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' is free. Now locking as 'exclusive' (1 active user) >Thread-15911403::DEBUG::2013-11-19 21:33:57,051::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`232a75f0-99b2-4d73-aff6-f7e9d1a1f5c4`::Granted request >Thread-15911404::DEBUG::2013-11-19 21:33:57,055::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'exclusive' (1 active user) >Thread-15911403::DEBUG::2013-11-19 21:33:57,055::spm::385::Storage.SPM::(_upgradePoolDomain) Domain `45cdff8b-89c0-4926-a8c7-80354064bfa6` is not a data domain it is an 2 domain, not upgrading >Thread-15911404::DEBUG::2013-11-19 21:33:57,056::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`2e031752-9d1a-4dd2-80a3-76c2c699d4b3`::Granted request >Thread-15911403::DEBUG::2013-11-19 21:33:57,057::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' >Thread-15911404::DEBUG::2013-11-19 21:33:57,063::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15911403::DEBUG::2013-11-19 21:33:57,067::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' (0 active users) >Thread-15911404::DEBUG::2013-11-19 21:33:57,067::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15911403::DEBUG::2013-11-19 21:33:57,068::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' is free, finding out if anyone is waiting for it. >Thread-15911404::DEBUG::2013-11-19 21:33:57,069::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15911403::DEBUG::2013-11-19 21:33:57,069::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6', Clearing records. >Thread-15911404::DEBUG::2013-11-19 21:33:57,070::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15911403::DEBUG::2013-11-19 21:33:57,071::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' >Thread-15911404::DEBUG::2013-11-19 21:33:57,072::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15911403::DEBUG::2013-11-19 21:33:57,077::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' (0 active users) >Thread-15911404::DEBUG::2013-11-19 21:33:57,078::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15911403::DEBUG::2013-11-19 21:33:57,079::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' is free, finding out if anyone is waiting for it. >Thread-15911403::DEBUG::2013-11-19 21:33:57,080::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6', Clearing records. >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:57,392::blockSD::791::Storage.Misc.excCmd::(mountMaster) SUCCESS: <err> = ''; <rc> = 0 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:57,393::blockSD::816::Storage.Misc.excCmd::(mountMaster) '/usr/bin/sudo -n /sbin/tune2fs -j /dev/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:33:57,700::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911404::DEBUG::2013-11-19 21:33:57,873::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911404::DEBUG::2013-11-19 21:33:57,882::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15911404::DEBUG::2013-11-19 21:33:57,883::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'SDUUID=772472cb-2429-466d-96cd-00b3301ef78a', u'VERSION=2', u'LOCKPOLICY=', u'VGUUID=XTb294-NYv6-dVpO-bB1n-4f9P-d0GL-dhd2DI', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'LEASETIMESEC=5', u'LOCKRENEWALINTERVALSEC=5', u'ROLE=Regular', u'PHYBLKSIZE=512', u'DESCRIPTION=Database', u'IOOPTIMEOUTSEC=1', u'CLASS=Data', u'TYPE=FCP', u'PV0=pv:3600c0ff0001520c7414f605001000000,uuid:NujM33-vNiD-sBHY-mrPo-EQ2f-1qsz-HXfn0q,pestart:0,pecount:8929,mapoffset:0', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'MASTER_VERSION=0', u'_SHA_CKSUM=ef8840d9fdef24516a56fd970bed1960d2b1ae92'] >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:57,884::blockSD::816::Storage.Misc.excCmd::(mountMaster) FAILED: <err> = 'The filesystem already has a journal.\n'; <rc> = 1 >Thread-15911404::DEBUG::2013-11-19 21:33:57,886::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15911404::DEBUG::2013-11-19 21:33:57,892::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15911404::DEBUG::2013-11-19 21:33:57,893::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:57,895::fileUtils::83::Storage.Misc.excCmd::(mount) '/usr/bin/sudo -n /bin/mount /dev/6b892067-ac7e-4039-a7be-5b7387525b16/master /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15911404::DEBUG::2013-11-19 21:33:57,896::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15911404::DEBUG::2013-11-19 21:33:58,115::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15911404::DEBUG::2013-11-19 21:33:58,117::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15911404::DEBUG::2013-11-19 21:33:58,118::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15911404::DEBUG::2013-11-19 21:33:58,119::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,325::fileUtils::83::Storage.Misc.excCmd::(mount) SUCCESS: <err> = ''; <rc> = 0 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,326::blockSD::833::Storage.Misc.excCmd::(mountMaster) '/usr/bin/sudo -n /bin/chown vdsm:kvm /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,746::blockSD::833::Storage.Misc.excCmd::(mountMaster) SUCCESS: <err> = ''; <rc> = 0 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::WARNING::2013-11-19 21:33:58,749::spm::490::Storage.SPM::(start) Backup domain validation failed >Traceback (most recent call last): > File "/usr/share/vdsm/storage/spm.py", line 488, in start > File "/usr/share/vdsm/storage/sp.py", line 1208, in checkBackupDomain > File "/usr/share/vdsm/storage/sd.py", line 666, in isBackup > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,751::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,752::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,752::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,753::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,754::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,754::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,755::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,756::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,756::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,757::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,757::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,758::persistentDict::146::Storage.PersistentDict::(transaction) Flushing changes >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,759::persistentDict::265::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['MASTER_VERSION=0', 'ROLE=Regular', '_SHA_CKSUM=1deb73127f89496e2cb17e51bbfae1df6c1879d1'] >d3f111cf-7b31-41dc-bcdf-c918a6697f62::WARNING::2013-11-19 21:33:58,760::persistentDict::150::Storage.PersistentDict::(transaction) Error in transaction, rolling back changes >Traceback (most recent call last): > File "/usr/share/vdsm/storage/persistentDict.py", line 147, in transaction > File "/usr/share/vdsm/storage/persistentDict.py", line 266, in flush > File "/usr/share/vdsm/storage/fileSD.py", line 58, in writelines > File "/usr/share/vdsm/storage/processPool.py", line 33, in wrapper > File "/usr/share/vdsm/storage/processPool.py", line 44, in runExternally > File "/usr/share/vdsm/storage/processPool.py", line 127, in runExternally >IOError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata.new' >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,761::persistentDict::265::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['_SHA_CKSUM=da39a3ee5e6b4b0d3255bfef95601890afd80709'] >d3f111cf-7b31-41dc-bcdf-c918a6697f62::ERROR::2013-11-19 21:33:58,762::spm::526::Storage.SPM::(start) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/spm.py", line 493, in start > File "/usr/share/vdsm/storage/sp.py", line 637, in _regulariseOutdatedMasters > File "/usr/share/vdsm/storage/sp.py", line 631, in _setRegularSD > File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ > File "/usr/share/vdsm/storage/persistentDict.py", line 152, in transaction > File "/usr/share/vdsm/storage/persistentDict.py", line 266, in flush > File "/usr/share/vdsm/storage/fileSD.py", line 58, in writelines > File "/usr/share/vdsm/storage/processPool.py", line 33, in wrapper > File "/usr/share/vdsm/storage/processPool.py", line 44, in runExternally > File "/usr/share/vdsm/storage/processPool.py", line 127, in runExternally >IOError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata.new' >d3f111cf-7b31-41dc-bcdf-c918a6697f62::ERROR::2013-11-19 21:33:58,763::spm::527::Storage.SPM::(start) failed: [Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata.new' >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,765::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6301b211-13c4-49fe-8aaa-74bb58f8a3e1`::Request was made in '/usr/share/vdsm/storage/spm.py' line '584' at '_stop' >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,765::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,766::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,767::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6301b211-13c4-49fe-8aaa-74bb58f8a3e1`::Granted request >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,769::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`1e585214-39b8-407f-94b4-a7beb4836335`::Request was made in '/usr/share/vdsm/storage/spm.py' line '601' at '_stop' >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,769::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' for lock type 'exclusive' >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,770::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' is free. Now locking as 'exclusive' (1 active user) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,771::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`1e585214-39b8-407f-94b4-a7beb4836335`::Granted request >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,772::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,773::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' (0 active users) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,773::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' is free, finding out if anyone is waiting for it. >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,774::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e', Clearing records. >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,775::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,776::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,777::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,777::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,778::taskManager::118::TaskManager::(unloadTasks) Entry. >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,779::taskManager::122::TaskManager::(unloadTasks) Return >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,782::spm::195::Storage.SPM::(__cleanupMasterMount) unmounting /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:58,901::fileUtils::112::Storage.Misc.excCmd::(umount) '/usr/bin/sudo -n /bin/umount -f /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:59,269::fileUtils::112::Storage.Misc.excCmd::(umount) SUCCESS: <err> = ''; <rc> = 0 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:59,272::spm::290::Storage.SPM::(__cleanupSPM) cleaning up SPM: c4e5889a-348d-11e2-83fe-9f72369eab30 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:59,273::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:59,273::persistentDict::146::Storage.PersistentDict::(transaction) Flushing changes >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:59,274::persistentDict::265::Storage.PersistentDict::(flush) about to write lines (VGTagMetadataRW)=[u'CLASS=Data', u'DESCRIPTION=OS', u'IOOPTIMEOUTSEC=10', u'LEASERETRIES=3', u'LEASETIMESEC=60', u'LOCKPOLICY=', u'LOCKRENEWALINTERVALSEC=5', u'LOGBLKSIZE=512', u'MASTER_VERSION=1', u'PHYBLKSIZE=512', u'POOL_DESCRIPTION=Default', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_ID=-1', u'POOL_SPM_LVER=284', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'ROLE=Master', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'TYPE=FCP', u'VERSION=2', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', '_SHA_CKSUM=fa1ddc0ebaad4a3d815671fdef2d27b7d9f8a82e'] >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:59,275::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:59,276::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:59,277::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:59,277::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:59,278::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:59,279::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:59,949::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:59,952::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:59,953::blockSD::145::storage.Metadata.VGTagMetadataRW::(writelines) Updating metadata adding=MDT_POOL_SPM_ID=-1, MDT__SHA_CKSUM=fa1ddc0ebaad4a3d815671fdef2d27b7d9f8a82e removing=MDT_POOL_SPM_ID=2, MDT__SHA_CKSUM=444b3b54067466f11e0599712ab301365a6be2aa >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:33:59,954::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --deltag MDT_POOL_SPM_ID=2 --deltag MDT__SHA_CKSUM=444b3b54067466f11e0599712ab301365a6be2aa --addtag MDT__SHA_CKSUM=fa1ddc0ebaad4a3d815671fdef2d27b7d9f8a82e --addtag MDT_POOL_SPM_ID=-1 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:00,637::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:00,638::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:00,639::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:00,639::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >d3f111cf-7b31-41dc-bcdf-c918a6697f62::INFO::2013-11-19 21:34:00,640::safelease::83::ClusterLock::(release) Releasing cluster lock for domain 6b892067-ac7e-4039-a7be-5b7387525b16 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:00,641::safelease::84::Storage.Misc.excCmd::(release) '/usr/libexec/vdsm/spmstop.sh 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd /usr/libexec/vdsm/) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,110::safelease::84::Storage.Misc.excCmd::(release) SUCCESS: <err> = ''; <rc> = 0 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,111::safelease::87::ClusterLock::(release) Cluster lock released successfully >d3f111cf-7b31-41dc-bcdf-c918a6697f62::ERROR::2013-11-19 21:34:02,112::task::868::TaskManager.Task::(_setError) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 876, in _run > File "/usr/share/vdsm/storage/task.py", line 303, in run > File "/usr/share/vdsm/storage/spm.py", line 493, in start > File "/usr/share/vdsm/storage/sp.py", line 637, in _regulariseOutdatedMasters > File "/usr/share/vdsm/storage/sp.py", line 631, in _setRegularSD > File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ > File "/usr/share/vdsm/storage/persistentDict.py", line 152, in transaction > File "/usr/share/vdsm/storage/persistentDict.py", line 266, in flush > File "/usr/share/vdsm/storage/fileSD.py", line 58, in writelines > File "/usr/share/vdsm/storage/processPool.py", line 33, in wrapper > File "/usr/share/vdsm/storage/processPool.py", line 44, in runExternally > File "/usr/share/vdsm/storage/processPool.py", line 127, in runExternally >IOError: [Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata.new' >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,113::task::495::TaskManager.Task::(_debug) Task d3f111cf-7b31-41dc-bcdf-c918a6697f62: Task._run: d3f111cf-7b31-41dc-bcdf-c918a6697f62 () {} failed - stopping task >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,113::task::495::TaskManager.Task::(_debug) Task d3f111cf-7b31-41dc-bcdf-c918a6697f62: stopping in state running (force False) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,114::task::495::TaskManager.Task::(_debug) Task d3f111cf-7b31-41dc-bcdf-c918a6697f62: ref 1 aborting True >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,115::task::918::TaskManager.Task::(_runJobs) aborting: Task is aborted: "[Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata.new'" - code 100 >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,115::task::495::TaskManager.Task::(_debug) Task d3f111cf-7b31-41dc-bcdf-c918a6697f62: ref 0 aborting True >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,116::task::495::TaskManager.Task::(_debug) Task d3f111cf-7b31-41dc-bcdf-c918a6697f62: Task._doAbort: force False >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,116::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,117::task::495::TaskManager.Task::(_debug) Task d3f111cf-7b31-41dc-bcdf-c918a6697f62: moving from state running -> state aborting >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,117::task::495::TaskManager.Task::(_debug) Task d3f111cf-7b31-41dc-bcdf-c918a6697f62: _aborting: recover policy auto >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,118::task::495::TaskManager.Task::(_debug) Task d3f111cf-7b31-41dc-bcdf-c918a6697f62: moving from state aborting -> state racquiring >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,118::task::495::TaskManager.Task::(_debug) Task d3f111cf-7b31-41dc-bcdf-c918a6697f62: moving from state racquiring -> state recovering >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,119::task::495::TaskManager.Task::(_debug) Task d3f111cf-7b31-41dc-bcdf-c918a6697f62: _recover >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,119::task::495::TaskManager.Task::(_debug) Task d3f111cf-7b31-41dc-bcdf-c918a6697f62: running recovery None >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,120::task::495::TaskManager.Task::(_debug) Task d3f111cf-7b31-41dc-bcdf-c918a6697f62: Recover Done: state recovering >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,120::task::495::TaskManager.Task::(_debug) Task d3f111cf-7b31-41dc-bcdf-c918a6697f62: moving from state recovering -> state recovered >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,121::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,121::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,123::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,123::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,124::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,124::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >d3f111cf-7b31-41dc-bcdf-c918a6697f62::DEBUG::2013-11-19 21:34:02,125::threadPool::47::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 >Thread-42838::DEBUG::2013-11-19 21:34:06,548::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-42838::DEBUG::2013-11-19 21:34:06,550::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 21:34:07,209::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-42838::DEBUG::2013-11-19 21:34:07,212::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1024767::ERROR::2013-11-19 21:34:07,866::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:34:17,879::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:34:27,893::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911406::INFO::2013-11-19 21:34:34,768::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911406::DEBUG::2013-11-19 21:34:34,769::task::495::TaskManager.Task::(_debug) Task 74fecb06-5d4f-4bbf-9783-2945d9e62584: moving from state init -> state preparing >Thread-15911406::DEBUG::2013-11-19 21:34:34,770::task::495::TaskManager.Task::(_debug) Task 74fecb06-5d4f-4bbf-9783-2945d9e62584: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000329971313477', 'lastCheck': 1384896867.2409911, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142693519592', 'lastCheck': 1384896872.21348, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896867.894151, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000482082366943', 'lastCheck': 1384896868.8891749, 'valid': True, 'code': 0}} >Thread-15911406::DEBUG::2013-11-19 21:34:34,771::task::495::TaskManager.Task::(_debug) Task 74fecb06-5d4f-4bbf-9783-2945d9e62584: moving from state preparing -> state finished >Thread-15911406::DEBUG::2013-11-19 21:34:34,771::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911406::DEBUG::2013-11-19 21:34:34,772::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911406::DEBUG::2013-11-19 21:34:34,773::task::495::TaskManager.Task::(_debug) Task 74fecb06-5d4f-4bbf-9783-2945d9e62584: ref 0 aborting False >Thread-15911406::INFO::2013-11-19 21:34:34,773::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000329971313477', 'lastCheck': 1384896867.2409911, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142693519592', 'lastCheck': 1384896872.21348, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896867.894151, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000482082366943', 'lastCheck': 1384896868.8891749, 'valid': True, 'code': 0}} >Thread-15911407::DEBUG::2013-11-19 21:34:34,906::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVdsCapabilities with () {} >Thread-15911407::DEBUG::2013-11-19 21:34:35,375::utils::579::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-img-rhev' (cwd None) >Thread-15911407::DEBUG::2013-11-19 21:34:35,896::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911407::DEBUG::2013-11-19 21:34:35,897::utils::579::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-kvm-rhev' (cwd None) >Thread-15911407::DEBUG::2013-11-19 21:34:36,385::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911407::DEBUG::2013-11-19 21:34:36,386::utils::579::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" vdsm' (cwd None) >Thread-15911407::DEBUG::2013-11-19 21:34:36,804::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911407::DEBUG::2013-11-19 21:34:36,806::utils::579::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" spice-server' (cwd None) >Thread-15911407::DEBUG::2013-11-19 21:34:37,238::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911407::DEBUG::2013-11-19 21:34:37,239::utils::579::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" libvirt' (cwd None) >Thread-15911407::DEBUG::2013-11-19 21:34:37,656::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911407::DEBUG::2013-11-19 21:34:37,670::clientIF::59::vds::(wrapper) return getVdsCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:c936ebb147a8'}], 'FC': []}, 'packages2': {'kernel': {'release': '279.11.1.el6.x86_64', 'buildtime': 1348297826.0, 'version': '2.6.32'}, 'spice-server': {'release': '10.el6', 'buildtime': '1337611492', 'version': '0.10.1'}, 'vdsm': {'release': '113.4.el6_3', 'buildtime': '1347534072', 'version': '4.9'}, 'qemu-kvm': {'release': '2.295.el6_3.2', 'buildtime': '1346673796', 'version': '0.12.1.2'}, 'libvirt': {'release': '21.el6_3.5', 'buildtime': '1348687577', 'version': '0.9.10'}, 'qemu-img': {'release': '2.295.el6_3.2', 'buildtime': '1346673796', 'version': '0.12.1.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz', 'hooks': {'before_vm_start': {'50_vhostmd': {'md5': '1635e45584995d9ad2629939cb0bcdbf'}}, 'after_vm_destroy': {'50_vhostmd': {'md5': '7e7509430304099f38e4a9de966af090'}}, 'before_vm_dehibernate': {'50_vhostmd': {'md5': '1635e45584995d9ad2629939cb0bcdbf'}}, 'before_vm_migrate_destination': {'50_vhostmd': {'md5': '1635e45584995d9ad2629939cb0bcdbf'}}}, 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'addr': '192.168.100.133', 'cfg': {'IPADDR': '192.168.100.133', 'GATEWAY': '192.168.100.1', 'DELAY': '0', 'NETMASK': '255.255.128.0', 'STP': 'no', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ports': ['bond0'], 'netmask': '255.255.128.0', 'stp': 'off', 'gateway': '192.168.100.1'}, 'DMZ': {'addr': '172.16.1.133', 'cfg': {'IPADDR': '172.16.1.133', 'DELAY': '0', 'NETMASK': '255.255.255.0', 'STP': 'no', 'DEVICE': 'DMZ', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ports': ['eth0'], 'netmask': '255.255.255.0', 'stp': 'off', 'gateway': '0.0.0.0'}}, 'uuid': '30343536-3138-5A43-4A32-333630373442_AC:16:2D:76:EB:F0', 'lastClientIface': 'rhevm', 'nics': {'eth3': {'hwaddr': 'AC:16:2D:76:EB:F2', 'netmask': '', 'speed': 1000, 'addr': '', 'permhwaddr': 'AC:16:2D:76:EB:F3'}, 'eth2': {'hwaddr': 'AC:16:2D:76:EB:F2', 'netmask': '', 'speed': 1000, 'addr': '', 'permhwaddr': 'AC:16:2D:76:EB:F2'}, 'eth1': {'hwaddr': 'AC:16:2D:76:EB:F1', 'netmask': '', 'speed': 0, 'addr': ''}, 'eth0': {'hwaddr': 'AC:16:2D:76:EB:F0', 'netmask': '', 'speed': 1000, 'addr': ''}}, 'software_revision': '113.4', 'management_ip': '192.168.100.133', 'clusterLevels': ['3.0'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,dca,sse4_1,sse4_2,x2apic,popcnt,aes,xsave,avx,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_486,model_pentium,model_pentium2,model_pentium3,model_pentiumpro,model_qemu32,model_cpu64-rhel5,model_cpu64-rhel6,model_kvm32,model_coreduo,model_kvm64,model_core2duo,model_n270,model_Conroe,model_Penryn,model_Nehalem,model_Westmere,model_SandyBridge,model_Opteron_G1', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:c936ebb147a8', 'memSize': '64392', 'reservedMem': '256', 'bondings': {'bond4': {'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'netmask': '', 'addr': '', 'slaves': []}, 'bond0': {'hwaddr': 'AC:16:2D:76:EB:F2', 'cfg': {'DEVICE': 'bond0', 'BRIDGE': 'rhevm', 'BOOTPROTO': 'none', 'ONBOOT': 'yes', 'BONDING_OPTS': 'mode=1 miimon=100'}, 'netmask': '', 'addr': '', 'slaves': ['eth2', 'eth3']}, 'bond1': {'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'netmask': '', 'addr': '', 'slaves': []}, 'bond2': {'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'netmask': '', 'addr': '', 'slaves': []}, 'bond3': {'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'netmask': '', 'addr': '', 'slaves': []}}, 'software_version': '4.9', 'cpuSpeed': '1995.126', 'cpuSockets': '2', 'vlans': {}, 'cpuCores': '12', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'version_name': 'Snow Man', 'emulatedMachines': [u'rhel6.3.0', u'pc', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '20121012.0.el6_3', 'version': '6.3', 'name': 'RHEV Hypervisor'}, 'lastClient': '192.168.100.135'}} >Thread-1024767::ERROR::2013-11-19 21:34:38,135::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911412::DEBUG::2013-11-19 21:34:44,937::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911412::INFO::2013-11-19 21:34:44,939::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911412::DEBUG::2013-11-19 21:34:44,939::task::495::TaskManager.Task::(_debug) Task f5fc2942-aa8e-4768-b5b7-1ef311448185: moving from state init -> state preparing >Thread-15911412::DEBUG::2013-11-19 21:34:44,940::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=Free spmLver=284 spmId=-1 >Thread-15911412::DEBUG::2013-11-19 21:34:44,941::task::495::TaskManager.Task::(_debug) Task f5fc2942-aa8e-4768-b5b7-1ef311448185: finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 284}} >Thread-15911412::DEBUG::2013-11-19 21:34:44,941::task::495::TaskManager.Task::(_debug) Task f5fc2942-aa8e-4768-b5b7-1ef311448185: moving from state preparing -> state finished >Thread-15911412::DEBUG::2013-11-19 21:34:44,942::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911412::DEBUG::2013-11-19 21:34:44,942::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911412::DEBUG::2013-11-19 21:34:44,943::task::495::TaskManager.Task::(_debug) Task f5fc2942-aa8e-4768-b5b7-1ef311448185: ref 0 aborting False >Thread-15911412::INFO::2013-11-19 21:34:44,943::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 284}} >Thread-15911413::DEBUG::2013-11-19 21:34:44,993::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911413::INFO::2013-11-19 21:34:44,994::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: spmStart, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 prevID=-1 prevLVER=284 recoveryMode=0 scsiFencing=false maxHostID=250 domVersion=2) >Thread-15911413::DEBUG::2013-11-19 21:34:44,995::task::495::TaskManager.Task::(_debug) Task 44a1be5a-8538-4745-8f2c-b3e70865e4e7: moving from state init -> state preparing >Thread-15911413::DEBUG::2013-11-19 21:34:44,996::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`33916515-dffe-468a-be56-a8223b622f4a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '330' at 'public_spmStart' >Thread-15911413::DEBUG::2013-11-19 21:34:44,997::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >Thread-15911413::DEBUG::2013-11-19 21:34:44,998::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >Thread-15911413::DEBUG::2013-11-19 21:34:44,999::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`33916515-dffe-468a-be56-a8223b622f4a`::Granted request >Thread-15911413::DEBUG::2013-11-19 21:34:45,000::task::495::TaskManager.Task::(_debug) Task 44a1be5a-8538-4745-8f2c-b3e70865e4e7: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (exclusive) >Thread-15911413::DEBUG::2013-11-19 21:34:45,000::task::495::TaskManager.Task::(_debug) Task 44a1be5a-8538-4745-8f2c-b3e70865e4e7: ref 1 aborting False >Thread-15911413::DEBUG::2013-11-19 21:34:45,001::task::495::TaskManager.Task::(_debug) Task 44a1be5a-8538-4745-8f2c-b3e70865e4e7: Prepare: 1 jobs exist, move to acquiring >Thread-15911413::DEBUG::2013-11-19 21:34:45,001::task::495::TaskManager.Task::(_debug) Task 44a1be5a-8538-4745-8f2c-b3e70865e4e7: moving from state preparing -> state acquiring >Thread-15911413::DEBUG::2013-11-19 21:34:45,002::task::495::TaskManager.Task::(_debug) Task 44a1be5a-8538-4745-8f2c-b3e70865e4e7: moving from state acquiring -> state queued >Thread-15911413::DEBUG::2013-11-19 21:34:45,002::taskManager::37::TaskManager::(_queueTask) queueing task: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911413::DEBUG::2013-11-19 21:34:45,003::taskManager::43::TaskManager::(_queueTask) task queued: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >934c4d35-aab6-4fba-9996-9cc07dd7d991::DEBUG::2013-11-19 21:34:45,003::threadPool::47::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-15911413::DEBUG::2013-11-19 21:34:45,004::task::1183::TaskManager.Task::(prepare) returning >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,005::threadPool::192::Misc.ThreadPool.WorkerThread::(run) Task: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 running: <bound method Task.commit of <storage.task.Task instance at 0x7f47180ba488>> with: None >Thread-15911413::DEBUG::2013-11-19 21:34:45,005::task::495::TaskManager.Task::(_debug) Task 44a1be5a-8538-4745-8f2c-b3e70865e4e7: ref 0 aborting False >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,006::task::1195::TaskManager.Task::(commit) Commiting task: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911413::INFO::2013-11-19 21:34:45,006::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: spmStart, Return response: {'status': {'message': 'OK', 'code': 0}, 'uuid': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,007::task::495::TaskManager.Task::(_debug) Task 44a1be5a-8538-4745-8f2c-b3e70865e4e7: moving from state queued -> state running >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,008::task::907::TaskManager.Task::(_runJobs) Task.run: running job 0: spmStart: <bound method SPM.start of <storage.spm.SPM instance at 0x1bf9f38>> (args: ('c4e5889a-348d-11e2-83fe-9f72369eab30', -1, '284', 0, 'false', 250, 2) kwargs: {}) >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,009::task::300::TaskManager.Task::(run) Job.run: running spmStart: <bound method SPM.start of <storage.spm.SPM instance at 0x1bf9f38>> (args: ('c4e5889a-348d-11e2-83fe-9f72369eab30', -1, '284', 0, 'false', 250, 2) kwargs: {}) callback None >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,009::spm::433::Storage.SPM::(start) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,010::misc::1010::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,010::misc::1012::SamplingMethod::(__call__) Got in to sampling method >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,011::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,011::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,012::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,013::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,013::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,014::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,664::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,668::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,669::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=284', u'_SHA_CKSUM=fa1ddc0ebaad4a3d815671fdef2d27b7d9f8a82e', u'POOL_SPM_ID=-1'] >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,670::misc::1020::SamplingMethod::(__call__) Returning last result >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,671::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,672::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,672::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,673::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,674::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:45,675::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15911414::DEBUG::2013-11-19 21:34:46,028::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911414::INFO::2013-11-19 21:34:46,030::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911414::DEBUG::2013-11-19 21:34:46,031::task::495::TaskManager.Task::(_debug) Task 1409ee5c-9802-489c-ab73-13069d3ce063: moving from state init -> state preparing >Thread-15911414::DEBUG::2013-11-19 21:34:46,032::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911414::DEBUG::2013-11-19 21:34:46,032::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911414::DEBUG::2013-11-19 21:34:46,033::task::495::TaskManager.Task::(_debug) Task 1409ee5c-9802-489c-ab73-13069d3ce063: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911414::DEBUG::2013-11-19 21:34:46,034::task::495::TaskManager.Task::(_debug) Task 1409ee5c-9802-489c-ab73-13069d3ce063: moving from state preparing -> state finished >Thread-15911414::DEBUG::2013-11-19 21:34:46,034::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911414::DEBUG::2013-11-19 21:34:46,035::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911414::DEBUG::2013-11-19 21:34:46,035::task::495::TaskManager.Task::(_debug) Task 1409ee5c-9802-489c-ab73-13069d3ce063: ref 0 aborting False >Thread-15911414::INFO::2013-11-19 21:34:46,036::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:46,281::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:46,284::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:46,285::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=284', u'_SHA_CKSUM=fa1ddc0ebaad4a3d815671fdef2d27b7d9f8a82e', u'POOL_SPM_ID=-1'] >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:46,287::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:46,288::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:46,289::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:46,289::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:46,290::persistentDict::163::Storage.PersistentDict::(__init__) Created a persistant dict with LvMetadataRW backend >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:46,291::blockSD::164::Storage.Misc.excCmd::(readlines) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:46,701::blockSD::164::Storage.Misc.excCmd::(readlines) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000226921 s, 9.0 MB/s\n'; <rc> = 0 >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:46,703::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000226921 s, 9.0 MB/s'], size: 2048 >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:46,703::persistentDict::204::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >44a1be5a-8538-4745-8f2c-b3e70865e4e7::WARNING::2013-11-19 21:34:46,704::persistentDict::226::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:46,704::persistentDict::163::Storage.PersistentDict::(__init__) Created a persistant dict with VGTagMetadataRW backend >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:46,705::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:46,706::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:46,706::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:46,707::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:46,708::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:46,709::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15911416::DEBUG::2013-11-19 21:34:47,059::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911416::INFO::2013-11-19 21:34:47,061::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911416::DEBUG::2013-11-19 21:34:47,062::task::495::TaskManager.Task::(_debug) Task 3956c1b1-80a9-4b7f-8d57-eee5239d1809: moving from state init -> state preparing >Thread-15911416::DEBUG::2013-11-19 21:34:47,063::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911416::DEBUG::2013-11-19 21:34:47,063::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911416::DEBUG::2013-11-19 21:34:47,064::task::495::TaskManager.Task::(_debug) Task 3956c1b1-80a9-4b7f-8d57-eee5239d1809: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911416::DEBUG::2013-11-19 21:34:47,065::task::495::TaskManager.Task::(_debug) Task 3956c1b1-80a9-4b7f-8d57-eee5239d1809: moving from state preparing -> state finished >Thread-15911416::DEBUG::2013-11-19 21:34:47,065::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911416::DEBUG::2013-11-19 21:34:47,066::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911416::DEBUG::2013-11-19 21:34:47,066::task::495::TaskManager.Task::(_debug) Task 3956c1b1-80a9-4b7f-8d57-eee5239d1809: ref 0 aborting False >Thread-15911416::INFO::2013-11-19 21:34:47,067::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:47,296::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:47,299::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:47,300::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=284', u'_SHA_CKSUM=fa1ddc0ebaad4a3d815671fdef2d27b7d9f8a82e', u'POOL_SPM_ID=-1'] >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:47,301::safelease::52::ClusterLock::(acquire) Acquiring cluster lock for domain 6b892067-ac7e-4039-a7be-5b7387525b16 >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:34:47,302::safelease::61::Storage.Misc.excCmd::(acquire) '/usr/bin/sudo -n /usr/bin/setsid /usr/bin/ionice -c1 -n0 /bin/su vdsm -s /bin/sh -c "/usr/libexec/vdsm/spmprotect.sh start 6b892067-ac7e-4039-a7be-5b7387525b16 2 5 /dev/6b892067-ac7e-4039-a7be-5b7387525b16/leases 60000 10000 3"' (cwd /usr/libexec/vdsm/) >Thread-42838::DEBUG::2013-11-19 21:34:47,516::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15911417::DEBUG::2013-11-19 21:34:48,079::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911417::INFO::2013-11-19 21:34:48,081::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911417::DEBUG::2013-11-19 21:34:48,081::task::495::TaskManager.Task::(_debug) Task 5b664412-99ef-423e-bdd2-c34bb4a26f29: moving from state init -> state preparing >Thread-15911417::DEBUG::2013-11-19 21:34:48,082::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911417::DEBUG::2013-11-19 21:34:48,085::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911417::DEBUG::2013-11-19 21:34:48,086::task::495::TaskManager.Task::(_debug) Task 5b664412-99ef-423e-bdd2-c34bb4a26f29: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911418::INFO::2013-11-19 21:34:48,088::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911417::DEBUG::2013-11-19 21:34:48,088::task::495::TaskManager.Task::(_debug) Task 5b664412-99ef-423e-bdd2-c34bb4a26f29: moving from state preparing -> state finished >Thread-15911418::DEBUG::2013-11-19 21:34:48,089::task::495::TaskManager.Task::(_debug) Task 146a0e76-4d15-4451-89b4-b4bd4ddf67a0: moving from state init -> state preparing >Thread-15911417::DEBUG::2013-11-19 21:34:48,089::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911418::DEBUG::2013-11-19 21:34:48,097::task::495::TaskManager.Task::(_debug) Task 146a0e76-4d15-4451-89b4-b4bd4ddf67a0: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000545978546143', 'lastCheck': 1384896877.4420099, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00162196159363', 'lastCheck': 1384896882.2284839, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896878.1361749, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00449299812317', 'lastCheck': 1384896879.028841, 'valid': True, 'code': 0}} >Thread-15911417::DEBUG::2013-11-19 21:34:48,097::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911418::DEBUG::2013-11-19 21:34:48,106::task::495::TaskManager.Task::(_debug) Task 146a0e76-4d15-4451-89b4-b4bd4ddf67a0: moving from state preparing -> state finished >Thread-15911417::DEBUG::2013-11-19 21:34:48,110::task::495::TaskManager.Task::(_debug) Task 5b664412-99ef-423e-bdd2-c34bb4a26f29: ref 0 aborting False >Thread-15911418::DEBUG::2013-11-19 21:34:48,110::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911417::INFO::2013-11-19 21:34:48,111::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911418::DEBUG::2013-11-19 21:34:48,113::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911418::DEBUG::2013-11-19 21:34:48,117::task::495::TaskManager.Task::(_debug) Task 146a0e76-4d15-4451-89b4-b4bd4ddf67a0: ref 0 aborting False >Thread-15911418::INFO::2013-11-19 21:34:48,118::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000545978546143', 'lastCheck': 1384896877.4420099, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00162196159363', 'lastCheck': 1384896882.2284839, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896878.1361749, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00449299812317', 'lastCheck': 1384896879.028841, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:34:48,242::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 21:34:48,331::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911420::DEBUG::2013-11-19 21:34:49,125::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911420::INFO::2013-11-19 21:34:49,132::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911420::DEBUG::2013-11-19 21:34:49,138::task::495::TaskManager.Task::(_debug) Task 2e2ce7dc-84b3-44b5-9caa-512848c7b094: moving from state init -> state preparing >Thread-15911420::DEBUG::2013-11-19 21:34:49,142::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911420::DEBUG::2013-11-19 21:34:49,145::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911420::DEBUG::2013-11-19 21:34:49,148::task::495::TaskManager.Task::(_debug) Task 2e2ce7dc-84b3-44b5-9caa-512848c7b094: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911420::DEBUG::2013-11-19 21:34:49,152::task::495::TaskManager.Task::(_debug) Task 2e2ce7dc-84b3-44b5-9caa-512848c7b094: moving from state preparing -> state finished >Thread-15911420::DEBUG::2013-11-19 21:34:49,155::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911420::DEBUG::2013-11-19 21:34:49,159::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911420::DEBUG::2013-11-19 21:34:49,159::task::495::TaskManager.Task::(_debug) Task 2e2ce7dc-84b3-44b5-9caa-512848c7b094: ref 0 aborting False >Thread-15911420::INFO::2013-11-19 21:34:49,160::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911421::DEBUG::2013-11-19 21:34:50,180::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911421::INFO::2013-11-19 21:34:50,184::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911421::DEBUG::2013-11-19 21:34:50,188::task::495::TaskManager.Task::(_debug) Task 0c59133d-6af2-4fff-9e88-196d2f31d71b: moving from state init -> state preparing >Thread-15911421::DEBUG::2013-11-19 21:34:50,194::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911421::DEBUG::2013-11-19 21:34:50,194::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911421::DEBUG::2013-11-19 21:34:50,200::task::495::TaskManager.Task::(_debug) Task 0c59133d-6af2-4fff-9e88-196d2f31d71b: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911421::DEBUG::2013-11-19 21:34:50,201::task::495::TaskManager.Task::(_debug) Task 0c59133d-6af2-4fff-9e88-196d2f31d71b: moving from state preparing -> state finished >Thread-15911421::DEBUG::2013-11-19 21:34:50,205::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911421::DEBUG::2013-11-19 21:34:50,205::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911421::DEBUG::2013-11-19 21:34:50,211::task::495::TaskManager.Task::(_debug) Task 0c59133d-6af2-4fff-9e88-196d2f31d71b: ref 0 aborting False >Thread-15911421::INFO::2013-11-19 21:34:50,216::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911423::DEBUG::2013-11-19 21:34:51,229::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911423::INFO::2013-11-19 21:34:51,230::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911423::DEBUG::2013-11-19 21:34:51,231::task::495::TaskManager.Task::(_debug) Task 9966fa7d-6086-4367-80cb-ca783f1488fa: moving from state init -> state preparing >Thread-15911423::DEBUG::2013-11-19 21:34:51,234::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911423::DEBUG::2013-11-19 21:34:51,239::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911423::DEBUG::2013-11-19 21:34:51,244::task::495::TaskManager.Task::(_debug) Task 9966fa7d-6086-4367-80cb-ca783f1488fa: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911423::DEBUG::2013-11-19 21:34:51,244::task::495::TaskManager.Task::(_debug) Task 9966fa7d-6086-4367-80cb-ca783f1488fa: moving from state preparing -> state finished >Thread-15911423::DEBUG::2013-11-19 21:34:51,247::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911423::DEBUG::2013-11-19 21:34:51,253::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911423::DEBUG::2013-11-19 21:34:51,256::task::495::TaskManager.Task::(_debug) Task 9966fa7d-6086-4367-80cb-ca783f1488fa: ref 0 aborting False >Thread-15911423::INFO::2013-11-19 21:34:51,263::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911424::DEBUG::2013-11-19 21:34:52,574::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911424::INFO::2013-11-19 21:34:52,576::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911424::DEBUG::2013-11-19 21:34:52,586::task::495::TaskManager.Task::(_debug) Task 987662e6-d2ad-46d5-89cc-96c9825fef0f: moving from state init -> state preparing >Thread-15911424::DEBUG::2013-11-19 21:34:52,592::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911424::DEBUG::2013-11-19 21:34:52,595::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911424::DEBUG::2013-11-19 21:34:52,600::task::495::TaskManager.Task::(_debug) Task 987662e6-d2ad-46d5-89cc-96c9825fef0f: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911424::DEBUG::2013-11-19 21:34:52,603::task::495::TaskManager.Task::(_debug) Task 987662e6-d2ad-46d5-89cc-96c9825fef0f: moving from state preparing -> state finished >Thread-15911424::DEBUG::2013-11-19 21:34:52,606::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911424::DEBUG::2013-11-19 21:34:52,609::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911424::DEBUG::2013-11-19 21:34:52,616::task::495::TaskManager.Task::(_debug) Task 987662e6-d2ad-46d5-89cc-96c9825fef0f: ref 0 aborting False >Thread-15911424::INFO::2013-11-19 21:34:52,622::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911426::DEBUG::2013-11-19 21:34:53,639::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911426::INFO::2013-11-19 21:34:53,646::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911426::DEBUG::2013-11-19 21:34:53,648::task::495::TaskManager.Task::(_debug) Task 64439d5a-03fd-420a-a9aa-d6d829380eb8: moving from state init -> state preparing >Thread-15911426::DEBUG::2013-11-19 21:34:53,654::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911426::DEBUG::2013-11-19 21:34:53,655::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911426::DEBUG::2013-11-19 21:34:53,661::task::495::TaskManager.Task::(_debug) Task 64439d5a-03fd-420a-a9aa-d6d829380eb8: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911426::DEBUG::2013-11-19 21:34:53,661::task::495::TaskManager.Task::(_debug) Task 64439d5a-03fd-420a-a9aa-d6d829380eb8: moving from state preparing -> state finished >Thread-15911426::DEBUG::2013-11-19 21:34:53,666::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911426::DEBUG::2013-11-19 21:34:53,667::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911426::DEBUG::2013-11-19 21:34:53,674::task::495::TaskManager.Task::(_debug) Task 64439d5a-03fd-420a-a9aa-d6d829380eb8: ref 0 aborting False >Thread-15911426::INFO::2013-11-19 21:34:53,678::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911428::DEBUG::2013-11-19 21:34:54,697::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911428::INFO::2013-11-19 21:34:54,701::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911428::DEBUG::2013-11-19 21:34:54,704::task::495::TaskManager.Task::(_debug) Task 4e20e2d5-a1b4-4e91-91b1-c3380541d604: moving from state init -> state preparing >Thread-15911428::DEBUG::2013-11-19 21:34:54,708::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911428::DEBUG::2013-11-19 21:34:54,709::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911428::DEBUG::2013-11-19 21:34:54,715::task::495::TaskManager.Task::(_debug) Task 4e20e2d5-a1b4-4e91-91b1-c3380541d604: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911428::DEBUG::2013-11-19 21:34:54,720::task::495::TaskManager.Task::(_debug) Task 4e20e2d5-a1b4-4e91-91b1-c3380541d604: moving from state preparing -> state finished >Thread-15911428::DEBUG::2013-11-19 21:34:54,725::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911428::DEBUG::2013-11-19 21:34:54,731::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911428::DEBUG::2013-11-19 21:34:54,731::task::495::TaskManager.Task::(_debug) Task 4e20e2d5-a1b4-4e91-91b1-c3380541d604: ref 0 aborting False >Thread-15911428::INFO::2013-11-19 21:34:54,737::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911429::DEBUG::2013-11-19 21:34:55,753::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911429::INFO::2013-11-19 21:34:55,754::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911429::DEBUG::2013-11-19 21:34:55,758::task::495::TaskManager.Task::(_debug) Task de5fd53d-2c16-4223-b696-d1cba209d2ca: moving from state init -> state preparing >Thread-15911429::DEBUG::2013-11-19 21:34:55,759::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911429::DEBUG::2013-11-19 21:34:55,762::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911429::DEBUG::2013-11-19 21:34:55,764::task::495::TaskManager.Task::(_debug) Task de5fd53d-2c16-4223-b696-d1cba209d2ca: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911429::DEBUG::2013-11-19 21:34:55,768::task::495::TaskManager.Task::(_debug) Task de5fd53d-2c16-4223-b696-d1cba209d2ca: moving from state preparing -> state finished >Thread-15911429::DEBUG::2013-11-19 21:34:55,768::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911429::DEBUG::2013-11-19 21:34:55,774::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911429::DEBUG::2013-11-19 21:34:55,781::task::495::TaskManager.Task::(_debug) Task de5fd53d-2c16-4223-b696-d1cba209d2ca: ref 0 aborting False >Thread-15911429::INFO::2013-11-19 21:34:55,781::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911431::DEBUG::2013-11-19 21:34:56,795::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911431::INFO::2013-11-19 21:34:56,796::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911431::DEBUG::2013-11-19 21:34:56,799::task::495::TaskManager.Task::(_debug) Task 049ed4f2-198e-42a4-a3d5-d705ce864fe8: moving from state init -> state preparing >Thread-15911431::DEBUG::2013-11-19 21:34:56,800::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911431::DEBUG::2013-11-19 21:34:56,800::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911431::DEBUG::2013-11-19 21:34:56,803::task::495::TaskManager.Task::(_debug) Task 049ed4f2-198e-42a4-a3d5-d705ce864fe8: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911431::DEBUG::2013-11-19 21:34:56,806::task::495::TaskManager.Task::(_debug) Task 049ed4f2-198e-42a4-a3d5-d705ce864fe8: moving from state preparing -> state finished >Thread-15911431::DEBUG::2013-11-19 21:34:56,811::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911431::DEBUG::2013-11-19 21:34:56,812::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911431::DEBUG::2013-11-19 21:34:56,815::task::495::TaskManager.Task::(_debug) Task 049ed4f2-198e-42a4-a3d5-d705ce864fe8: ref 0 aborting False >Thread-15911431::INFO::2013-11-19 21:34:56,817::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911432::DEBUG::2013-11-19 21:34:57,837::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911432::INFO::2013-11-19 21:34:57,843::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911432::DEBUG::2013-11-19 21:34:57,852::task::495::TaskManager.Task::(_debug) Task c7985d5e-c173-489a-9176-6bf5b58bc0c6: moving from state init -> state preparing >Thread-15911432::DEBUG::2013-11-19 21:34:57,855::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911432::DEBUG::2013-11-19 21:34:57,861::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911432::DEBUG::2013-11-19 21:34:57,867::task::495::TaskManager.Task::(_debug) Task c7985d5e-c173-489a-9176-6bf5b58bc0c6: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911432::DEBUG::2013-11-19 21:34:57,868::task::495::TaskManager.Task::(_debug) Task c7985d5e-c173-489a-9176-6bf5b58bc0c6: moving from state preparing -> state finished >Thread-15911432::DEBUG::2013-11-19 21:34:57,871::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911432::DEBUG::2013-11-19 21:34:57,875::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911432::DEBUG::2013-11-19 21:34:57,884::task::495::TaskManager.Task::(_debug) Task c7985d5e-c173-489a-9176-6bf5b58bc0c6: ref 0 aborting False >Thread-15911432::INFO::2013-11-19 21:34:57,887::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-1024767::ERROR::2013-11-19 21:34:58,255::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911433::INFO::2013-11-19 21:34:58,633::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911433::DEBUG::2013-11-19 21:34:58,641::task::495::TaskManager.Task::(_debug) Task 914ff2a9-4653-485a-bdb3-852b5d15688e: moving from state init -> state preparing >Thread-15911433::DEBUG::2013-11-19 21:34:58,642::task::495::TaskManager.Task::(_debug) Task 914ff2a9-4653-485a-bdb3-852b5d15688e: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00032901763916', 'lastCheck': 1384896898.346746, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00251817703247', 'lastCheck': 1384896892.2490489, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896898.256459, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000525951385498', 'lastCheck': 1384896889.2436521, 'valid': True, 'code': 0}} >Thread-15911433::DEBUG::2013-11-19 21:34:58,651::task::495::TaskManager.Task::(_debug) Task 914ff2a9-4653-485a-bdb3-852b5d15688e: moving from state preparing -> state finished >Thread-15911433::DEBUG::2013-11-19 21:34:58,661::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911433::DEBUG::2013-11-19 21:34:58,670::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911433::DEBUG::2013-11-19 21:34:58,676::task::495::TaskManager.Task::(_debug) Task 914ff2a9-4653-485a-bdb3-852b5d15688e: ref 0 aborting False >Thread-15911433::INFO::2013-11-19 21:34:58,682::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00032901763916', 'lastCheck': 1384896898.346746, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00251817703247', 'lastCheck': 1384896892.2490489, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896898.256459, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000525951385498', 'lastCheck': 1384896889.2436521, 'valid': True, 'code': 0}} >Thread-15911435::DEBUG::2013-11-19 21:34:58,908::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911435::INFO::2013-11-19 21:34:58,909::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911435::DEBUG::2013-11-19 21:34:58,915::task::495::TaskManager.Task::(_debug) Task d86eddd2-4092-45d9-8a96-c8fd6371b116: moving from state init -> state preparing >Thread-15911435::DEBUG::2013-11-19 21:34:58,919::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911435::DEBUG::2013-11-19 21:34:58,925::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911435::DEBUG::2013-11-19 21:34:58,930::task::495::TaskManager.Task::(_debug) Task d86eddd2-4092-45d9-8a96-c8fd6371b116: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911435::DEBUG::2013-11-19 21:34:58,930::task::495::TaskManager.Task::(_debug) Task d86eddd2-4092-45d9-8a96-c8fd6371b116: moving from state preparing -> state finished >Thread-15911435::DEBUG::2013-11-19 21:34:58,931::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911435::DEBUG::2013-11-19 21:34:58,941::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911435::DEBUG::2013-11-19 21:34:58,941::task::495::TaskManager.Task::(_debug) Task d86eddd2-4092-45d9-8a96-c8fd6371b116: ref 0 aborting False >Thread-15911435::INFO::2013-11-19 21:34:58,945::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911436::DEBUG::2013-11-19 21:34:59,963::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911436::INFO::2013-11-19 21:34:59,967::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911436::DEBUG::2013-11-19 21:34:59,968::task::495::TaskManager.Task::(_debug) Task 42efc547-9c33-4b9a-854f-682cdf7a486c: moving from state init -> state preparing >Thread-15911436::DEBUG::2013-11-19 21:34:59,971::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911436::DEBUG::2013-11-19 21:34:59,972::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911436::DEBUG::2013-11-19 21:34:59,972::task::495::TaskManager.Task::(_debug) Task 42efc547-9c33-4b9a-854f-682cdf7a486c: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911436::DEBUG::2013-11-19 21:34:59,974::task::495::TaskManager.Task::(_debug) Task 42efc547-9c33-4b9a-854f-682cdf7a486c: moving from state preparing -> state finished >Thread-15911436::DEBUG::2013-11-19 21:34:59,975::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911436::DEBUG::2013-11-19 21:34:59,978::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911436::DEBUG::2013-11-19 21:34:59,978::task::495::TaskManager.Task::(_debug) Task 42efc547-9c33-4b9a-854f-682cdf7a486c: ref 0 aborting False >Thread-15911436::INFO::2013-11-19 21:34:59,979::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911438::DEBUG::2013-11-19 21:35:00,998::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911438::INFO::2013-11-19 21:35:01,003::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911438::DEBUG::2013-11-19 21:35:01,010::task::495::TaskManager.Task::(_debug) Task 02ff22b0-6da5-4a96-bf7c-fdc1fc4faec1: moving from state init -> state preparing >Thread-15911438::DEBUG::2013-11-19 21:35:01,013::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911438::DEBUG::2013-11-19 21:35:01,018::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911438::DEBUG::2013-11-19 21:35:01,021::task::495::TaskManager.Task::(_debug) Task 02ff22b0-6da5-4a96-bf7c-fdc1fc4faec1: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911438::DEBUG::2013-11-19 21:35:01,026::task::495::TaskManager.Task::(_debug) Task 02ff22b0-6da5-4a96-bf7c-fdc1fc4faec1: moving from state preparing -> state finished >Thread-15911438::DEBUG::2013-11-19 21:35:01,029::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911438::DEBUG::2013-11-19 21:35:01,036::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911438::DEBUG::2013-11-19 21:35:01,041::task::495::TaskManager.Task::(_debug) Task 02ff22b0-6da5-4a96-bf7c-fdc1fc4faec1: ref 0 aborting False >Thread-15911438::INFO::2013-11-19 21:35:01,044::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911439::DEBUG::2013-11-19 21:35:02,069::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911439::INFO::2013-11-19 21:35:02,076::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911439::DEBUG::2013-11-19 21:35:02,081::task::495::TaskManager.Task::(_debug) Task 3abc0e92-9957-4ebc-b4e6-3275f7f32f6d: moving from state init -> state preparing >Thread-15911439::DEBUG::2013-11-19 21:35:02,085::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911439::DEBUG::2013-11-19 21:35:02,090::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911439::DEBUG::2013-11-19 21:35:02,092::task::495::TaskManager.Task::(_debug) Task 3abc0e92-9957-4ebc-b4e6-3275f7f32f6d: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911439::DEBUG::2013-11-19 21:35:02,093::task::495::TaskManager.Task::(_debug) Task 3abc0e92-9957-4ebc-b4e6-3275f7f32f6d: moving from state preparing -> state finished >Thread-15911439::DEBUG::2013-11-19 21:35:02,098::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911439::DEBUG::2013-11-19 21:35:02,103::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911439::DEBUG::2013-11-19 21:35:02,109::task::495::TaskManager.Task::(_debug) Task 3abc0e92-9957-4ebc-b4e6-3275f7f32f6d: ref 0 aborting False >Thread-15911439::INFO::2013-11-19 21:35:02,111::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911441::DEBUG::2013-11-19 21:35:03,127::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911441::INFO::2013-11-19 21:35:03,131::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911441::DEBUG::2013-11-19 21:35:03,134::task::495::TaskManager.Task::(_debug) Task d9a39074-3189-4555-9a89-d0b8bf0e6a9e: moving from state init -> state preparing >Thread-15911441::DEBUG::2013-11-19 21:35:03,139::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911441::DEBUG::2013-11-19 21:35:03,140::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911441::DEBUG::2013-11-19 21:35:03,144::task::495::TaskManager.Task::(_debug) Task d9a39074-3189-4555-9a89-d0b8bf0e6a9e: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911441::DEBUG::2013-11-19 21:35:03,149::task::495::TaskManager.Task::(_debug) Task d9a39074-3189-4555-9a89-d0b8bf0e6a9e: moving from state preparing -> state finished >Thread-15911441::DEBUG::2013-11-19 21:35:03,149::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911441::DEBUG::2013-11-19 21:35:03,158::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911441::DEBUG::2013-11-19 21:35:03,159::task::495::TaskManager.Task::(_debug) Task d9a39074-3189-4555-9a89-d0b8bf0e6a9e: ref 0 aborting False >Thread-15911441::INFO::2013-11-19 21:35:03,170::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911442::DEBUG::2013-11-19 21:35:04,184::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911442::INFO::2013-11-19 21:35:04,185::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911442::DEBUG::2013-11-19 21:35:04,193::task::495::TaskManager.Task::(_debug) Task 4379b735-bf62-4a24-9707-0e83bff324fc: moving from state init -> state preparing >Thread-15911442::DEBUG::2013-11-19 21:35:04,198::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911442::DEBUG::2013-11-19 21:35:04,203::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911442::DEBUG::2013-11-19 21:35:04,207::task::495::TaskManager.Task::(_debug) Task 4379b735-bf62-4a24-9707-0e83bff324fc: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911442::DEBUG::2013-11-19 21:35:04,214::task::495::TaskManager.Task::(_debug) Task 4379b735-bf62-4a24-9707-0e83bff324fc: moving from state preparing -> state finished >Thread-15911442::DEBUG::2013-11-19 21:35:04,215::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911442::DEBUG::2013-11-19 21:35:04,218::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911442::DEBUG::2013-11-19 21:35:04,218::task::495::TaskManager.Task::(_debug) Task 4379b735-bf62-4a24-9707-0e83bff324fc: ref 0 aborting False >Thread-15911442::INFO::2013-11-19 21:35:04,221::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911444::DEBUG::2013-11-19 21:35:05,231::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911444::INFO::2013-11-19 21:35:05,233::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911444::DEBUG::2013-11-19 21:35:05,234::task::495::TaskManager.Task::(_debug) Task 1d28f981-ab63-4aef-b873-d498cafd6c66: moving from state init -> state preparing >Thread-15911444::DEBUG::2013-11-19 21:35:05,234::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911444::DEBUG::2013-11-19 21:35:05,235::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911444::DEBUG::2013-11-19 21:35:05,239::task::495::TaskManager.Task::(_debug) Task 1d28f981-ab63-4aef-b873-d498cafd6c66: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911444::DEBUG::2013-11-19 21:35:05,248::task::495::TaskManager.Task::(_debug) Task 1d28f981-ab63-4aef-b873-d498cafd6c66: moving from state preparing -> state finished >Thread-15911444::DEBUG::2013-11-19 21:35:05,249::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911444::DEBUG::2013-11-19 21:35:05,255::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911444::DEBUG::2013-11-19 21:35:05,257::task::495::TaskManager.Task::(_debug) Task 1d28f981-ab63-4aef-b873-d498cafd6c66: ref 0 aborting False >Thread-15911444::INFO::2013-11-19 21:35:05,258::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911445::DEBUG::2013-11-19 21:35:06,277::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911445::INFO::2013-11-19 21:35:06,285::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911445::DEBUG::2013-11-19 21:35:06,290::task::495::TaskManager.Task::(_debug) Task 47f3b5eb-1f8c-4a6f-a715-6cf0fd684bde: moving from state init -> state preparing >Thread-15911445::DEBUG::2013-11-19 21:35:06,291::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911445::DEBUG::2013-11-19 21:35:06,291::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911445::DEBUG::2013-11-19 21:35:06,292::task::495::TaskManager.Task::(_debug) Task 47f3b5eb-1f8c-4a6f-a715-6cf0fd684bde: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911445::DEBUG::2013-11-19 21:35:06,292::task::495::TaskManager.Task::(_debug) Task 47f3b5eb-1f8c-4a6f-a715-6cf0fd684bde: moving from state preparing -> state finished >Thread-15911445::DEBUG::2013-11-19 21:35:06,293::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911445::DEBUG::2013-11-19 21:35:06,299::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911445::DEBUG::2013-11-19 21:35:06,305::task::495::TaskManager.Task::(_debug) Task 47f3b5eb-1f8c-4a6f-a715-6cf0fd684bde: ref 0 aborting False >Thread-15911445::INFO::2013-11-19 21:35:06,311::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911447::DEBUG::2013-11-19 21:35:07,327::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911447::INFO::2013-11-19 21:35:07,337::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911447::DEBUG::2013-11-19 21:35:07,341::task::495::TaskManager.Task::(_debug) Task 1afaac2d-d76b-4122-bf5a-1f787e836fe4: moving from state init -> state preparing >Thread-15911447::DEBUG::2013-11-19 21:35:07,349::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911447::DEBUG::2013-11-19 21:35:07,352::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911447::DEBUG::2013-11-19 21:35:07,358::task::495::TaskManager.Task::(_debug) Task 1afaac2d-d76b-4122-bf5a-1f787e836fe4: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911447::DEBUG::2013-11-19 21:35:07,359::task::495::TaskManager.Task::(_debug) Task 1afaac2d-d76b-4122-bf5a-1f787e836fe4: moving from state preparing -> state finished >Thread-15911447::DEBUG::2013-11-19 21:35:07,364::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911447::DEBUG::2013-11-19 21:35:07,368::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911447::DEBUG::2013-11-19 21:35:07,373::task::495::TaskManager.Task::(_debug) Task 1afaac2d-d76b-4122-bf5a-1f787e836fe4: ref 0 aborting False >Thread-15911447::INFO::2013-11-19 21:35:07,379::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:07,908::safelease::61::Storage.Misc.excCmd::(acquire) SUCCESS: <err> = ''; <rc> = 0 >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:07,908::safelease::65::ClusterLock::(acquire) Clustered lock acquired successfully >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:07,909::spm::468::Storage.SPM::(start) spm lock acquired successfully >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:07,910::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:07,910::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:07,911::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:07,912::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:07,912::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:07,913::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-1024767::ERROR::2013-11-19 21:35:08,279::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 21:35:08,358::lvm::439::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-42838::DEBUG::2013-11-19 21:35:08,359::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15911448::DEBUG::2013-11-19 21:35:08,562::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911448::INFO::2013-11-19 21:35:08,568::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911448::DEBUG::2013-11-19 21:35:08,571::task::495::TaskManager.Task::(_debug) Task d14c31b7-efaf-474a-8bd4-35148ea36dbb: moving from state init -> state preparing >Thread-15911448::DEBUG::2013-11-19 21:35:08,575::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911448::DEBUG::2013-11-19 21:35:08,579::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911448::DEBUG::2013-11-19 21:35:08,585::task::495::TaskManager.Task::(_debug) Task d14c31b7-efaf-474a-8bd4-35148ea36dbb: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:08,586::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911448::DEBUG::2013-11-19 21:35:08,588::task::495::TaskManager.Task::(_debug) Task d14c31b7-efaf-474a-8bd4-35148ea36dbb: moving from state preparing -> state finished >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:08,592::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=284', u'_SHA_CKSUM=fa1ddc0ebaad4a3d815671fdef2d27b7d9f8a82e', u'POOL_SPM_ID=-1'] >Thread-15911448::DEBUG::2013-11-19 21:35:08,596::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:08,597::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >Thread-15911448::DEBUG::2013-11-19 21:35:08,599::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:08,600::persistentDict::146::Storage.PersistentDict::(transaction) Flushing changes >Thread-15911448::DEBUG::2013-11-19 21:35:08,602::task::495::TaskManager.Task::(_debug) Task d14c31b7-efaf-474a-8bd4-35148ea36dbb: ref 0 aborting False >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:08,603::persistentDict::265::Storage.PersistentDict::(flush) about to write lines (VGTagMetadataRW)=[u'CLASS=Data', u'DESCRIPTION=OS', u'IOOPTIMEOUTSEC=10', u'LEASERETRIES=3', u'LEASETIMESEC=60', u'LOCKPOLICY=', u'LOCKRENEWALINTERVALSEC=5', u'LOGBLKSIZE=512', u'MASTER_VERSION=1', u'PHYBLKSIZE=512', u'POOL_DESCRIPTION=Default', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_ID=2', u'POOL_SPM_LVER=285', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'ROLE=Master', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'TYPE=FCP', u'VERSION=2', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', '_SHA_CKSUM=901e51f2c0f3aefcf8d2e501fb4543e2352ea1bc'] >Thread-15911448::INFO::2013-11-19 21:35:08,606::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:08,607::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting... >Thread-15911449::INFO::2013-11-19 21:35:08,872::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911449::DEBUG::2013-11-19 21:35:08,880::task::495::TaskManager.Task::(_debug) Task c2ba812e-d618-429b-90fb-150f9af4b999: moving from state init -> state preparing >Thread-15911449::DEBUG::2013-11-19 21:35:08,887::task::495::TaskManager.Task::(_debug) Task c2ba812e-d618-429b-90fb-150f9af4b999: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00032901763916', 'lastCheck': 1384896898.346746, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00134897232056', 'lastCheck': 1384896902.2645841, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896908.2812121, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00050687789917', 'lastCheck': 1384896899.254674, 'valid': True, 'code': 0}} >Thread-15911449::DEBUG::2013-11-19 21:35:08,888::task::495::TaskManager.Task::(_debug) Task c2ba812e-d618-429b-90fb-150f9af4b999: moving from state preparing -> state finished >Thread-15911449::DEBUG::2013-11-19 21:35:08,898::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911449::DEBUG::2013-11-19 21:35:08,903::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911449::DEBUG::2013-11-19 21:35:08,912::task::495::TaskManager.Task::(_debug) Task c2ba812e-d618-429b-90fb-150f9af4b999: ref 0 aborting False >Thread-15911449::INFO::2013-11-19 21:35:08,917::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00032901763916', 'lastCheck': 1384896898.346746, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00134897232056', 'lastCheck': 1384896902.2645841, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896908.2812121, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00050687789917', 'lastCheck': 1384896899.254674, 'valid': True, 'code': 0}} >Thread-42838::DEBUG::2013-11-19 21:35:09,045::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-42838::DEBUG::2013-11-19 21:35:09,048::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:09,049::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:09,050::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:09,050::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:09,051::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:09,052::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:09,053::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15911451::DEBUG::2013-11-19 21:35:09,620::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911451::INFO::2013-11-19 21:35:09,624::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911451::DEBUG::2013-11-19 21:35:09,625::task::495::TaskManager.Task::(_debug) Task eea33e2c-d6c2-4dd1-8227-64975bdae578: moving from state init -> state preparing >Thread-15911451::DEBUG::2013-11-19 21:35:09,626::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911451::DEBUG::2013-11-19 21:35:09,626::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911451::DEBUG::2013-11-19 21:35:09,636::task::495::TaskManager.Task::(_debug) Task eea33e2c-d6c2-4dd1-8227-64975bdae578: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911451::DEBUG::2013-11-19 21:35:09,643::task::495::TaskManager.Task::(_debug) Task eea33e2c-d6c2-4dd1-8227-64975bdae578: moving from state preparing -> state finished >Thread-15911451::DEBUG::2013-11-19 21:35:09,643::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911451::DEBUG::2013-11-19 21:35:09,661::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911451::DEBUG::2013-11-19 21:35:09,674::task::495::TaskManager.Task::(_debug) Task eea33e2c-d6c2-4dd1-8227-64975bdae578: ref 0 aborting False >Thread-15911451::INFO::2013-11-19 21:35:09,683::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:09,744::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:09,748::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:09,749::blockSD::145::storage.Metadata.VGTagMetadataRW::(writelines) Updating metadata adding=MDT__SHA_CKSUM=901e51f2c0f3aefcf8d2e501fb4543e2352ea1bc, MDT_POOL_SPM_LVER=285, MDT_POOL_SPM_ID=2 removing=MDT_POOL_SPM_ID=-1, MDT_POOL_SPM_LVER=284, MDT__SHA_CKSUM=fa1ddc0ebaad4a3d815671fdef2d27b7d9f8a82e >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:09,750::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --deltag MDT__SHA_CKSUM=fa1ddc0ebaad4a3d815671fdef2d27b7d9f8a82e --deltag MDT_POOL_SPM_ID=-1 --deltag MDT_POOL_SPM_LVER=284 --addtag MDT_POOL_SPM_ID=2 --addtag MDT__SHA_CKSUM=901e51f2c0f3aefcf8d2e501fb4543e2352ea1bc --addtag MDT_POOL_SPM_LVER=285 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:10,465::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:10,466::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:10,467::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:10,468::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:10,470::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`201777e5-cf9c-49e9-b517-96c0f26fafb4`::Request was made in '/usr/share/vdsm/storage/spm.py' line '330' at '_upgradePool' >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:10,472::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:10,473::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:10,474::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`201777e5-cf9c-49e9-b517-96c0f26fafb4`::Granted request >44a1be5a-8538-4745-8f2c-b3e70865e4e7::INFO::2013-11-19 21:35:10,475::spm::336::Storage.SPM::(_upgradePool) Trying to upgrade master domain `6b892067-ac7e-4039-a7be-5b7387525b16` >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:10,476::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`3dc875ce-7dc3-43e9-b81c-7fc08564bce9`::Request was made in '/usr/share/vdsm/storage/spm.py' line '337' at '_upgradePool' >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:10,476::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'exclusive' >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:10,477::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'exclusive' (1 active user) >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:10,477::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`3dc875ce-7dc3-43e9-b81c-7fc08564bce9`::Granted request >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:10,479::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:10,479::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:10,480::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:10,480::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:10,481::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:10,482::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15911452::DEBUG::2013-11-19 21:35:10,744::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911452::INFO::2013-11-19 21:35:10,748::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911452::DEBUG::2013-11-19 21:35:10,752::task::495::TaskManager.Task::(_debug) Task de0467ae-0023-4b90-860f-9ef088f0a445: moving from state init -> state preparing >Thread-15911452::DEBUG::2013-11-19 21:35:10,759::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911452::DEBUG::2013-11-19 21:35:10,764::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911452::DEBUG::2013-11-19 21:35:10,772::task::495::TaskManager.Task::(_debug) Task de0467ae-0023-4b90-860f-9ef088f0a445: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911452::DEBUG::2013-11-19 21:35:10,773::task::495::TaskManager.Task::(_debug) Task de0467ae-0023-4b90-860f-9ef088f0a445: moving from state preparing -> state finished >Thread-15911452::DEBUG::2013-11-19 21:35:10,779::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911452::DEBUG::2013-11-19 21:35:10,783::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911452::DEBUG::2013-11-19 21:35:10,790::task::495::TaskManager.Task::(_debug) Task de0467ae-0023-4b90-860f-9ef088f0a445: ref 0 aborting False >Thread-15911452::INFO::2013-11-19 21:35:10,795::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:11,087::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:11,090::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:11,091::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_ID=2', u'_SHA_CKSUM=901e51f2c0f3aefcf8d2e501fb4543e2352ea1bc', u'POOL_SPM_LVER=285'] >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:11,092::blockSD::625::Storage.StorageDomain::(upgrade) Trying to upgrade domain `6b892067-ac7e-4039-a7be-5b7387525b16` from version 2 to version 2 >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:11,093::blockSD::631::Storage.StorageDomain::(upgrade) No need to upgrade domain `6b892067-ac7e-4039-a7be-5b7387525b16`, leaving unchanged >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:11,094::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:11,094::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:11,095::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:11,095::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:11,096::spm::340::Storage.SPM::(_upgradePool) Marking all domains for upgrade >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:11,097::spm::347::Storage.SPM::(_upgradePool) Registering with state change event >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:11,097::spm::349::Storage.SPM::(_upgradePool) Running initial domain upgrade threads >Thread-15911453::DEBUG::2013-11-19 21:35:11,098::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 45cdff8b-89c0-4926-a8c7-80354064bfa6 >Thread-15911454::DEBUG::2013-11-19 21:35:11,102::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 772472cb-2429-466d-96cd-00b3301ef78a >Thread-15911455::DEBUG::2013-11-19 21:35:11,104::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 5d7b5178-3790-472c-a13b-b481eb27a61e >Thread-15911453::DEBUG::2013-11-19 21:35:11,105::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`4b3764fe-2219-4c03-b8db-6ecd3c6d5eb3`::Request was made in '/usr/share/vdsm/storage/spm.py' line '376' at '_upgradePoolDomain' >Thread-15911454::DEBUG::2013-11-19 21:35:11,106::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`96bae22c-bbc3-4c3e-a656-96b2523b87c8`::Request was made in '/usr/share/vdsm/storage/spm.py' line '376' at '_upgradePoolDomain' >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:11,107::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911455::ERROR::2013-11-19 21:35:11,114::spm::373::Storage.SPM::(_upgradePoolDomain) Error while preparing domain `5d7b5178-3790-472c-a13b-b481eb27a61e` upgrade >Traceback (most recent call last): > File "/usr/share/vdsm/storage/spm.py", line 369, in _upgradePoolDomain >IndexError: list index out of range >Thread-15911453::DEBUG::2013-11-19 21:35:11,117::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' for lock type 'exclusive' >Thread-15911454::DEBUG::2013-11-19 21:35:11,118::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'exclusive' >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:11,119::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:11,134::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:11,135::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911453::DEBUG::2013-11-19 21:35:11,136::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' is free. Now locking as 'exclusive' (1 active user) >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:11,137::blockSD::791::Storage.Misc.excCmd::(mountMaster) '/usr/bin/sudo -n /sbin/e2fsck -p /dev/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15911453::DEBUG::2013-11-19 21:35:11,142::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`4b3764fe-2219-4c03-b8db-6ecd3c6d5eb3`::Granted request >Thread-15911454::DEBUG::2013-11-19 21:35:11,376::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'exclusive' (1 active user) >Thread-15911454::DEBUG::2013-11-19 21:35:11,378::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`96bae22c-bbc3-4c3e-a656-96b2523b87c8`::Granted request >Thread-15911453::DEBUG::2013-11-19 21:35:11,381::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`d5a88e0f-dd3d-4ceb-be98-d5389402a101`::Request was made in '/usr/share/vdsm/storage/spm.py' line '377' at '_upgradePoolDomain' >Thread-15911453::DEBUG::2013-11-19 21:35:11,387::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' for lock type 'exclusive' >Thread-15911454::DEBUG::2013-11-19 21:35:11,388::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`1c5c7d84-8041-4da3-9920-6644cbc6aac4`::Request was made in '/usr/share/vdsm/storage/spm.py' line '377' at '_upgradePoolDomain' >Thread-15911453::DEBUG::2013-11-19 21:35:11,401::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' is free. Now locking as 'exclusive' (1 active user) >Thread-15911454::DEBUG::2013-11-19 21:35:11,401::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'exclusive' >Thread-15911453::DEBUG::2013-11-19 21:35:11,404::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`d5a88e0f-dd3d-4ceb-be98-d5389402a101`::Granted request >Thread-15911454::DEBUG::2013-11-19 21:35:11,407::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'exclusive' (1 active user) >Thread-15911453::DEBUG::2013-11-19 21:35:11,407::spm::385::Storage.SPM::(_upgradePoolDomain) Domain `45cdff8b-89c0-4926-a8c7-80354064bfa6` is not a data domain it is an 2 domain, not upgrading >Thread-15911454::DEBUG::2013-11-19 21:35:11,408::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`1c5c7d84-8041-4da3-9920-6644cbc6aac4`::Granted request >Thread-15911453::DEBUG::2013-11-19 21:35:11,412::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' >Thread-15911454::DEBUG::2013-11-19 21:35:11,418::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15911453::DEBUG::2013-11-19 21:35:11,423::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' (0 active users) >Thread-15911454::DEBUG::2013-11-19 21:35:11,424::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15911453::DEBUG::2013-11-19 21:35:11,428::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' is free, finding out if anyone is waiting for it. >Thread-15911454::DEBUG::2013-11-19 21:35:11,429::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15911453::DEBUG::2013-11-19 21:35:11,434::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6', Clearing records. >Thread-15911454::DEBUG::2013-11-19 21:35:11,435::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15911454::DEBUG::2013-11-19 21:35:11,439::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15911453::DEBUG::2013-11-19 21:35:11,439::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' >Thread-15911454::DEBUG::2013-11-19 21:35:11,440::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15911453::DEBUG::2013-11-19 21:35:11,444::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' (0 active users) >Thread-15911453::DEBUG::2013-11-19 21:35:11,705::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' is free, finding out if anyone is waiting for it. >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:11,706::blockSD::791::Storage.Misc.excCmd::(mountMaster) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911453::DEBUG::2013-11-19 21:35:11,709::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6', Clearing records. >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:11,710::blockSD::816::Storage.Misc.excCmd::(mountMaster) '/usr/bin/sudo -n /sbin/tune2fs -j /dev/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15911457::DEBUG::2013-11-19 21:35:12,082::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911457::INFO::2013-11-19 21:35:12,096::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911457::DEBUG::2013-11-19 21:35:12,097::task::495::TaskManager.Task::(_debug) Task 35d88432-def4-4052-90fb-6ca028e2f45c: moving from state init -> state preparing >Thread-15911457::DEBUG::2013-11-19 21:35:12,101::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911457::DEBUG::2013-11-19 21:35:12,103::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911457::DEBUG::2013-11-19 21:35:12,104::task::495::TaskManager.Task::(_debug) Task 35d88432-def4-4052-90fb-6ca028e2f45c: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911457::DEBUG::2013-11-19 21:35:12,108::task::495::TaskManager.Task::(_debug) Task 35d88432-def4-4052-90fb-6ca028e2f45c: moving from state preparing -> state finished >Thread-15911457::DEBUG::2013-11-19 21:35:12,109::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911457::DEBUG::2013-11-19 21:35:12,109::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911457::DEBUG::2013-11-19 21:35:12,110::task::495::TaskManager.Task::(_debug) Task 35d88432-def4-4052-90fb-6ca028e2f45c: ref 0 aborting False >Thread-15911457::INFO::2013-11-19 21:35:12,118::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:12,227::blockSD::816::Storage.Misc.excCmd::(mountMaster) FAILED: <err> = 'The filesystem already has a journal.\n'; <rc> = 1 >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:12,230::fileUtils::83::Storage.Misc.excCmd::(mount) '/usr/bin/sudo -n /bin/mount /dev/6b892067-ac7e-4039-a7be-5b7387525b16/master /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15911454::DEBUG::2013-11-19 21:35:12,435::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911454::DEBUG::2013-11-19 21:35:12,442::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15911454::DEBUG::2013-11-19 21:35:12,444::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'SDUUID=772472cb-2429-466d-96cd-00b3301ef78a', u'VERSION=2', u'LOCKPOLICY=', u'VGUUID=XTb294-NYv6-dVpO-bB1n-4f9P-d0GL-dhd2DI', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'LEASETIMESEC=5', u'LOCKRENEWALINTERVALSEC=5', u'ROLE=Regular', u'PHYBLKSIZE=512', u'DESCRIPTION=Database', u'IOOPTIMEOUTSEC=1', u'CLASS=Data', u'TYPE=FCP', u'PV0=pv:3600c0ff0001520c7414f605001000000,uuid:NujM33-vNiD-sBHY-mrPo-EQ2f-1qsz-HXfn0q,pestart:0,pecount:8929,mapoffset:0', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'MASTER_VERSION=0', u'_SHA_CKSUM=ef8840d9fdef24516a56fd970bed1960d2b1ae92'] >Thread-15911454::DEBUG::2013-11-19 21:35:12,446::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15911454::DEBUG::2013-11-19 21:35:12,447::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15911454::DEBUG::2013-11-19 21:35:12,448::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15911454::DEBUG::2013-11-19 21:35:12,448::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15911454::DEBUG::2013-11-19 21:35:12,449::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15911454::DEBUG::2013-11-19 21:35:12,450::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15911454::DEBUG::2013-11-19 21:35:12,451::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15911454::DEBUG::2013-11-19 21:35:12,451::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:12,641::fileUtils::83::Storage.Misc.excCmd::(mount) SUCCESS: <err> = ''; <rc> = 0 >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:12,642::blockSD::833::Storage.Misc.excCmd::(mountMaster) '/usr/bin/sudo -n /bin/chown vdsm:kvm /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,042::blockSD::833::Storage.Misc.excCmd::(mountMaster) SUCCESS: <err> = ''; <rc> = 0 >44a1be5a-8538-4745-8f2c-b3e70865e4e7::WARNING::2013-11-19 21:35:13,045::spm::490::Storage.SPM::(start) Backup domain validation failed >Traceback (most recent call last): > File "/usr/share/vdsm/storage/spm.py", line 488, in start > File "/usr/share/vdsm/storage/sp.py", line 1208, in checkBackupDomain > File "/usr/share/vdsm/storage/sd.py", line 666, in isBackup > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,046::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,047::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,047::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,048::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,048::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,049::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,050::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,050::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,051::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,051::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,052::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,052::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,053::threadPool::25::Misc.ThreadPool::(__init__) Enter - numThreads: 5.0, waitTimeout: 3, maxTasks: 500.0 >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,067::storage_mailbox::486::Storage.MailBox.SpmMailMonitor::(__init__) SPM_MailMonitor - clearing outgoing mail, command is: ['dd', 'of=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/outbox', 'oflag=direct', 'iflag=fullblock', 'conv=notrunc', 'count=1'] >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,068::storage_mailbox::488::Storage.Misc.excCmd::(__init__) 'dd of=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/outbox oflag=direct iflag=fullblock conv=notrunc count=1 bs=1024000' (cwd None) >Thread-15911463::DEBUG::2013-11-19 21:35:13,285::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911463::INFO::2013-11-19 21:35:13,288::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911463::DEBUG::2013-11-19 21:35:13,289::task::495::TaskManager.Task::(_debug) Task 42424bc7-7227-47f7-ba8a-dc8d765a8620: moving from state init -> state preparing >Thread-15911463::DEBUG::2013-11-19 21:35:13,289::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911463::DEBUG::2013-11-19 21:35:13,290::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911463::DEBUG::2013-11-19 21:35:13,290::task::495::TaskManager.Task::(_debug) Task 42424bc7-7227-47f7-ba8a-dc8d765a8620: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911463::DEBUG::2013-11-19 21:35:13,291::task::495::TaskManager.Task::(_debug) Task 42424bc7-7227-47f7-ba8a-dc8d765a8620: moving from state preparing -> state finished >Thread-15911463::DEBUG::2013-11-19 21:35:13,292::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911463::DEBUG::2013-11-19 21:35:13,292::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911463::DEBUG::2013-11-19 21:35:13,293::task::495::TaskManager.Task::(_debug) Task 42424bc7-7227-47f7-ba8a-dc8d765a8620: ref 0 aborting False >Thread-15911463::INFO::2013-11-19 21:35:13,293::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,510::storage_mailbox::488::Storage.Misc.excCmd::(__init__) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.040803 s, 25.1 MB/s\n'; <rc> = 0 >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,511::storage_mailbox::493::Storage.MailBox.SpmMailMonitor::(__init__) SPM_MailMonitor created for pool c4e5889a-348d-11e2-83fe-9f72369eab30 >Dummy-15911465::DEBUG::2013-11-19 21:35:13,511::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,513::spm::523::Storage.SPM::(start) ended. >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,515::task::495::TaskManager.Task::(_debug) Task 44a1be5a-8538-4745-8f2c-b3e70865e4e7: moving from state running -> state finished >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,796::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,809::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,817::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,818::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,818::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,819::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,820::task::915::TaskManager.Task::(_runJobs) Task.run: exit - success: result >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,820::task::495::TaskManager.Task::(_debug) Task 44a1be5a-8538-4745-8f2c-b3e70865e4e7: ref 0 aborting False >44a1be5a-8538-4745-8f2c-b3e70865e4e7::DEBUG::2013-11-19 21:35:13,821::threadPool::47::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 >Dummy-15911465::DEBUG::2013-11-19 21:35:14,071::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0707385 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15911466::DEBUG::2013-11-19 21:35:14,305::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911466::INFO::2013-11-19 21:35:14,306::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911466::DEBUG::2013-11-19 21:35:14,307::task::495::TaskManager.Task::(_debug) Task e3c2bb0e-ec58-4702-969c-0c72482fb8b0: moving from state init -> state preparing >Thread-15911466::DEBUG::2013-11-19 21:35:14,308::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911466::DEBUG::2013-11-19 21:35:14,308::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': '1 jobs completed successfuly', 'taskState': 'finished', 'taskResult': 'success', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'} >Thread-15911466::DEBUG::2013-11-19 21:35:14,309::task::495::TaskManager.Task::(_debug) Task e3c2bb0e-ec58-4702-969c-0c72482fb8b0: finished: {'taskStatus': {'code': 0, 'message': '1 jobs completed successfuly', 'taskState': 'finished', 'taskResult': 'success', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911466::DEBUG::2013-11-19 21:35:14,310::task::495::TaskManager.Task::(_debug) Task e3c2bb0e-ec58-4702-969c-0c72482fb8b0: moving from state preparing -> state finished >Thread-15911466::DEBUG::2013-11-19 21:35:14,310::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911466::DEBUG::2013-11-19 21:35:14,311::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911466::DEBUG::2013-11-19 21:35:14,311::task::495::TaskManager.Task::(_debug) Task e3c2bb0e-ec58-4702-969c-0c72482fb8b0: ref 0 aborting False >Thread-15911466::INFO::2013-11-19 21:35:14,312::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': '1 jobs completed successfuly', 'taskState': 'finished', 'taskResult': 'success', 'taskID': '44a1be5a-8538-4745-8f2c-b3e70865e4e7'}} >Thread-15911467::DEBUG::2013-11-19 21:35:14,319::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911467::INFO::2013-11-19 21:35:14,320::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911467::DEBUG::2013-11-19 21:35:14,320::task::495::TaskManager.Task::(_debug) Task 99c637b6-9c6c-46a4-8b2a-1c37dac491c0: moving from state init -> state preparing >Thread-15911467::DEBUG::2013-11-19 21:35:14,321::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911467::DEBUG::2013-11-19 21:35:14,322::task::495::TaskManager.Task::(_debug) Task 99c637b6-9c6c-46a4-8b2a-1c37dac491c0: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911467::DEBUG::2013-11-19 21:35:14,322::task::495::TaskManager.Task::(_debug) Task 99c637b6-9c6c-46a4-8b2a-1c37dac491c0: moving from state preparing -> state finished >Thread-15911467::DEBUG::2013-11-19 21:35:14,323::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911467::DEBUG::2013-11-19 21:35:14,323::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911467::DEBUG::2013-11-19 21:35:14,324::task::495::TaskManager.Task::(_debug) Task 99c637b6-9c6c-46a4-8b2a-1c37dac491c0: ref 0 aborting False >Thread-15911467::INFO::2013-11-19 21:35:14,324::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911468::DEBUG::2013-11-19 21:35:14,340::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911468::INFO::2013-11-19 21:35:14,341::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: clearTask, args: ( taskID=44a1be5a-8538-4745-8f2c-b3e70865e4e7) >Thread-15911468::DEBUG::2013-11-19 21:35:14,341::task::495::TaskManager.Task::(_debug) Task a9abba42-2b11-4e0c-b0ae-da44dd8049ea: moving from state init -> state preparing >Thread-15911468::DEBUG::2013-11-19 21:35:14,342::taskManager::146::TaskManager::(clearTask) Entry. taskID: 44a1be5a-8538-4745-8f2c-b3e70865e4e7 >Thread-15911468::DEBUG::2013-11-19 21:35:14,343::taskManager::151::TaskManager::(clearTask) Return. >Thread-15911468::DEBUG::2013-11-19 21:35:14,343::task::495::TaskManager.Task::(_debug) Task a9abba42-2b11-4e0c-b0ae-da44dd8049ea: finished: None >Thread-15911468::DEBUG::2013-11-19 21:35:14,344::task::495::TaskManager.Task::(_debug) Task a9abba42-2b11-4e0c-b0ae-da44dd8049ea: moving from state preparing -> state finished >Thread-15911468::DEBUG::2013-11-19 21:35:14,344::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911468::DEBUG::2013-11-19 21:35:14,345::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911468::DEBUG::2013-11-19 21:35:14,345::task::495::TaskManager.Task::(_debug) Task a9abba42-2b11-4e0c-b0ae-da44dd8049ea: ref 0 aborting False >Thread-15911468::INFO::2013-11-19 21:35:14,346::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: clearTask, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15911469::DEBUG::2013-11-19 21:35:14,411::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911469::INFO::2013-11-19 21:35:14,412::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911469::DEBUG::2013-11-19 21:35:14,413::task::495::TaskManager.Task::(_debug) Task 2bff2eef-c59b-4069-b260-a7877b83e086: moving from state init -> state preparing >Thread-15911469::DEBUG::2013-11-19 21:35:14,414::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3ec94a27-f0a9-4c60-b2a8-f2051c6b2363`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911469::DEBUG::2013-11-19 21:35:14,415::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911469::DEBUG::2013-11-19 21:35:14,415::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911469::DEBUG::2013-11-19 21:35:14,416::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3ec94a27-f0a9-4c60-b2a8-f2051c6b2363`::Granted request >Thread-15911469::DEBUG::2013-11-19 21:35:14,417::task::495::TaskManager.Task::(_debug) Task 2bff2eef-c59b-4069-b260-a7877b83e086: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911469::DEBUG::2013-11-19 21:35:14,417::task::495::TaskManager.Task::(_debug) Task 2bff2eef-c59b-4069-b260-a7877b83e086: ref 1 aborting False >Thread-15911469::WARNING::2013-11-19 21:35:14,419::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911469::DEBUG::2013-11-19 21:35:14,420::task::495::TaskManager.Task::(_debug) Task 2bff2eef-c59b-4069-b260-a7877b83e086: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911469::DEBUG::2013-11-19 21:35:14,421::task::495::TaskManager.Task::(_debug) Task 2bff2eef-c59b-4069-b260-a7877b83e086: moving from state preparing -> state finished >Thread-15911469::DEBUG::2013-11-19 21:35:14,422::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911469::DEBUG::2013-11-19 21:35:14,423::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911469::DEBUG::2013-11-19 21:35:14,423::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911469::DEBUG::2013-11-19 21:35:14,424::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911469::DEBUG::2013-11-19 21:35:14,424::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911469::DEBUG::2013-11-19 21:35:14,425::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911469::DEBUG::2013-11-19 21:35:14,426::task::495::TaskManager.Task::(_debug) Task 2bff2eef-c59b-4069-b260-a7877b83e086: ref 0 aborting False >Thread-15911469::INFO::2013-11-19 21:35:14,426::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911470::DEBUG::2013-11-19 21:35:14,586::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911470::INFO::2013-11-19 21:35:14,587::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksInfo, args: () >Thread-15911470::DEBUG::2013-11-19 21:35:14,587::task::495::TaskManager.Task::(_debug) Task 9b58b6ed-7cb0-4fa3-aada-fa3c28e1dd12: moving from state init -> state preparing >Thread-15911470::DEBUG::2013-11-19 21:35:14,588::taskManager::168::TaskManager::(getAllTasksInfo) Entry. >Thread-15911470::DEBUG::2013-11-19 21:35:14,588::taskManager::173::TaskManager::(getAllTasksInfo) Return. Response: {} >Thread-15911470::DEBUG::2013-11-19 21:35:14,589::task::495::TaskManager.Task::(_debug) Task 9b58b6ed-7cb0-4fa3-aada-fa3c28e1dd12: finished: {'allTasksInfo': {}} >Thread-15911470::DEBUG::2013-11-19 21:35:14,589::task::495::TaskManager.Task::(_debug) Task 9b58b6ed-7cb0-4fa3-aada-fa3c28e1dd12: moving from state preparing -> state finished >Thread-15911470::DEBUG::2013-11-19 21:35:14,590::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911470::DEBUG::2013-11-19 21:35:14,590::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911470::DEBUG::2013-11-19 21:35:14,591::task::495::TaskManager.Task::(_debug) Task 9b58b6ed-7cb0-4fa3-aada-fa3c28e1dd12: ref 0 aborting False >Thread-15911470::INFO::2013-11-19 21:35:14,592::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'allTasksInfo': {}} >Thread-15911471::DEBUG::2013-11-19 21:35:14,612::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911471::INFO::2013-11-19 21:35:14,612::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksInfo, args: () >Thread-15911471::DEBUG::2013-11-19 21:35:14,613::task::495::TaskManager.Task::(_debug) Task b20c9d99-8954-451c-b44d-ef3e60f29934: moving from state init -> state preparing >Thread-15911471::DEBUG::2013-11-19 21:35:14,614::taskManager::168::TaskManager::(getAllTasksInfo) Entry. >Thread-15911471::DEBUG::2013-11-19 21:35:14,614::taskManager::173::TaskManager::(getAllTasksInfo) Return. Response: {} >Thread-15911471::DEBUG::2013-11-19 21:35:14,615::task::495::TaskManager.Task::(_debug) Task b20c9d99-8954-451c-b44d-ef3e60f29934: finished: {'allTasksInfo': {}} >Thread-15911471::DEBUG::2013-11-19 21:35:14,615::task::495::TaskManager.Task::(_debug) Task b20c9d99-8954-451c-b44d-ef3e60f29934: moving from state preparing -> state finished >Thread-15911471::DEBUG::2013-11-19 21:35:14,616::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911471::DEBUG::2013-11-19 21:35:14,616::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911471::DEBUG::2013-11-19 21:35:14,617::task::495::TaskManager.Task::(_debug) Task b20c9d99-8954-451c-b44d-ef3e60f29934: ref 0 aborting False >Thread-15911471::INFO::2013-11-19 21:35:14,617::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'allTasksInfo': {}} >Dummy-15911465::DEBUG::2013-11-19 21:35:16,086::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:35:16,611::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.070185 s, 14.6 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:35:18,438::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:35:18,626::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:35:19,141::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0724285 s, 14.1 MB/s\n'; <rc> = 0 >Thread-15911474::INFO::2013-11-19 21:35:19,426::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911474::DEBUG::2013-11-19 21:35:19,427::task::495::TaskManager.Task::(_debug) Task a740853c-738c-4a8b-b3de-2df5d7ceb3fa: moving from state init -> state preparing >Thread-15911474::WARNING::2013-11-19 21:35:19,428::hsm::1895::Storage.HSM::(public_repoStats) repoStats detected invalid master:6b892067-ac7e-4039-a7be-5b7387525b16 {'finish': 1384896910.473666, 'mdasize': '134479872', 'mdathreshold': True, 'mdavalid': True, 'diskfree': '1142595518464', 'disktotal': '3595827150848', 'result': {'delay': '0.000699043273926', 'lastCheck': 1384896910.473666, 'valid': True, 'code': 0}, 'masterValidate': {'mount': False, 'valid': True}, 'mdafree': '67205632'} >Thread-15911474::DEBUG::2013-11-19 21:35:19,429::task::495::TaskManager.Task::(_debug) Task a740853c-738c-4a8b-b3de-2df5d7ceb3fa: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000699043273926', 'lastCheck': 1384896910.473666, 'valid': True, 'code': 355}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00635194778442', 'lastCheck': 1384896912.444699, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896918.438978, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00066089630127', 'lastCheck': 1384896909.266923, 'valid': True, 'code': 0}} >Thread-15911474::DEBUG::2013-11-19 21:35:19,439::task::495::TaskManager.Task::(_debug) Task a740853c-738c-4a8b-b3de-2df5d7ceb3fa: moving from state preparing -> state finished >Thread-15911474::DEBUG::2013-11-19 21:35:19,440::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911474::DEBUG::2013-11-19 21:35:19,441::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911474::DEBUG::2013-11-19 21:35:19,441::task::495::TaskManager.Task::(_debug) Task a740853c-738c-4a8b-b3de-2df5d7ceb3fa: ref 0 aborting False >Thread-15911474::INFO::2013-11-19 21:35:19,442::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000699043273926', 'lastCheck': 1384896910.473666, 'valid': True, 'code': 355}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00635194778442', 'lastCheck': 1384896912.444699, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896918.438978, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00066089630127', 'lastCheck': 1384896909.266923, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:35:21,157::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:35:21,689::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.070086 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:35:23,704::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:35:24,235::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0710186 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15911478::DEBUG::2013-11-19 21:35:24,583::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911478::INFO::2013-11-19 21:35:24,584::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911478::DEBUG::2013-11-19 21:35:24,585::task::495::TaskManager.Task::(_debug) Task 1bc4d5a1-55ec-431a-a4ef-b2b4008bf67f: moving from state init -> state preparing >Thread-15911478::DEBUG::2013-11-19 21:35:24,586::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911478::DEBUG::2013-11-19 21:35:24,587::task::495::TaskManager.Task::(_debug) Task 1bc4d5a1-55ec-431a-a4ef-b2b4008bf67f: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911478::DEBUG::2013-11-19 21:35:24,587::task::495::TaskManager.Task::(_debug) Task 1bc4d5a1-55ec-431a-a4ef-b2b4008bf67f: moving from state preparing -> state finished >Thread-15911478::DEBUG::2013-11-19 21:35:24,588::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911478::DEBUG::2013-11-19 21:35:24,588::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911478::DEBUG::2013-11-19 21:35:24,589::task::495::TaskManager.Task::(_debug) Task 1bc4d5a1-55ec-431a-a4ef-b2b4008bf67f: ref 0 aborting False >Thread-15911478::INFO::2013-11-19 21:35:24,590::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911479::DEBUG::2013-11-19 21:35:24,596::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911479::INFO::2013-11-19 21:35:24,597::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911479::DEBUG::2013-11-19 21:35:24,597::task::495::TaskManager.Task::(_debug) Task ad2e8349-7f14-4d9f-a511-08e4c53f2592: moving from state init -> state preparing >Thread-15911479::DEBUG::2013-11-19 21:35:24,598::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e721a359-5a68-4f71-be55-a46892b361f3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911479::DEBUG::2013-11-19 21:35:24,599::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911479::DEBUG::2013-11-19 21:35:24,600::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911479::DEBUG::2013-11-19 21:35:24,600::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e721a359-5a68-4f71-be55-a46892b361f3`::Granted request >Thread-15911479::DEBUG::2013-11-19 21:35:24,602::task::495::TaskManager.Task::(_debug) Task ad2e8349-7f14-4d9f-a511-08e4c53f2592: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911479::DEBUG::2013-11-19 21:35:24,602::task::495::TaskManager.Task::(_debug) Task ad2e8349-7f14-4d9f-a511-08e4c53f2592: ref 1 aborting False >Thread-15911479::WARNING::2013-11-19 21:35:24,604::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911479::DEBUG::2013-11-19 21:35:24,605::task::495::TaskManager.Task::(_debug) Task ad2e8349-7f14-4d9f-a511-08e4c53f2592: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911479::DEBUG::2013-11-19 21:35:24,606::task::495::TaskManager.Task::(_debug) Task ad2e8349-7f14-4d9f-a511-08e4c53f2592: moving from state preparing -> state finished >Thread-15911479::DEBUG::2013-11-19 21:35:24,607::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911479::DEBUG::2013-11-19 21:35:24,607::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911479::DEBUG::2013-11-19 21:35:24,608::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911479::DEBUG::2013-11-19 21:35:24,609::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911479::DEBUG::2013-11-19 21:35:24,609::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911479::DEBUG::2013-11-19 21:35:24,610::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911479::DEBUG::2013-11-19 21:35:24,610::task::495::TaskManager.Task::(_debug) Task ad2e8349-7f14-4d9f-a511-08e4c53f2592: ref 0 aborting False >Thread-15911479::INFO::2013-11-19 21:35:24,611::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:35:26,250::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:35:26,764::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0699634 s, 14.6 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:35:28,455::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:35:28,780::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:35:29,316::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0700693 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15911482::INFO::2013-11-19 21:35:29,577::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911482::DEBUG::2013-11-19 21:35:29,578::task::495::TaskManager.Task::(_debug) Task 4fb253b1-736e-4279-86dc-1b984d3cbb2f: moving from state init -> state preparing >Thread-15911482::DEBUG::2013-11-19 21:35:29,579::task::495::TaskManager.Task::(_debug) Task 4fb253b1-736e-4279-86dc-1b984d3cbb2f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000339984893799', 'lastCheck': 1384896920.869468, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00151491165161', 'lastCheck': 1384896922.4618061, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896928.4563911, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000466108322144', 'lastCheck': 1384896929.457324, 'valid': True, 'code': 0}} >Thread-15911482::DEBUG::2013-11-19 21:35:29,583::task::495::TaskManager.Task::(_debug) Task 4fb253b1-736e-4279-86dc-1b984d3cbb2f: moving from state preparing -> state finished >Thread-15911482::DEBUG::2013-11-19 21:35:29,584::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911482::DEBUG::2013-11-19 21:35:29,584::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911482::DEBUG::2013-11-19 21:35:29,585::task::495::TaskManager.Task::(_debug) Task 4fb253b1-736e-4279-86dc-1b984d3cbb2f: ref 0 aborting False >Thread-15911482::INFO::2013-11-19 21:35:29,587::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000339984893799', 'lastCheck': 1384896920.869468, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00151491165161', 'lastCheck': 1384896922.4618061, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896928.4563911, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000466108322144', 'lastCheck': 1384896929.457324, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:35:31,331::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:35:31,868::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.079768 s, 12.8 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:35:33,883::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:35:34,427::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0703569 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15911486::DEBUG::2013-11-19 21:35:34,790::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911486::INFO::2013-11-19 21:35:34,791::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911486::DEBUG::2013-11-19 21:35:34,792::task::495::TaskManager.Task::(_debug) Task 35b5af62-61c6-4923-a86b-2ed001072389: moving from state init -> state preparing >Thread-15911486::DEBUG::2013-11-19 21:35:34,793::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911486::DEBUG::2013-11-19 21:35:34,793::task::495::TaskManager.Task::(_debug) Task 35b5af62-61c6-4923-a86b-2ed001072389: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911486::DEBUG::2013-11-19 21:35:34,794::task::495::TaskManager.Task::(_debug) Task 35b5af62-61c6-4923-a86b-2ed001072389: moving from state preparing -> state finished >Thread-15911486::DEBUG::2013-11-19 21:35:34,794::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911486::DEBUG::2013-11-19 21:35:34,795::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911486::DEBUG::2013-11-19 21:35:34,796::task::495::TaskManager.Task::(_debug) Task 35b5af62-61c6-4923-a86b-2ed001072389: ref 0 aborting False >Thread-15911486::INFO::2013-11-19 21:35:34,796::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911487::DEBUG::2013-11-19 21:35:34,802::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911487::INFO::2013-11-19 21:35:34,803::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911487::DEBUG::2013-11-19 21:35:34,804::task::495::TaskManager.Task::(_debug) Task 47fc1bfe-5aa0-4864-a2b2-1a344ff277c6: moving from state init -> state preparing >Thread-15911487::DEBUG::2013-11-19 21:35:34,805::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`088623de-3cc9-4d3a-ba46-4c15cabf668d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911487::DEBUG::2013-11-19 21:35:34,806::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911487::DEBUG::2013-11-19 21:35:34,807::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911487::DEBUG::2013-11-19 21:35:34,807::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`088623de-3cc9-4d3a-ba46-4c15cabf668d`::Granted request >Thread-15911487::DEBUG::2013-11-19 21:35:34,808::task::495::TaskManager.Task::(_debug) Task 47fc1bfe-5aa0-4864-a2b2-1a344ff277c6: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911487::DEBUG::2013-11-19 21:35:34,809::task::495::TaskManager.Task::(_debug) Task 47fc1bfe-5aa0-4864-a2b2-1a344ff277c6: ref 1 aborting False >Thread-15911487::WARNING::2013-11-19 21:35:34,811::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911487::DEBUG::2013-11-19 21:35:34,812::task::495::TaskManager.Task::(_debug) Task 47fc1bfe-5aa0-4864-a2b2-1a344ff277c6: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911487::DEBUG::2013-11-19 21:35:34,813::task::495::TaskManager.Task::(_debug) Task 47fc1bfe-5aa0-4864-a2b2-1a344ff277c6: moving from state preparing -> state finished >Thread-15911487::DEBUG::2013-11-19 21:35:34,813::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911487::DEBUG::2013-11-19 21:35:34,814::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911487::DEBUG::2013-11-19 21:35:34,815::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911487::DEBUG::2013-11-19 21:35:34,815::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911487::DEBUG::2013-11-19 21:35:34,816::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911487::DEBUG::2013-11-19 21:35:34,816::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911487::DEBUG::2013-11-19 21:35:34,817::task::495::TaskManager.Task::(_debug) Task 47fc1bfe-5aa0-4864-a2b2-1a344ff277c6: ref 0 aborting False >Thread-15911487::INFO::2013-11-19 21:35:34,818::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:35:36,443::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:35:36,967::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0704573 s, 14.5 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:35:38,659::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:35:38,982::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:35:39,410::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0446349 s, 22.9 MB/s\n'; <rc> = 0 >Thread-15911490::INFO::2013-11-19 21:35:39,783::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911490::DEBUG::2013-11-19 21:35:39,784::task::495::TaskManager.Task::(_debug) Task d91b8791-1ae1-4d19-9edf-c1529e30af8e: moving from state init -> state preparing >Thread-15911490::DEBUG::2013-11-19 21:35:39,794::task::495::TaskManager.Task::(_debug) Task d91b8791-1ae1-4d19-9edf-c1529e30af8e: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000402927398682', 'lastCheck': 1384896931.0225, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147008895874', 'lastCheck': 1384896932.5518439, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896938.660331, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000509977340698', 'lastCheck': 1384896939.658987, 'valid': True, 'code': 0}} >Thread-15911490::DEBUG::2013-11-19 21:35:39,804::task::495::TaskManager.Task::(_debug) Task d91b8791-1ae1-4d19-9edf-c1529e30af8e: moving from state preparing -> state finished >Thread-15911490::DEBUG::2013-11-19 21:35:39,805::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911490::DEBUG::2013-11-19 21:35:39,806::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911490::DEBUG::2013-11-19 21:35:39,806::task::495::TaskManager.Task::(_debug) Task d91b8791-1ae1-4d19-9edf-c1529e30af8e: ref 0 aborting False >Thread-15911490::INFO::2013-11-19 21:35:39,807::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000402927398682', 'lastCheck': 1384896931.0225, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147008895874', 'lastCheck': 1384896932.5518439, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896938.660331, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000509977340698', 'lastCheck': 1384896939.658987, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:35:41,420::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:35:41,961::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0746533 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:35:43,976::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:35:44,499::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0692762 s, 14.8 MB/s\n'; <rc> = 0 >Thread-15911494::DEBUG::2013-11-19 21:35:44,923::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911494::INFO::2013-11-19 21:35:44,924::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911494::DEBUG::2013-11-19 21:35:44,933::task::495::TaskManager.Task::(_debug) Task 8db61e71-e341-44eb-a2a9-880192e86fb0: moving from state init -> state preparing >Thread-15911494::DEBUG::2013-11-19 21:35:44,934::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911494::DEBUG::2013-11-19 21:35:44,934::task::495::TaskManager.Task::(_debug) Task 8db61e71-e341-44eb-a2a9-880192e86fb0: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911494::DEBUG::2013-11-19 21:35:44,935::task::495::TaskManager.Task::(_debug) Task 8db61e71-e341-44eb-a2a9-880192e86fb0: moving from state preparing -> state finished >Thread-15911494::DEBUG::2013-11-19 21:35:44,936::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911494::DEBUG::2013-11-19 21:35:44,936::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911494::DEBUG::2013-11-19 21:35:44,937::task::495::TaskManager.Task::(_debug) Task 8db61e71-e341-44eb-a2a9-880192e86fb0: ref 0 aborting False >Thread-15911494::INFO::2013-11-19 21:35:44,937::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911495::DEBUG::2013-11-19 21:35:44,947::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911495::INFO::2013-11-19 21:35:44,948::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911495::DEBUG::2013-11-19 21:35:44,949::task::495::TaskManager.Task::(_debug) Task f247bfd2-31ca-463a-99aa-840558a703e7: moving from state init -> state preparing >Thread-15911495::DEBUG::2013-11-19 21:35:44,950::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5164ae1d-8348-4ad3-a550-646a1159a63e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911495::DEBUG::2013-11-19 21:35:44,951::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911495::DEBUG::2013-11-19 21:35:44,952::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911495::DEBUG::2013-11-19 21:35:44,952::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5164ae1d-8348-4ad3-a550-646a1159a63e`::Granted request >Thread-15911495::DEBUG::2013-11-19 21:35:44,953::task::495::TaskManager.Task::(_debug) Task f247bfd2-31ca-463a-99aa-840558a703e7: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911495::DEBUG::2013-11-19 21:35:44,954::task::495::TaskManager.Task::(_debug) Task f247bfd2-31ca-463a-99aa-840558a703e7: ref 1 aborting False >Thread-15911495::WARNING::2013-11-19 21:35:44,956::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911495::DEBUG::2013-11-19 21:35:44,957::task::495::TaskManager.Task::(_debug) Task f247bfd2-31ca-463a-99aa-840558a703e7: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911495::DEBUG::2013-11-19 21:35:44,968::task::495::TaskManager.Task::(_debug) Task f247bfd2-31ca-463a-99aa-840558a703e7: moving from state preparing -> state finished >Thread-15911495::DEBUG::2013-11-19 21:35:44,969::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911495::DEBUG::2013-11-19 21:35:44,969::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911495::DEBUG::2013-11-19 21:35:44,970::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911495::DEBUG::2013-11-19 21:35:44,970::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911495::DEBUG::2013-11-19 21:35:44,971::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911495::DEBUG::2013-11-19 21:35:44,971::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911495::DEBUG::2013-11-19 21:35:44,972::task::495::TaskManager.Task::(_debug) Task f247bfd2-31ca-463a-99aa-840558a703e7: ref 0 aborting False >Thread-15911495::INFO::2013-11-19 21:35:44,973::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:35:46,515::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:35:47,033::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0692758 s, 14.8 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:35:48,740::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:35:49,048::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:35:49,591::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0704397 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15911498::INFO::2013-11-19 21:35:49,897::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911498::DEBUG::2013-11-19 21:35:49,898::task::495::TaskManager.Task::(_debug) Task 600ba0b8-e302-4f8d-b38e-bfb2f54c2188: moving from state init -> state preparing >Thread-15911498::DEBUG::2013-11-19 21:35:49,898::task::495::TaskManager.Task::(_debug) Task 600ba0b8-e302-4f8d-b38e-bfb2f54c2188: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000434160232544', 'lastCheck': 1384896941.219434, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144791603088', 'lastCheck': 1384896942.666486, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896948.7416871, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00051212310791', 'lastCheck': 1384896949.7396619, 'valid': True, 'code': 0}} >Thread-15911498::DEBUG::2013-11-19 21:35:49,905::task::495::TaskManager.Task::(_debug) Task 600ba0b8-e302-4f8d-b38e-bfb2f54c2188: moving from state preparing -> state finished >Thread-15911498::DEBUG::2013-11-19 21:35:49,906::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911498::DEBUG::2013-11-19 21:35:49,906::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911498::DEBUG::2013-11-19 21:35:49,907::task::495::TaskManager.Task::(_debug) Task 600ba0b8-e302-4f8d-b38e-bfb2f54c2188: ref 0 aborting False >Thread-15911498::INFO::2013-11-19 21:35:49,908::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000434160232544', 'lastCheck': 1384896941.219434, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144791603088', 'lastCheck': 1384896942.666486, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896948.7416871, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00051212310791', 'lastCheck': 1384896949.7396619, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:35:51,607::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:35:52,124::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0696205 s, 14.7 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:35:54,139::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:35:54,694::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0704281 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15911502::DEBUG::2013-11-19 21:35:55,063::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911502::INFO::2013-11-19 21:35:55,064::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911502::DEBUG::2013-11-19 21:35:55,065::task::495::TaskManager.Task::(_debug) Task 8388d43d-6abf-490d-8748-ed7ab41d2ae4: moving from state init -> state preparing >Thread-15911502::DEBUG::2013-11-19 21:35:55,066::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911502::DEBUG::2013-11-19 21:35:55,066::task::495::TaskManager.Task::(_debug) Task 8388d43d-6abf-490d-8748-ed7ab41d2ae4: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911502::DEBUG::2013-11-19 21:35:55,067::task::495::TaskManager.Task::(_debug) Task 8388d43d-6abf-490d-8748-ed7ab41d2ae4: moving from state preparing -> state finished >Thread-15911502::DEBUG::2013-11-19 21:35:55,067::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911502::DEBUG::2013-11-19 21:35:55,068::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911502::DEBUG::2013-11-19 21:35:55,068::task::495::TaskManager.Task::(_debug) Task 8388d43d-6abf-490d-8748-ed7ab41d2ae4: ref 0 aborting False >Thread-15911502::INFO::2013-11-19 21:35:55,069::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911503::DEBUG::2013-11-19 21:35:55,080::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911503::INFO::2013-11-19 21:35:55,081::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911503::DEBUG::2013-11-19 21:35:55,081::task::495::TaskManager.Task::(_debug) Task cdd03187-c99c-42ee-9401-b17ff24811f6: moving from state init -> state preparing >Thread-15911503::DEBUG::2013-11-19 21:35:55,083::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`904ae441-f0e3-42a6-9910-f9c00095af37`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911503::DEBUG::2013-11-19 21:35:55,083::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911503::DEBUG::2013-11-19 21:35:55,084::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911503::DEBUG::2013-11-19 21:35:55,085::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`904ae441-f0e3-42a6-9910-f9c00095af37`::Granted request >Thread-15911503::DEBUG::2013-11-19 21:35:55,086::task::495::TaskManager.Task::(_debug) Task cdd03187-c99c-42ee-9401-b17ff24811f6: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911503::DEBUG::2013-11-19 21:35:55,086::task::495::TaskManager.Task::(_debug) Task cdd03187-c99c-42ee-9401-b17ff24811f6: ref 1 aborting False >Thread-15911503::WARNING::2013-11-19 21:35:55,088::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911503::DEBUG::2013-11-19 21:35:55,089::task::495::TaskManager.Task::(_debug) Task cdd03187-c99c-42ee-9401-b17ff24811f6: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911503::DEBUG::2013-11-19 21:35:55,090::task::495::TaskManager.Task::(_debug) Task cdd03187-c99c-42ee-9401-b17ff24811f6: moving from state preparing -> state finished >Thread-15911503::DEBUG::2013-11-19 21:35:55,091::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911503::DEBUG::2013-11-19 21:35:55,092::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911503::DEBUG::2013-11-19 21:35:55,092::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911503::DEBUG::2013-11-19 21:35:55,093::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911503::DEBUG::2013-11-19 21:35:55,093::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911503::DEBUG::2013-11-19 21:35:55,094::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911503::DEBUG::2013-11-19 21:35:55,095::task::495::TaskManager.Task::(_debug) Task cdd03187-c99c-42ee-9401-b17ff24811f6: ref 0 aborting False >Thread-15911503::INFO::2013-11-19 21:35:55,098::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:35:56,709::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:35:57,241::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0696661 s, 14.7 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:35:58,931::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:35:59,255::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:35:59,812::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0702366 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15911506::INFO::2013-11-19 21:36:00,015::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911506::DEBUG::2013-11-19 21:36:00,015::task::495::TaskManager.Task::(_debug) Task 9663cdd5-7fde-42ed-8761-51d99f720662: moving from state init -> state preparing >Thread-15911506::DEBUG::2013-11-19 21:36:00,016::task::495::TaskManager.Task::(_debug) Task 9663cdd5-7fde-42ed-8761-51d99f720662: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000297069549561', 'lastCheck': 1384896951.276562, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00225305557251', 'lastCheck': 1384896952.821914, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896958.9328351, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000590085983276', 'lastCheck': 1384896959.9311881, 'valid': True, 'code': 0}} >Thread-15911506::DEBUG::2013-11-19 21:36:00,022::task::495::TaskManager.Task::(_debug) Task 9663cdd5-7fde-42ed-8761-51d99f720662: moving from state preparing -> state finished >Thread-15911506::DEBUG::2013-11-19 21:36:00,023::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911506::DEBUG::2013-11-19 21:36:00,024::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911506::DEBUG::2013-11-19 21:36:00,024::task::495::TaskManager.Task::(_debug) Task 9663cdd5-7fde-42ed-8761-51d99f720662: ref 0 aborting False >Thread-15911506::INFO::2013-11-19 21:36:00,025::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000297069549561', 'lastCheck': 1384896951.276562, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00225305557251', 'lastCheck': 1384896952.821914, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896958.9328351, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000590085983276', 'lastCheck': 1384896959.9311881, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:36:01,829::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:02,375::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0844066 s, 12.1 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:36:04,390::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:04,942::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0704773 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15911510::DEBUG::2013-11-19 21:36:05,230::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911510::INFO::2013-11-19 21:36:05,237::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911510::DEBUG::2013-11-19 21:36:05,240::task::495::TaskManager.Task::(_debug) Task 35c27000-ee94-4725-8cf5-e8a2a80031bd: moving from state init -> state preparing >Thread-15911510::DEBUG::2013-11-19 21:36:05,248::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911510::DEBUG::2013-11-19 21:36:05,252::task::495::TaskManager.Task::(_debug) Task 35c27000-ee94-4725-8cf5-e8a2a80031bd: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911510::DEBUG::2013-11-19 21:36:05,255::task::495::TaskManager.Task::(_debug) Task 35c27000-ee94-4725-8cf5-e8a2a80031bd: moving from state preparing -> state finished >Thread-15911510::DEBUG::2013-11-19 21:36:05,256::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911510::DEBUG::2013-11-19 21:36:05,257::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911510::DEBUG::2013-11-19 21:36:05,257::task::495::TaskManager.Task::(_debug) Task 35c27000-ee94-4725-8cf5-e8a2a80031bd: ref 0 aborting False >Thread-15911510::INFO::2013-11-19 21:36:05,258::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911511::DEBUG::2013-11-19 21:36:05,267::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911511::INFO::2013-11-19 21:36:05,268::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911511::DEBUG::2013-11-19 21:36:05,268::task::495::TaskManager.Task::(_debug) Task 416aaf45-e0d6-468a-8212-68664d4e78c3: moving from state init -> state preparing >Thread-15911511::DEBUG::2013-11-19 21:36:05,270::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2cf421fc-0ef0-41c2-8cc3-27f7f876e83d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911511::DEBUG::2013-11-19 21:36:05,270::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911511::DEBUG::2013-11-19 21:36:05,271::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911511::DEBUG::2013-11-19 21:36:05,272::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2cf421fc-0ef0-41c2-8cc3-27f7f876e83d`::Granted request >Thread-15911511::DEBUG::2013-11-19 21:36:05,273::task::495::TaskManager.Task::(_debug) Task 416aaf45-e0d6-468a-8212-68664d4e78c3: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911511::DEBUG::2013-11-19 21:36:05,273::task::495::TaskManager.Task::(_debug) Task 416aaf45-e0d6-468a-8212-68664d4e78c3: ref 1 aborting False >Thread-15911511::WARNING::2013-11-19 21:36:05,275::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911511::DEBUG::2013-11-19 21:36:05,276::task::495::TaskManager.Task::(_debug) Task 416aaf45-e0d6-468a-8212-68664d4e78c3: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911511::DEBUG::2013-11-19 21:36:05,277::task::495::TaskManager.Task::(_debug) Task 416aaf45-e0d6-468a-8212-68664d4e78c3: moving from state preparing -> state finished >Thread-15911511::DEBUG::2013-11-19 21:36:05,278::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911511::DEBUG::2013-11-19 21:36:05,278::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911511::DEBUG::2013-11-19 21:36:05,279::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911511::DEBUG::2013-11-19 21:36:05,280::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911511::DEBUG::2013-11-19 21:36:05,280::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911511::DEBUG::2013-11-19 21:36:05,281::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911511::DEBUG::2013-11-19 21:36:05,281::task::495::TaskManager.Task::(_debug) Task 416aaf45-e0d6-468a-8212-68664d4e78c3: ref 0 aborting False >Thread-15911511::INFO::2013-11-19 21:36:05,282::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:36:06,958::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:07,486::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0703399 s, 14.6 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:36:09,175::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:36:09,500::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:10,011::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0700202 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15911514::INFO::2013-11-19 21:36:10,212::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911514::DEBUG::2013-11-19 21:36:10,212::task::495::TaskManager.Task::(_debug) Task a56927d7-7a63-435f-bf22-c5301270d369: moving from state init -> state preparing >Thread-15911514::DEBUG::2013-11-19 21:36:10,213::task::495::TaskManager.Task::(_debug) Task a56927d7-7a63-435f-bf22-c5301270d369: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000431060791016', 'lastCheck': 1384896961.5056629, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147986412048', 'lastCheck': 1384896963.049371, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896969.175921, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000556945800781', 'lastCheck': 1384896970.1740961, 'valid': True, 'code': 0}} >Thread-15911514::DEBUG::2013-11-19 21:36:10,220::task::495::TaskManager.Task::(_debug) Task a56927d7-7a63-435f-bf22-c5301270d369: moving from state preparing -> state finished >Thread-15911514::DEBUG::2013-11-19 21:36:10,221::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911514::DEBUG::2013-11-19 21:36:10,221::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911514::DEBUG::2013-11-19 21:36:10,222::task::495::TaskManager.Task::(_debug) Task a56927d7-7a63-435f-bf22-c5301270d369: ref 0 aborting False >Thread-15911514::INFO::2013-11-19 21:36:10,223::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000431060791016', 'lastCheck': 1384896961.5056629, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147986412048', 'lastCheck': 1384896963.049371, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896969.175921, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000556945800781', 'lastCheck': 1384896970.1740961, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:36:12,026::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:12,584::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0709732 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:36:14,599::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:15,124::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0699282 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15911518::DEBUG::2013-11-19 21:36:15,438::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911518::INFO::2013-11-19 21:36:15,442::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911518::DEBUG::2013-11-19 21:36:15,443::task::495::TaskManager.Task::(_debug) Task 45ab61d4-b262-400f-b3d4-0f0d74ed4b86: moving from state init -> state preparing >Thread-15911518::DEBUG::2013-11-19 21:36:15,444::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911518::DEBUG::2013-11-19 21:36:15,445::task::495::TaskManager.Task::(_debug) Task 45ab61d4-b262-400f-b3d4-0f0d74ed4b86: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911518::DEBUG::2013-11-19 21:36:15,445::task::495::TaskManager.Task::(_debug) Task 45ab61d4-b262-400f-b3d4-0f0d74ed4b86: moving from state preparing -> state finished >Thread-15911518::DEBUG::2013-11-19 21:36:15,446::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911518::DEBUG::2013-11-19 21:36:15,446::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911518::DEBUG::2013-11-19 21:36:15,447::task::495::TaskManager.Task::(_debug) Task 45ab61d4-b262-400f-b3d4-0f0d74ed4b86: ref 0 aborting False >Thread-15911518::INFO::2013-11-19 21:36:15,448::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911519::DEBUG::2013-11-19 21:36:15,454::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911519::INFO::2013-11-19 21:36:15,455::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911519::DEBUG::2013-11-19 21:36:15,455::task::495::TaskManager.Task::(_debug) Task c903ad96-62d4-4f92-82ab-74367775ebf1: moving from state init -> state preparing >Thread-15911519::DEBUG::2013-11-19 21:36:15,457::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`8275ee56-83e1-441a-b08a-f7671697f2a0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911519::DEBUG::2013-11-19 21:36:15,457::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911519::DEBUG::2013-11-19 21:36:15,458::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911519::DEBUG::2013-11-19 21:36:15,459::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`8275ee56-83e1-441a-b08a-f7671697f2a0`::Granted request >Thread-15911519::DEBUG::2013-11-19 21:36:15,460::task::495::TaskManager.Task::(_debug) Task c903ad96-62d4-4f92-82ab-74367775ebf1: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911519::DEBUG::2013-11-19 21:36:15,460::task::495::TaskManager.Task::(_debug) Task c903ad96-62d4-4f92-82ab-74367775ebf1: ref 1 aborting False >Thread-15911519::WARNING::2013-11-19 21:36:15,462::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911519::DEBUG::2013-11-19 21:36:15,463::task::495::TaskManager.Task::(_debug) Task c903ad96-62d4-4f92-82ab-74367775ebf1: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911519::DEBUG::2013-11-19 21:36:15,464::task::495::TaskManager.Task::(_debug) Task c903ad96-62d4-4f92-82ab-74367775ebf1: moving from state preparing -> state finished >Thread-15911519::DEBUG::2013-11-19 21:36:15,465::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911519::DEBUG::2013-11-19 21:36:15,465::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911519::DEBUG::2013-11-19 21:36:15,466::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911519::DEBUG::2013-11-19 21:36:15,467::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911519::DEBUG::2013-11-19 21:36:15,467::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911519::DEBUG::2013-11-19 21:36:15,468::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911519::DEBUG::2013-11-19 21:36:15,468::task::495::TaskManager.Task::(_debug) Task c903ad96-62d4-4f92-82ab-74367775ebf1: ref 0 aborting False >Thread-15911519::INFO::2013-11-19 21:36:15,469::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:36:17,142::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:17,689::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0703454 s, 14.6 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:36:19,403::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:36:19,705::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:20,236::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.070886 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15911522::INFO::2013-11-19 21:36:20,334::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911522::DEBUG::2013-11-19 21:36:20,335::task::495::TaskManager.Task::(_debug) Task d50313f5-7fee-4a0b-b3fa-85f9e1865ec9: moving from state init -> state preparing >Thread-15911522::DEBUG::2013-11-19 21:36:20,336::task::495::TaskManager.Task::(_debug) Task d50313f5-7fee-4a0b-b3fa-85f9e1865ec9: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000438928604126', 'lastCheck': 1384896971.7007599, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00143098831177', 'lastCheck': 1384896973.264729, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896979.4038291, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000556945800781', 'lastCheck': 1384896970.1740961, 'valid': True, 'code': 0}} >Thread-15911522::DEBUG::2013-11-19 21:36:20,337::task::495::TaskManager.Task::(_debug) Task d50313f5-7fee-4a0b-b3fa-85f9e1865ec9: moving from state preparing -> state finished >Thread-15911522::DEBUG::2013-11-19 21:36:20,338::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911522::DEBUG::2013-11-19 21:36:20,338::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911522::DEBUG::2013-11-19 21:36:20,339::task::495::TaskManager.Task::(_debug) Task d50313f5-7fee-4a0b-b3fa-85f9e1865ec9: ref 0 aborting False >Thread-15911522::INFO::2013-11-19 21:36:20,340::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000438928604126', 'lastCheck': 1384896971.7007599, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00143098831177', 'lastCheck': 1384896973.264729, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896979.4038291, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000556945800781', 'lastCheck': 1384896970.1740961, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:36:22,250::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:22,760::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0699207 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:36:24,776::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:25,319::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0706312 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15911526::DEBUG::2013-11-19 21:36:25,577::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911526::INFO::2013-11-19 21:36:25,578::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911526::DEBUG::2013-11-19 21:36:25,579::task::495::TaskManager.Task::(_debug) Task bfa01b48-11a3-468a-9b17-157c58ed5d75: moving from state init -> state preparing >Thread-15911526::DEBUG::2013-11-19 21:36:25,580::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911526::DEBUG::2013-11-19 21:36:25,580::task::495::TaskManager.Task::(_debug) Task bfa01b48-11a3-468a-9b17-157c58ed5d75: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911526::DEBUG::2013-11-19 21:36:25,581::task::495::TaskManager.Task::(_debug) Task bfa01b48-11a3-468a-9b17-157c58ed5d75: moving from state preparing -> state finished >Thread-15911526::DEBUG::2013-11-19 21:36:25,582::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911526::DEBUG::2013-11-19 21:36:25,582::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911526::DEBUG::2013-11-19 21:36:25,583::task::495::TaskManager.Task::(_debug) Task bfa01b48-11a3-468a-9b17-157c58ed5d75: ref 0 aborting False >Thread-15911526::INFO::2013-11-19 21:36:25,583::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911527::DEBUG::2013-11-19 21:36:25,589::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911527::INFO::2013-11-19 21:36:25,590::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911527::DEBUG::2013-11-19 21:36:25,591::task::495::TaskManager.Task::(_debug) Task 1666d51d-2307-41ee-bfa5-ee9b114042dc: moving from state init -> state preparing >Thread-15911527::DEBUG::2013-11-19 21:36:25,592::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c35dfaf8-0e8b-4867-bc91-c0846f13889e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911527::DEBUG::2013-11-19 21:36:25,593::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911527::DEBUG::2013-11-19 21:36:25,594::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911527::DEBUG::2013-11-19 21:36:25,594::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c35dfaf8-0e8b-4867-bc91-c0846f13889e`::Granted request >Thread-15911527::DEBUG::2013-11-19 21:36:25,604::task::495::TaskManager.Task::(_debug) Task 1666d51d-2307-41ee-bfa5-ee9b114042dc: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911527::DEBUG::2013-11-19 21:36:25,605::task::495::TaskManager.Task::(_debug) Task 1666d51d-2307-41ee-bfa5-ee9b114042dc: ref 1 aborting False >Thread-15911527::WARNING::2013-11-19 21:36:25,607::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911527::DEBUG::2013-11-19 21:36:25,608::task::495::TaskManager.Task::(_debug) Task 1666d51d-2307-41ee-bfa5-ee9b114042dc: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911527::DEBUG::2013-11-19 21:36:25,609::task::495::TaskManager.Task::(_debug) Task 1666d51d-2307-41ee-bfa5-ee9b114042dc: moving from state preparing -> state finished >Thread-15911527::DEBUG::2013-11-19 21:36:25,609::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911527::DEBUG::2013-11-19 21:36:25,610::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911527::DEBUG::2013-11-19 21:36:25,611::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911527::DEBUG::2013-11-19 21:36:25,611::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911527::DEBUG::2013-11-19 21:36:25,612::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911527::DEBUG::2013-11-19 21:36:25,612::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911527::DEBUG::2013-11-19 21:36:25,613::task::495::TaskManager.Task::(_debug) Task 1666d51d-2307-41ee-bfa5-ee9b114042dc: ref 0 aborting False >Thread-15911527::INFO::2013-11-19 21:36:25,614::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:36:27,334::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:27,865::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0704832 s, 14.5 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:36:29,576::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:36:29,880::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:30,408::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0703108 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15911530::INFO::2013-11-19 21:36:30,514::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911530::DEBUG::2013-11-19 21:36:30,515::task::495::TaskManager.Task::(_debug) Task 7a65f44c-1ee6-47f4-9f16-c21ded082352: moving from state init -> state preparing >Thread-15911530::DEBUG::2013-11-19 21:36:30,516::task::495::TaskManager.Task::(_debug) Task 7a65f44c-1ee6-47f4-9f16-c21ded082352: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000430107116699', 'lastCheck': 1384896981.922365, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00120496749878', 'lastCheck': 1384896983.4630129, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896989.5774319, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000617027282715', 'lastCheck': 1384896980.3921421, 'valid': True, 'code': 0}} >Thread-15911530::DEBUG::2013-11-19 21:36:30,521::task::495::TaskManager.Task::(_debug) Task 7a65f44c-1ee6-47f4-9f16-c21ded082352: moving from state preparing -> state finished >Thread-15911530::DEBUG::2013-11-19 21:36:30,522::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911530::DEBUG::2013-11-19 21:36:30,522::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911530::DEBUG::2013-11-19 21:36:30,523::task::495::TaskManager.Task::(_debug) Task 7a65f44c-1ee6-47f4-9f16-c21ded082352: ref 0 aborting False >Thread-15911530::INFO::2013-11-19 21:36:30,523::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000430107116699', 'lastCheck': 1384896981.922365, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00120496749878', 'lastCheck': 1384896983.4630129, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896989.5774319, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000617027282715', 'lastCheck': 1384896980.3921421, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:36:32,422::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:32,963::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0705334 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:36:34,978::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:35,525::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0952947 s, 10.7 MB/s\n'; <rc> = 0 >Thread-15911534::DEBUG::2013-11-19 21:36:35,743::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911534::INFO::2013-11-19 21:36:35,744::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911534::DEBUG::2013-11-19 21:36:35,745::task::495::TaskManager.Task::(_debug) Task 8b6366a5-ad71-4704-a10a-7e71e28f1fb1: moving from state init -> state preparing >Thread-15911534::DEBUG::2013-11-19 21:36:35,746::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911534::DEBUG::2013-11-19 21:36:35,747::task::495::TaskManager.Task::(_debug) Task 8b6366a5-ad71-4704-a10a-7e71e28f1fb1: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911534::DEBUG::2013-11-19 21:36:35,747::task::495::TaskManager.Task::(_debug) Task 8b6366a5-ad71-4704-a10a-7e71e28f1fb1: moving from state preparing -> state finished >Thread-15911534::DEBUG::2013-11-19 21:36:35,748::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911534::DEBUG::2013-11-19 21:36:35,748::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911534::DEBUG::2013-11-19 21:36:35,749::task::495::TaskManager.Task::(_debug) Task 8b6366a5-ad71-4704-a10a-7e71e28f1fb1: ref 0 aborting False >Thread-15911534::INFO::2013-11-19 21:36:35,750::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911535::DEBUG::2013-11-19 21:36:35,764::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911535::INFO::2013-11-19 21:36:35,765::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911535::DEBUG::2013-11-19 21:36:35,765::task::495::TaskManager.Task::(_debug) Task dda0638a-1667-4f2a-bf48-d0751c7970ee: moving from state init -> state preparing >Thread-15911535::DEBUG::2013-11-19 21:36:35,767::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`4adb4385-ee98-44e4-8a38-3fdd36aecfbd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911535::DEBUG::2013-11-19 21:36:35,767::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911535::DEBUG::2013-11-19 21:36:35,768::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911535::DEBUG::2013-11-19 21:36:35,769::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`4adb4385-ee98-44e4-8a38-3fdd36aecfbd`::Granted request >Thread-15911535::DEBUG::2013-11-19 21:36:35,770::task::495::TaskManager.Task::(_debug) Task dda0638a-1667-4f2a-bf48-d0751c7970ee: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911535::DEBUG::2013-11-19 21:36:35,770::task::495::TaskManager.Task::(_debug) Task dda0638a-1667-4f2a-bf48-d0751c7970ee: ref 1 aborting False >Thread-15911535::WARNING::2013-11-19 21:36:35,772::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911535::DEBUG::2013-11-19 21:36:35,773::task::495::TaskManager.Task::(_debug) Task dda0638a-1667-4f2a-bf48-d0751c7970ee: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911535::DEBUG::2013-11-19 21:36:35,774::task::495::TaskManager.Task::(_debug) Task dda0638a-1667-4f2a-bf48-d0751c7970ee: moving from state preparing -> state finished >Thread-15911535::DEBUG::2013-11-19 21:36:35,775::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911535::DEBUG::2013-11-19 21:36:35,775::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911535::DEBUG::2013-11-19 21:36:35,776::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911535::DEBUG::2013-11-19 21:36:35,777::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911535::DEBUG::2013-11-19 21:36:35,777::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911535::DEBUG::2013-11-19 21:36:35,778::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911535::DEBUG::2013-11-19 21:36:35,778::task::495::TaskManager.Task::(_debug) Task dda0638a-1667-4f2a-bf48-d0751c7970ee: ref 0 aborting False >Thread-15911535::INFO::2013-11-19 21:36:35,779::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:36:37,540::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:38,091::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0709523 s, 14.4 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:36:39,774::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:36:40,107::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:40,629::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0695903 s, 14.7 MB/s\n'; <rc> = 0 >Thread-15911538::INFO::2013-11-19 21:36:40,728::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911538::DEBUG::2013-11-19 21:36:40,729::task::495::TaskManager.Task::(_debug) Task 67fe47f6-b987-486c-80bc-547ffa448560: moving from state init -> state preparing >Thread-15911538::DEBUG::2013-11-19 21:36:40,730::task::495::TaskManager.Task::(_debug) Task 67fe47f6-b987-486c-80bc-547ffa448560: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000355958938599', 'lastCheck': 1384896992.1142321, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146102905273', 'lastCheck': 1384896993.6575191, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896999.775434, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00046706199646', 'lastCheck': 1384896990.5561421, 'valid': True, 'code': 0}} >Thread-15911538::DEBUG::2013-11-19 21:36:40,738::task::495::TaskManager.Task::(_debug) Task 67fe47f6-b987-486c-80bc-547ffa448560: moving from state preparing -> state finished >Thread-15911538::DEBUG::2013-11-19 21:36:40,738::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911538::DEBUG::2013-11-19 21:36:40,739::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911538::DEBUG::2013-11-19 21:36:40,740::task::495::TaskManager.Task::(_debug) Task 67fe47f6-b987-486c-80bc-547ffa448560: ref 0 aborting False >Thread-15911538::INFO::2013-11-19 21:36:40,740::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000355958938599', 'lastCheck': 1384896992.1142321, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146102905273', 'lastCheck': 1384896993.6575191, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384896999.775434, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00046706199646', 'lastCheck': 1384896990.5561421, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:36:42,645::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:43,192::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0703361 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:36:45,207::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:45,749::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0698755 s, 14.7 MB/s\n'; <rc> = 0 >Thread-15911542::DEBUG::2013-11-19 21:36:45,900::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911542::INFO::2013-11-19 21:36:45,902::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911542::DEBUG::2013-11-19 21:36:45,902::task::495::TaskManager.Task::(_debug) Task d81a27e7-0167-44de-8dd3-226c936aee4d: moving from state init -> state preparing >Thread-15911542::DEBUG::2013-11-19 21:36:45,903::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911542::DEBUG::2013-11-19 21:36:45,904::task::495::TaskManager.Task::(_debug) Task d81a27e7-0167-44de-8dd3-226c936aee4d: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911542::DEBUG::2013-11-19 21:36:45,904::task::495::TaskManager.Task::(_debug) Task d81a27e7-0167-44de-8dd3-226c936aee4d: moving from state preparing -> state finished >Thread-15911542::DEBUG::2013-11-19 21:36:45,905::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911542::DEBUG::2013-11-19 21:36:45,906::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911542::DEBUG::2013-11-19 21:36:45,906::task::495::TaskManager.Task::(_debug) Task d81a27e7-0167-44de-8dd3-226c936aee4d: ref 0 aborting False >Thread-15911542::INFO::2013-11-19 21:36:45,907::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911543::DEBUG::2013-11-19 21:36:45,913::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911543::INFO::2013-11-19 21:36:45,913::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911543::DEBUG::2013-11-19 21:36:45,914::task::495::TaskManager.Task::(_debug) Task a0d33a37-467c-44ff-989d-c1d48a5e0d81: moving from state init -> state preparing >Thread-15911543::DEBUG::2013-11-19 21:36:45,915::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`4c9af96e-5c06-4ec5-ab29-5dbd9756a261`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911543::DEBUG::2013-11-19 21:36:45,916::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911543::DEBUG::2013-11-19 21:36:45,917::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911543::DEBUG::2013-11-19 21:36:45,917::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`4c9af96e-5c06-4ec5-ab29-5dbd9756a261`::Granted request >Thread-15911543::DEBUG::2013-11-19 21:36:45,918::task::495::TaskManager.Task::(_debug) Task a0d33a37-467c-44ff-989d-c1d48a5e0d81: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911543::DEBUG::2013-11-19 21:36:45,919::task::495::TaskManager.Task::(_debug) Task a0d33a37-467c-44ff-989d-c1d48a5e0d81: ref 1 aborting False >Thread-15911543::WARNING::2013-11-19 21:36:45,921::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911543::DEBUG::2013-11-19 21:36:45,922::task::495::TaskManager.Task::(_debug) Task a0d33a37-467c-44ff-989d-c1d48a5e0d81: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911543::DEBUG::2013-11-19 21:36:45,923::task::495::TaskManager.Task::(_debug) Task a0d33a37-467c-44ff-989d-c1d48a5e0d81: moving from state preparing -> state finished >Thread-15911543::DEBUG::2013-11-19 21:36:45,924::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911543::DEBUG::2013-11-19 21:36:45,924::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911543::DEBUG::2013-11-19 21:36:45,925::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911543::DEBUG::2013-11-19 21:36:45,926::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911543::DEBUG::2013-11-19 21:36:45,926::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911543::DEBUG::2013-11-19 21:36:45,927::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911543::DEBUG::2013-11-19 21:36:45,927::task::495::TaskManager.Task::(_debug) Task a0d33a37-467c-44ff-989d-c1d48a5e0d81: ref 0 aborting False >Thread-15911543::INFO::2013-11-19 21:36:45,928::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:36:47,764::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:48,320::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0703347 s, 14.6 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:36:50,004::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:36:50,335::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:50,863::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0702332 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15911546::INFO::2013-11-19 21:36:50,919::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911546::DEBUG::2013-11-19 21:36:50,920::task::495::TaskManager.Task::(_debug) Task 02c78b33-c7c2-46c9-b36e-133f539a51ea: moving from state init -> state preparing >Thread-15911546::DEBUG::2013-11-19 21:36:50,921::task::495::TaskManager.Task::(_debug) Task 02c78b33-c7c2-46c9-b36e-133f539a51ea: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000331163406372', 'lastCheck': 1384897002.3182349, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147199630737', 'lastCheck': 1384897003.879705, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897010.0051179, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000424861907959', 'lastCheck': 1384897000.7737031, 'valid': True, 'code': 0}} >Thread-15911546::DEBUG::2013-11-19 21:36:50,922::task::495::TaskManager.Task::(_debug) Task 02c78b33-c7c2-46c9-b36e-133f539a51ea: moving from state preparing -> state finished >Thread-15911546::DEBUG::2013-11-19 21:36:50,923::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911546::DEBUG::2013-11-19 21:36:50,923::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911546::DEBUG::2013-11-19 21:36:50,924::task::495::TaskManager.Task::(_debug) Task 02c78b33-c7c2-46c9-b36e-133f539a51ea: ref 0 aborting False >Thread-15911546::INFO::2013-11-19 21:36:50,929::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000331163406372', 'lastCheck': 1384897002.3182349, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147199630737', 'lastCheck': 1384897003.879705, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897010.0051179, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000424861907959', 'lastCheck': 1384897000.7737031, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:36:52,878::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:53,399::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0691949 s, 14.8 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:36:55,414::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:55,962::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0712482 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15911550::DEBUG::2013-11-19 21:36:56,032::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911550::INFO::2013-11-19 21:36:56,034::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911550::DEBUG::2013-11-19 21:36:56,034::task::495::TaskManager.Task::(_debug) Task 216b2af3-11bf-4b2e-b10c-1c5060835987: moving from state init -> state preparing >Thread-15911550::DEBUG::2013-11-19 21:36:56,035::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911550::DEBUG::2013-11-19 21:36:56,036::task::495::TaskManager.Task::(_debug) Task 216b2af3-11bf-4b2e-b10c-1c5060835987: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911550::DEBUG::2013-11-19 21:36:56,036::task::495::TaskManager.Task::(_debug) Task 216b2af3-11bf-4b2e-b10c-1c5060835987: moving from state preparing -> state finished >Thread-15911550::DEBUG::2013-11-19 21:36:56,037::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911550::DEBUG::2013-11-19 21:36:56,038::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911550::DEBUG::2013-11-19 21:36:56,038::task::495::TaskManager.Task::(_debug) Task 216b2af3-11bf-4b2e-b10c-1c5060835987: ref 0 aborting False >Thread-15911550::INFO::2013-11-19 21:36:56,039::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911551::DEBUG::2013-11-19 21:36:56,051::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911551::INFO::2013-11-19 21:36:56,052::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911551::DEBUG::2013-11-19 21:36:56,053::task::495::TaskManager.Task::(_debug) Task c3d2852b-a664-419d-8c59-100f16e74990: moving from state init -> state preparing >Thread-15911551::DEBUG::2013-11-19 21:36:56,054::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`96615427-72a9-4bee-9dc1-d1670993ae33`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911551::DEBUG::2013-11-19 21:36:56,055::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911551::DEBUG::2013-11-19 21:36:56,055::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911551::DEBUG::2013-11-19 21:36:56,056::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`96615427-72a9-4bee-9dc1-d1670993ae33`::Granted request >Thread-15911551::DEBUG::2013-11-19 21:36:56,057::task::495::TaskManager.Task::(_debug) Task c3d2852b-a664-419d-8c59-100f16e74990: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911551::DEBUG::2013-11-19 21:36:56,058::task::495::TaskManager.Task::(_debug) Task c3d2852b-a664-419d-8c59-100f16e74990: ref 1 aborting False >Thread-15911551::WARNING::2013-11-19 21:36:56,060::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911551::DEBUG::2013-11-19 21:36:56,061::task::495::TaskManager.Task::(_debug) Task c3d2852b-a664-419d-8c59-100f16e74990: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911551::DEBUG::2013-11-19 21:36:56,062::task::495::TaskManager.Task::(_debug) Task c3d2852b-a664-419d-8c59-100f16e74990: moving from state preparing -> state finished >Thread-15911551::DEBUG::2013-11-19 21:36:56,062::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911551::DEBUG::2013-11-19 21:36:56,063::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911551::DEBUG::2013-11-19 21:36:56,063::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911551::DEBUG::2013-11-19 21:36:56,064::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911551::DEBUG::2013-11-19 21:36:56,065::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911551::DEBUG::2013-11-19 21:36:56,065::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911551::DEBUG::2013-11-19 21:36:56,066::task::495::TaskManager.Task::(_debug) Task c3d2852b-a664-419d-8c59-100f16e74990: ref 0 aborting False >Thread-15911551::INFO::2013-11-19 21:36:56,067::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:36:57,977::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:36:58,501::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0700069 s, 14.6 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:37:00,190::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:37:00,517::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:01,068::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0703909 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15911554::INFO::2013-11-19 21:37:01,151::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911554::DEBUG::2013-11-19 21:37:01,152::task::495::TaskManager.Task::(_debug) Task eb20e2f6-3f89-4a36-8c4b-c4165bf2b7cc: moving from state init -> state preparing >Thread-15911554::DEBUG::2013-11-19 21:37:01,153::task::495::TaskManager.Task::(_debug) Task eb20e2f6-3f89-4a36-8c4b-c4165bf2b7cc: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000578165054321', 'lastCheck': 1384897012.5628331, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00189709663391', 'lastCheck': 1384897014.113179, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897020.190887, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000512838363647', 'lastCheck': 1384897011.0029211, 'valid': True, 'code': 0}} >Thread-15911554::DEBUG::2013-11-19 21:37:01,162::task::495::TaskManager.Task::(_debug) Task eb20e2f6-3f89-4a36-8c4b-c4165bf2b7cc: moving from state preparing -> state finished >Thread-15911554::DEBUG::2013-11-19 21:37:01,163::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911554::DEBUG::2013-11-19 21:37:01,164::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911554::DEBUG::2013-11-19 21:37:01,164::task::495::TaskManager.Task::(_debug) Task eb20e2f6-3f89-4a36-8c4b-c4165bf2b7cc: ref 0 aborting False >Thread-15911554::INFO::2013-11-19 21:37:01,166::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000578165054321', 'lastCheck': 1384897012.5628331, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00189709663391', 'lastCheck': 1384897014.113179, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897020.190887, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000512838363647', 'lastCheck': 1384897011.0029211, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:37:03,083::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:03,614::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0706885 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:37:05,629::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:06,160::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.070228 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15911558::DEBUG::2013-11-19 21:37:06,239::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911558::INFO::2013-11-19 21:37:06,241::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911558::DEBUG::2013-11-19 21:37:06,241::task::495::TaskManager.Task::(_debug) Task 37d78de6-b775-4a4a-9daf-d9cc3216d1e5: moving from state init -> state preparing >Thread-15911558::DEBUG::2013-11-19 21:37:06,242::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911558::DEBUG::2013-11-19 21:37:06,243::task::495::TaskManager.Task::(_debug) Task 37d78de6-b775-4a4a-9daf-d9cc3216d1e5: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911558::DEBUG::2013-11-19 21:37:06,243::task::495::TaskManager.Task::(_debug) Task 37d78de6-b775-4a4a-9daf-d9cc3216d1e5: moving from state preparing -> state finished >Thread-15911558::DEBUG::2013-11-19 21:37:06,244::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911558::DEBUG::2013-11-19 21:37:06,244::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911558::DEBUG::2013-11-19 21:37:06,245::task::495::TaskManager.Task::(_debug) Task 37d78de6-b775-4a4a-9daf-d9cc3216d1e5: ref 0 aborting False >Thread-15911558::INFO::2013-11-19 21:37:06,246::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911559::DEBUG::2013-11-19 21:37:06,252::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911559::INFO::2013-11-19 21:37:06,253::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911559::DEBUG::2013-11-19 21:37:06,253::task::495::TaskManager.Task::(_debug) Task 1c6ebe12-eb8e-4591-bec6-5720ef196d0f: moving from state init -> state preparing >Thread-15911559::DEBUG::2013-11-19 21:37:06,254::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a69b1614-e0eb-4687-8787-9af00b9388b4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911559::DEBUG::2013-11-19 21:37:06,255::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911559::DEBUG::2013-11-19 21:37:06,256::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911559::DEBUG::2013-11-19 21:37:06,256::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a69b1614-e0eb-4687-8787-9af00b9388b4`::Granted request >Thread-15911559::DEBUG::2013-11-19 21:37:06,257::task::495::TaskManager.Task::(_debug) Task 1c6ebe12-eb8e-4591-bec6-5720ef196d0f: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911559::DEBUG::2013-11-19 21:37:06,258::task::495::TaskManager.Task::(_debug) Task 1c6ebe12-eb8e-4591-bec6-5720ef196d0f: ref 1 aborting False >Thread-15911559::WARNING::2013-11-19 21:37:06,260::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911559::DEBUG::2013-11-19 21:37:06,261::task::495::TaskManager.Task::(_debug) Task 1c6ebe12-eb8e-4591-bec6-5720ef196d0f: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911559::DEBUG::2013-11-19 21:37:06,262::task::495::TaskManager.Task::(_debug) Task 1c6ebe12-eb8e-4591-bec6-5720ef196d0f: moving from state preparing -> state finished >Thread-15911559::DEBUG::2013-11-19 21:37:06,263::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911559::DEBUG::2013-11-19 21:37:06,263::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911559::DEBUG::2013-11-19 21:37:06,264::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911559::DEBUG::2013-11-19 21:37:06,265::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911559::DEBUG::2013-11-19 21:37:06,265::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911559::DEBUG::2013-11-19 21:37:06,266::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911559::DEBUG::2013-11-19 21:37:06,266::task::495::TaskManager.Task::(_debug) Task 1c6ebe12-eb8e-4591-bec6-5720ef196d0f: ref 0 aborting False >Thread-15911559::INFO::2013-11-19 21:37:06,267::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:37:08,175::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:08,712::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.070183 s, 14.6 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:37:10,401::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:37:10,726::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:11,242::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0702783 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15911562::INFO::2013-11-19 21:37:11,374::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911562::DEBUG::2013-11-19 21:37:11,375::task::495::TaskManager.Task::(_debug) Task d88f5615-f4e8-45b1-971f-c8830f50c978: moving from state init -> state preparing >Thread-15911562::DEBUG::2013-11-19 21:37:11,375::task::495::TaskManager.Task::(_debug) Task d88f5615-f4e8-45b1-971f-c8830f50c978: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000396966934204', 'lastCheck': 1384897022.775394, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00134110450745', 'lastCheck': 1384897024.2985539, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897030.401911, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000432014465332', 'lastCheck': 1384897021.189374, 'valid': True, 'code': 0}} >Thread-15911562::DEBUG::2013-11-19 21:37:11,381::task::495::TaskManager.Task::(_debug) Task d88f5615-f4e8-45b1-971f-c8830f50c978: moving from state preparing -> state finished >Thread-15911562::DEBUG::2013-11-19 21:37:11,381::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911562::DEBUG::2013-11-19 21:37:11,382::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911562::DEBUG::2013-11-19 21:37:11,383::task::495::TaskManager.Task::(_debug) Task d88f5615-f4e8-45b1-971f-c8830f50c978: ref 0 aborting False >Thread-15911562::INFO::2013-11-19 21:37:11,386::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000396966934204', 'lastCheck': 1384897022.775394, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00134110450745', 'lastCheck': 1384897024.2985539, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897030.401911, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000432014465332', 'lastCheck': 1384897021.189374, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:37:13,257::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:13,778::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0693848 s, 14.8 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:37:15,793::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:16,335::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0702338 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15911566::DEBUG::2013-11-19 21:37:16,439::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911566::INFO::2013-11-19 21:37:16,440::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911566::DEBUG::2013-11-19 21:37:16,451::task::495::TaskManager.Task::(_debug) Task 522f552f-8eff-4d0b-9c6a-088723de4a40: moving from state init -> state preparing >Thread-15911566::DEBUG::2013-11-19 21:37:16,452::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911566::DEBUG::2013-11-19 21:37:16,453::task::495::TaskManager.Task::(_debug) Task 522f552f-8eff-4d0b-9c6a-088723de4a40: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911566::DEBUG::2013-11-19 21:37:16,453::task::495::TaskManager.Task::(_debug) Task 522f552f-8eff-4d0b-9c6a-088723de4a40: moving from state preparing -> state finished >Thread-15911566::DEBUG::2013-11-19 21:37:16,454::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911566::DEBUG::2013-11-19 21:37:16,454::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911566::DEBUG::2013-11-19 21:37:16,455::task::495::TaskManager.Task::(_debug) Task 522f552f-8eff-4d0b-9c6a-088723de4a40: ref 0 aborting False >Thread-15911566::INFO::2013-11-19 21:37:16,456::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911567::DEBUG::2013-11-19 21:37:16,461::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911567::INFO::2013-11-19 21:37:16,462::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911567::DEBUG::2013-11-19 21:37:16,463::task::495::TaskManager.Task::(_debug) Task 449c5bff-aab9-4d89-8f8c-8567f24e1ba0: moving from state init -> state preparing >Thread-15911567::DEBUG::2013-11-19 21:37:16,464::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f408f6a5-1a9e-4136-b031-8cc2670b97b7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911567::DEBUG::2013-11-19 21:37:16,465::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911567::DEBUG::2013-11-19 21:37:16,466::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911567::DEBUG::2013-11-19 21:37:16,466::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f408f6a5-1a9e-4136-b031-8cc2670b97b7`::Granted request >Thread-15911567::DEBUG::2013-11-19 21:37:16,467::task::495::TaskManager.Task::(_debug) Task 449c5bff-aab9-4d89-8f8c-8567f24e1ba0: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911567::DEBUG::2013-11-19 21:37:16,468::task::495::TaskManager.Task::(_debug) Task 449c5bff-aab9-4d89-8f8c-8567f24e1ba0: ref 1 aborting False >Thread-15911567::WARNING::2013-11-19 21:37:16,470::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911567::DEBUG::2013-11-19 21:37:16,471::task::495::TaskManager.Task::(_debug) Task 449c5bff-aab9-4d89-8f8c-8567f24e1ba0: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911567::DEBUG::2013-11-19 21:37:16,472::task::495::TaskManager.Task::(_debug) Task 449c5bff-aab9-4d89-8f8c-8567f24e1ba0: moving from state preparing -> state finished >Thread-15911567::DEBUG::2013-11-19 21:37:16,472::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911567::DEBUG::2013-11-19 21:37:16,473::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911567::DEBUG::2013-11-19 21:37:16,474::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911567::DEBUG::2013-11-19 21:37:16,474::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911567::DEBUG::2013-11-19 21:37:16,475::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911567::DEBUG::2013-11-19 21:37:16,475::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911567::DEBUG::2013-11-19 21:37:16,476::task::495::TaskManager.Task::(_debug) Task 449c5bff-aab9-4d89-8f8c-8567f24e1ba0: ref 0 aborting False >Thread-15911567::INFO::2013-11-19 21:37:16,477::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:37:18,350::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:18,853::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0783585 s, 13.1 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:37:20,559::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:37:20,864::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:21,329::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0675208 s, 15.2 MB/s\n'; <rc> = 0 >Thread-15911570::INFO::2013-11-19 21:37:21,546::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911570::DEBUG::2013-11-19 21:37:21,547::task::495::TaskManager.Task::(_debug) Task a50a5412-590a-4580-873d-d5c6eadb8675: moving from state init -> state preparing >Thread-42840::DEBUG::2013-11-19 21:37:21,560::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15911570::DEBUG::2013-11-19 21:37:21,561::task::495::TaskManager.Task::(_debug) Task a50a5412-590a-4580-873d-d5c6eadb8675: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000391006469727', 'lastCheck': 1384897032.933435, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00149297714233', 'lastCheck': 1384897034.484406, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897040.5614519, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000517129898071', 'lastCheck': 1384897031.4028809, 'valid': True, 'code': 0}} >Thread-15911570::DEBUG::2013-11-19 21:37:21,813::task::495::TaskManager.Task::(_debug) Task a50a5412-590a-4580-873d-d5c6eadb8675: moving from state preparing -> state finished >Thread-15911570::DEBUG::2013-11-19 21:37:21,815::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911570::DEBUG::2013-11-19 21:37:21,816::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911570::DEBUG::2013-11-19 21:37:21,817::task::495::TaskManager.Task::(_debug) Task a50a5412-590a-4580-873d-d5c6eadb8675: ref 0 aborting False >Thread-15911570::INFO::2013-11-19 21:37:21,817::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000391006469727', 'lastCheck': 1384897032.933435, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00149297714233', 'lastCheck': 1384897034.484406, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897040.5614519, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000517129898071', 'lastCheck': 1384897031.4028809, 'valid': True, 'code': 0}} >Thread-42840::DEBUG::2013-11-19 21:37:22,200::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:37:23,345::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:23,870::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0684624 s, 15.0 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:37:25,885::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:26,404::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0694182 s, 14.8 MB/s\n'; <rc> = 0 >Thread-15911574::DEBUG::2013-11-19 21:37:26,653::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911574::INFO::2013-11-19 21:37:26,655::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911574::DEBUG::2013-11-19 21:37:26,664::task::495::TaskManager.Task::(_debug) Task da35be39-100f-4434-81c4-92b9569351bb: moving from state init -> state preparing >Thread-15911574::DEBUG::2013-11-19 21:37:26,665::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911574::DEBUG::2013-11-19 21:37:26,665::task::495::TaskManager.Task::(_debug) Task da35be39-100f-4434-81c4-92b9569351bb: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911574::DEBUG::2013-11-19 21:37:26,666::task::495::TaskManager.Task::(_debug) Task da35be39-100f-4434-81c4-92b9569351bb: moving from state preparing -> state finished >Thread-15911574::DEBUG::2013-11-19 21:37:26,667::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911574::DEBUG::2013-11-19 21:37:26,667::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911574::DEBUG::2013-11-19 21:37:26,668::task::495::TaskManager.Task::(_debug) Task da35be39-100f-4434-81c4-92b9569351bb: ref 0 aborting False >Thread-15911574::INFO::2013-11-19 21:37:26,668::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911575::DEBUG::2013-11-19 21:37:26,674::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911575::INFO::2013-11-19 21:37:26,675::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911575::DEBUG::2013-11-19 21:37:26,676::task::495::TaskManager.Task::(_debug) Task a300ab53-7971-48d4-9c88-2b0aa277f707: moving from state init -> state preparing >Thread-15911575::DEBUG::2013-11-19 21:37:26,677::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a98774e2-1eac-49e8-8cfd-44507f48a1a5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911575::DEBUG::2013-11-19 21:37:26,677::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911575::DEBUG::2013-11-19 21:37:26,678::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911575::DEBUG::2013-11-19 21:37:26,679::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a98774e2-1eac-49e8-8cfd-44507f48a1a5`::Granted request >Thread-15911575::DEBUG::2013-11-19 21:37:26,680::task::495::TaskManager.Task::(_debug) Task a300ab53-7971-48d4-9c88-2b0aa277f707: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911575::DEBUG::2013-11-19 21:37:26,680::task::495::TaskManager.Task::(_debug) Task a300ab53-7971-48d4-9c88-2b0aa277f707: ref 1 aborting False >Thread-15911575::WARNING::2013-11-19 21:37:26,682::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911575::DEBUG::2013-11-19 21:37:26,683::task::495::TaskManager.Task::(_debug) Task a300ab53-7971-48d4-9c88-2b0aa277f707: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911575::DEBUG::2013-11-19 21:37:26,684::task::495::TaskManager.Task::(_debug) Task a300ab53-7971-48d4-9c88-2b0aa277f707: moving from state preparing -> state finished >Thread-15911575::DEBUG::2013-11-19 21:37:26,685::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911575::DEBUG::2013-11-19 21:37:26,686::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911575::DEBUG::2013-11-19 21:37:26,686::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911575::DEBUG::2013-11-19 21:37:26,687::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911575::DEBUG::2013-11-19 21:37:26,687::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911575::DEBUG::2013-11-19 21:37:26,688::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911575::DEBUG::2013-11-19 21:37:26,689::task::495::TaskManager.Task::(_debug) Task a300ab53-7971-48d4-9c88-2b0aa277f707: ref 0 aborting False >Thread-15911575::INFO::2013-11-19 21:37:26,689::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:37:28,419::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:28,952::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0705261 s, 14.5 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:37:30,641::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:37:30,967::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:31,497::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0700654 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15911578::INFO::2013-11-19 21:37:32,144::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911578::DEBUG::2013-11-19 21:37:32,145::task::495::TaskManager.Task::(_debug) Task 9bf2d344-b6d0-4c66-83d9-e86d99488c96: moving from state init -> state preparing >Thread-15911578::DEBUG::2013-11-19 21:37:32,145::task::495::TaskManager.Task::(_debug) Task 9bf2d344-b6d0-4c66-83d9-e86d99488c96: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000404119491577', 'lastCheck': 1384897043.0767901, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154399871826', 'lastCheck': 1384897044.565289, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897050.6426411, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000310897827148', 'lastCheck': 1384897042.2026429, 'valid': True, 'code': 0}} >Thread-15911578::DEBUG::2013-11-19 21:37:32,151::task::495::TaskManager.Task::(_debug) Task 9bf2d344-b6d0-4c66-83d9-e86d99488c96: moving from state preparing -> state finished >Thread-15911578::DEBUG::2013-11-19 21:37:32,151::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911578::DEBUG::2013-11-19 21:37:32,152::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911578::DEBUG::2013-11-19 21:37:32,153::task::495::TaskManager.Task::(_debug) Task 9bf2d344-b6d0-4c66-83d9-e86d99488c96: ref 0 aborting False >Thread-15911578::INFO::2013-11-19 21:37:32,153::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000404119491577', 'lastCheck': 1384897043.0767901, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154399871826', 'lastCheck': 1384897044.565289, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897050.6426411, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000310897827148', 'lastCheck': 1384897042.2026429, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:37:33,511::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:34,057::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0703092 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:37:36,072::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:36,600::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0706245 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15911582::DEBUG::2013-11-19 21:37:36,815::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911582::INFO::2013-11-19 21:37:36,820::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911582::DEBUG::2013-11-19 21:37:36,821::task::495::TaskManager.Task::(_debug) Task 5cfd2524-6b2c-4e58-8854-44d6dc212df8: moving from state init -> state preparing >Thread-15911582::DEBUG::2013-11-19 21:37:36,822::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911582::DEBUG::2013-11-19 21:37:36,823::task::495::TaskManager.Task::(_debug) Task 5cfd2524-6b2c-4e58-8854-44d6dc212df8: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911582::DEBUG::2013-11-19 21:37:36,823::task::495::TaskManager.Task::(_debug) Task 5cfd2524-6b2c-4e58-8854-44d6dc212df8: moving from state preparing -> state finished >Thread-15911582::DEBUG::2013-11-19 21:37:36,824::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911582::DEBUG::2013-11-19 21:37:36,824::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911582::DEBUG::2013-11-19 21:37:36,825::task::495::TaskManager.Task::(_debug) Task 5cfd2524-6b2c-4e58-8854-44d6dc212df8: ref 0 aborting False >Thread-15911582::INFO::2013-11-19 21:37:36,825::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911583::DEBUG::2013-11-19 21:37:36,833::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911583::INFO::2013-11-19 21:37:36,834::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911583::DEBUG::2013-11-19 21:37:36,834::task::495::TaskManager.Task::(_debug) Task d2caa9fd-d879-43bc-9770-9aef11f0ac48: moving from state init -> state preparing >Thread-15911583::DEBUG::2013-11-19 21:37:36,836::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`66a70e0c-0e21-41a6-b426-7b2aec8e0914`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911583::DEBUG::2013-11-19 21:37:36,836::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911583::DEBUG::2013-11-19 21:37:36,837::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911583::DEBUG::2013-11-19 21:37:36,838::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`66a70e0c-0e21-41a6-b426-7b2aec8e0914`::Granted request >Thread-15911583::DEBUG::2013-11-19 21:37:36,839::task::495::TaskManager.Task::(_debug) Task d2caa9fd-d879-43bc-9770-9aef11f0ac48: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911583::DEBUG::2013-11-19 21:37:36,839::task::495::TaskManager.Task::(_debug) Task d2caa9fd-d879-43bc-9770-9aef11f0ac48: ref 1 aborting False >Thread-15911583::WARNING::2013-11-19 21:37:36,842::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911583::DEBUG::2013-11-19 21:37:36,843::task::495::TaskManager.Task::(_debug) Task d2caa9fd-d879-43bc-9770-9aef11f0ac48: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911583::DEBUG::2013-11-19 21:37:36,844::task::495::TaskManager.Task::(_debug) Task d2caa9fd-d879-43bc-9770-9aef11f0ac48: moving from state preparing -> state finished >Thread-15911583::DEBUG::2013-11-19 21:37:36,844::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911583::DEBUG::2013-11-19 21:37:36,845::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911583::DEBUG::2013-11-19 21:37:36,845::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911583::DEBUG::2013-11-19 21:37:36,846::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911583::DEBUG::2013-11-19 21:37:36,847::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911583::DEBUG::2013-11-19 21:37:36,847::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911583::DEBUG::2013-11-19 21:37:36,848::task::495::TaskManager.Task::(_debug) Task d2caa9fd-d879-43bc-9770-9aef11f0ac48: ref 0 aborting False >Thread-15911583::INFO::2013-11-19 21:37:36,849::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:37:38,615::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:39,125::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0704312 s, 14.5 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:37:40,818::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:37:41,140::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:41,703::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0899135 s, 11.4 MB/s\n'; <rc> = 0 >Thread-15911586::INFO::2013-11-19 21:37:42,344::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911586::DEBUG::2013-11-19 21:37:42,345::task::495::TaskManager.Task::(_debug) Task ca410588-02c4-4970-8a7c-dd222e67f528: moving from state init -> state preparing >Thread-15911586::DEBUG::2013-11-19 21:37:42,345::task::495::TaskManager.Task::(_debug) Task ca410588-02c4-4970-8a7c-dd222e67f528: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000373840332031', 'lastCheck': 1384897053.181349, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00166392326355', 'lastCheck': 1384897054.7461939, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897060.8196089, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000485897064209', 'lastCheck': 1384897052.2140279, 'valid': True, 'code': 0}} >Thread-15911586::DEBUG::2013-11-19 21:37:42,357::task::495::TaskManager.Task::(_debug) Task ca410588-02c4-4970-8a7c-dd222e67f528: moving from state preparing -> state finished >Thread-15911586::DEBUG::2013-11-19 21:37:42,357::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911586::DEBUG::2013-11-19 21:37:42,358::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911586::DEBUG::2013-11-19 21:37:42,358::task::495::TaskManager.Task::(_debug) Task ca410588-02c4-4970-8a7c-dd222e67f528: ref 0 aborting False >Thread-15911586::INFO::2013-11-19 21:37:42,359::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000373840332031', 'lastCheck': 1384897053.181349, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00166392326355', 'lastCheck': 1384897054.7461939, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897060.8196089, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000485897064209', 'lastCheck': 1384897052.2140279, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:37:43,719::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:44,236::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.068329 s, 15.0 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:37:46,250::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:46,767::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0734635 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15911590::DEBUG::2013-11-19 21:37:46,993::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911590::INFO::2013-11-19 21:37:46,995::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911590::DEBUG::2013-11-19 21:37:46,996::task::495::TaskManager.Task::(_debug) Task d49d2336-24f8-4a8f-9dd4-06020f160542: moving from state init -> state preparing >Thread-15911590::DEBUG::2013-11-19 21:37:46,997::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911590::DEBUG::2013-11-19 21:37:46,997::task::495::TaskManager.Task::(_debug) Task d49d2336-24f8-4a8f-9dd4-06020f160542: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911590::DEBUG::2013-11-19 21:37:46,998::task::495::TaskManager.Task::(_debug) Task d49d2336-24f8-4a8f-9dd4-06020f160542: moving from state preparing -> state finished >Thread-15911590::DEBUG::2013-11-19 21:37:46,998::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911590::DEBUG::2013-11-19 21:37:46,999::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911590::DEBUG::2013-11-19 21:37:47,000::task::495::TaskManager.Task::(_debug) Task d49d2336-24f8-4a8f-9dd4-06020f160542: ref 0 aborting False >Thread-15911590::INFO::2013-11-19 21:37:47,000::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911591::DEBUG::2013-11-19 21:37:47,006::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911591::INFO::2013-11-19 21:37:47,007::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911591::DEBUG::2013-11-19 21:37:47,008::task::495::TaskManager.Task::(_debug) Task df4594fa-432a-4fe5-9f22-94572ab1562a: moving from state init -> state preparing >Thread-15911591::DEBUG::2013-11-19 21:37:47,009::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2674479d-4651-4689-bfa1-14f7fca26d68`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911591::DEBUG::2013-11-19 21:37:47,010::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911591::DEBUG::2013-11-19 21:37:47,011::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911591::DEBUG::2013-11-19 21:37:47,011::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2674479d-4651-4689-bfa1-14f7fca26d68`::Granted request >Thread-15911591::DEBUG::2013-11-19 21:37:47,012::task::495::TaskManager.Task::(_debug) Task df4594fa-432a-4fe5-9f22-94572ab1562a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911591::DEBUG::2013-11-19 21:37:47,013::task::495::TaskManager.Task::(_debug) Task df4594fa-432a-4fe5-9f22-94572ab1562a: ref 1 aborting False >Thread-15911591::WARNING::2013-11-19 21:37:47,015::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911591::DEBUG::2013-11-19 21:37:47,016::task::495::TaskManager.Task::(_debug) Task df4594fa-432a-4fe5-9f22-94572ab1562a: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911591::DEBUG::2013-11-19 21:37:47,017::task::495::TaskManager.Task::(_debug) Task df4594fa-432a-4fe5-9f22-94572ab1562a: moving from state preparing -> state finished >Thread-15911591::DEBUG::2013-11-19 21:37:47,017::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911591::DEBUG::2013-11-19 21:37:47,018::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911591::DEBUG::2013-11-19 21:37:47,019::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911591::DEBUG::2013-11-19 21:37:47,019::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911591::DEBUG::2013-11-19 21:37:47,020::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911591::DEBUG::2013-11-19 21:37:47,020::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911591::DEBUG::2013-11-19 21:37:47,021::task::495::TaskManager.Task::(_debug) Task df4594fa-432a-4fe5-9f22-94572ab1562a: ref 0 aborting False >Thread-15911591::INFO::2013-11-19 21:37:47,022::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:37:48,783::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:49,321::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.075561 s, 13.6 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:37:51,011::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:37:51,336::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:51,865::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.070863 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15911594::INFO::2013-11-19 21:37:52,484::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911594::DEBUG::2013-11-19 21:37:52,485::task::495::TaskManager.Task::(_debug) Task fb83372f-cf89-462c-83ae-c794cb19ff66: moving from state init -> state preparing >Thread-15911594::DEBUG::2013-11-19 21:37:52,486::task::495::TaskManager.Task::(_debug) Task fb83372f-cf89-462c-83ae-c794cb19ff66: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000411987304688', 'lastCheck': 1384897063.376287, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150299072266', 'lastCheck': 1384897064.9300129, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897071.0119519, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000520944595337', 'lastCheck': 1384897062.3721781, 'valid': True, 'code': 0}} >Thread-15911594::DEBUG::2013-11-19 21:37:52,487::task::495::TaskManager.Task::(_debug) Task fb83372f-cf89-462c-83ae-c794cb19ff66: moving from state preparing -> state finished >Thread-15911594::DEBUG::2013-11-19 21:37:52,487::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911594::DEBUG::2013-11-19 21:37:52,488::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911594::DEBUG::2013-11-19 21:37:52,488::task::495::TaskManager.Task::(_debug) Task fb83372f-cf89-462c-83ae-c794cb19ff66: ref 0 aborting False >Thread-15911594::INFO::2013-11-19 21:37:52,496::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000411987304688', 'lastCheck': 1384897063.376287, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150299072266', 'lastCheck': 1384897064.9300129, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897071.0119519, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000520944595337', 'lastCheck': 1384897062.3721781, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:37:53,880::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:54,428::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917731 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:37:56,443::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:56,980::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0721752 s, 14.2 MB/s\n'; <rc> = 0 >Thread-15911598::DEBUG::2013-11-19 21:37:57,161::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911598::INFO::2013-11-19 21:37:57,162::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911598::DEBUG::2013-11-19 21:37:57,163::task::495::TaskManager.Task::(_debug) Task bfc372a4-21ab-4174-bc33-ed933221f3f7: moving from state init -> state preparing >Thread-15911598::DEBUG::2013-11-19 21:37:57,164::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911598::DEBUG::2013-11-19 21:37:57,164::task::495::TaskManager.Task::(_debug) Task bfc372a4-21ab-4174-bc33-ed933221f3f7: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911598::DEBUG::2013-11-19 21:37:57,165::task::495::TaskManager.Task::(_debug) Task bfc372a4-21ab-4174-bc33-ed933221f3f7: moving from state preparing -> state finished >Thread-15911598::DEBUG::2013-11-19 21:37:57,165::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911598::DEBUG::2013-11-19 21:37:57,166::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911598::DEBUG::2013-11-19 21:37:57,167::task::495::TaskManager.Task::(_debug) Task bfc372a4-21ab-4174-bc33-ed933221f3f7: ref 0 aborting False >Thread-15911598::INFO::2013-11-19 21:37:57,167::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911599::DEBUG::2013-11-19 21:37:57,173::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911599::INFO::2013-11-19 21:37:57,174::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911599::DEBUG::2013-11-19 21:37:57,174::task::495::TaskManager.Task::(_debug) Task 44a5ef15-d697-4373-bd4e-7d6860b518e6: moving from state init -> state preparing >Thread-15911599::DEBUG::2013-11-19 21:37:57,176::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d280ddae-7682-446a-8ef1-35d68a4dd7f4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911599::DEBUG::2013-11-19 21:37:57,176::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911599::DEBUG::2013-11-19 21:37:57,177::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911599::DEBUG::2013-11-19 21:37:57,178::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d280ddae-7682-446a-8ef1-35d68a4dd7f4`::Granted request >Thread-15911599::DEBUG::2013-11-19 21:37:57,179::task::495::TaskManager.Task::(_debug) Task 44a5ef15-d697-4373-bd4e-7d6860b518e6: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911599::DEBUG::2013-11-19 21:37:57,179::task::495::TaskManager.Task::(_debug) Task 44a5ef15-d697-4373-bd4e-7d6860b518e6: ref 1 aborting False >Thread-15911599::WARNING::2013-11-19 21:37:57,181::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911599::DEBUG::2013-11-19 21:37:57,182::task::495::TaskManager.Task::(_debug) Task 44a5ef15-d697-4373-bd4e-7d6860b518e6: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911599::DEBUG::2013-11-19 21:37:57,183::task::495::TaskManager.Task::(_debug) Task 44a5ef15-d697-4373-bd4e-7d6860b518e6: moving from state preparing -> state finished >Thread-15911599::DEBUG::2013-11-19 21:37:57,184::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911599::DEBUG::2013-11-19 21:37:57,184::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911599::DEBUG::2013-11-19 21:37:57,185::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911599::DEBUG::2013-11-19 21:37:57,186::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911599::DEBUG::2013-11-19 21:37:57,186::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911599::DEBUG::2013-11-19 21:37:57,187::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911599::DEBUG::2013-11-19 21:37:57,187::task::495::TaskManager.Task::(_debug) Task 44a5ef15-d697-4373-bd4e-7d6860b518e6: ref 0 aborting False >Thread-15911599::INFO::2013-11-19 21:37:57,188::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:37:58,996::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:37:59,506::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0693539 s, 14.8 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:38:01,206::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:38:01,521::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:02,065::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0718432 s, 14.3 MB/s\n'; <rc> = 0 >Thread-15911602::INFO::2013-11-19 21:38:02,724::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911602::DEBUG::2013-11-19 21:38:02,725::task::495::TaskManager.Task::(_debug) Task 6707ec49-aaa9-4f32-aa29-00b56277220f: moving from state init -> state preparing >Thread-15911602::DEBUG::2013-11-19 21:38:02,725::task::495::TaskManager.Task::(_debug) Task 6707ec49-aaa9-4f32-aa29-00b56277220f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000331878662109', 'lastCheck': 1384897073.553632, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00213408470154', 'lastCheck': 1384897075.1159239, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897081.206986, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000448942184448', 'lastCheck': 1384897072.550302, 'valid': True, 'code': 0}} >Thread-15911602::DEBUG::2013-11-19 21:38:02,736::task::495::TaskManager.Task::(_debug) Task 6707ec49-aaa9-4f32-aa29-00b56277220f: moving from state preparing -> state finished >Thread-15911602::DEBUG::2013-11-19 21:38:02,736::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911602::DEBUG::2013-11-19 21:38:02,737::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911602::DEBUG::2013-11-19 21:38:02,737::task::495::TaskManager.Task::(_debug) Task 6707ec49-aaa9-4f32-aa29-00b56277220f: ref 0 aborting False >Thread-15911602::INFO::2013-11-19 21:38:02,738::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000331878662109', 'lastCheck': 1384897073.553632, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00213408470154', 'lastCheck': 1384897075.1159239, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897081.206986, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000448942184448', 'lastCheck': 1384897072.550302, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:38:04,081::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:04,616::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0699461 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:38:06,631::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:07,165::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0757589 s, 13.5 MB/s\n'; <rc> = 0 >Thread-15911606::DEBUG::2013-11-19 21:38:07,314::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911606::INFO::2013-11-19 21:38:07,316::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911606::DEBUG::2013-11-19 21:38:07,317::task::495::TaskManager.Task::(_debug) Task 2f1ce627-5979-4d39-9ec4-254e03a260ce: moving from state init -> state preparing >Thread-15911606::DEBUG::2013-11-19 21:38:07,318::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911606::DEBUG::2013-11-19 21:38:07,320::task::495::TaskManager.Task::(_debug) Task 2f1ce627-5979-4d39-9ec4-254e03a260ce: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911606::DEBUG::2013-11-19 21:38:07,326::task::495::TaskManager.Task::(_debug) Task 2f1ce627-5979-4d39-9ec4-254e03a260ce: moving from state preparing -> state finished >Thread-15911606::DEBUG::2013-11-19 21:38:07,329::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911606::DEBUG::2013-11-19 21:38:07,331::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911606::DEBUG::2013-11-19 21:38:07,332::task::495::TaskManager.Task::(_debug) Task 2f1ce627-5979-4d39-9ec4-254e03a260ce: ref 0 aborting False >Thread-15911606::INFO::2013-11-19 21:38:07,333::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911607::DEBUG::2013-11-19 21:38:07,346::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911607::INFO::2013-11-19 21:38:07,347::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911607::DEBUG::2013-11-19 21:38:07,347::task::495::TaskManager.Task::(_debug) Task 5373b3d9-9178-481e-a32d-6fb8fc324e05: moving from state init -> state preparing >Thread-15911607::DEBUG::2013-11-19 21:38:07,349::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`97eb92f9-18fa-408c-b1fc-8b51c36a7a1f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911607::DEBUG::2013-11-19 21:38:07,349::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911607::DEBUG::2013-11-19 21:38:07,350::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911607::DEBUG::2013-11-19 21:38:07,351::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`97eb92f9-18fa-408c-b1fc-8b51c36a7a1f`::Granted request >Thread-15911607::DEBUG::2013-11-19 21:38:07,352::task::495::TaskManager.Task::(_debug) Task 5373b3d9-9178-481e-a32d-6fb8fc324e05: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911607::DEBUG::2013-11-19 21:38:07,352::task::495::TaskManager.Task::(_debug) Task 5373b3d9-9178-481e-a32d-6fb8fc324e05: ref 1 aborting False >Thread-15911607::WARNING::2013-11-19 21:38:07,354::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911607::DEBUG::2013-11-19 21:38:07,355::task::495::TaskManager.Task::(_debug) Task 5373b3d9-9178-481e-a32d-6fb8fc324e05: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911607::DEBUG::2013-11-19 21:38:07,356::task::495::TaskManager.Task::(_debug) Task 5373b3d9-9178-481e-a32d-6fb8fc324e05: moving from state preparing -> state finished >Thread-15911607::DEBUG::2013-11-19 21:38:07,357::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911607::DEBUG::2013-11-19 21:38:07,357::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911607::DEBUG::2013-11-19 21:38:07,358::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911607::DEBUG::2013-11-19 21:38:07,359::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911607::DEBUG::2013-11-19 21:38:07,359::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911607::DEBUG::2013-11-19 21:38:07,360::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911607::DEBUG::2013-11-19 21:38:07,361::task::495::TaskManager.Task::(_debug) Task 5373b3d9-9178-481e-a32d-6fb8fc324e05: ref 0 aborting False >Thread-15911607::INFO::2013-11-19 21:38:07,361::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:38:09,187::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:09,725::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0706175 s, 14.5 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:38:11,410::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:38:11,741::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:12,263::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0702749 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15911610::INFO::2013-11-19 21:38:12,911::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911610::DEBUG::2013-11-19 21:38:12,912::task::495::TaskManager.Task::(_debug) Task 8eb48158-64da-4aac-a6b0-8f07b324fe1b: moving from state init -> state preparing >Thread-15911610::DEBUG::2013-11-19 21:38:12,913::task::495::TaskManager.Task::(_debug) Task 8eb48158-64da-4aac-a6b0-8f07b324fe1b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000566005706787', 'lastCheck': 1384897083.752562, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152397155762', 'lastCheck': 1384897085.310354, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897091.4113779, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000468969345093', 'lastCheck': 1384897082.7467189, 'valid': True, 'code': 0}} >Thread-15911610::DEBUG::2013-11-19 21:38:12,914::task::495::TaskManager.Task::(_debug) Task 8eb48158-64da-4aac-a6b0-8f07b324fe1b: moving from state preparing -> state finished >Thread-15911610::DEBUG::2013-11-19 21:38:12,914::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911610::DEBUG::2013-11-19 21:38:12,915::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911610::DEBUG::2013-11-19 21:38:12,916::task::495::TaskManager.Task::(_debug) Task 8eb48158-64da-4aac-a6b0-8f07b324fe1b: ref 0 aborting False >Thread-15911610::INFO::2013-11-19 21:38:12,916::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000566005706787', 'lastCheck': 1384897083.752562, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152397155762', 'lastCheck': 1384897085.310354, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897091.4113779, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000468969345093', 'lastCheck': 1384897082.7467189, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:38:14,279::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:14,826::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0705791 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:38:16,841::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:17,367::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0701666 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15911614::DEBUG::2013-11-19 21:38:17,472::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911614::INFO::2013-11-19 21:38:17,474::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911614::DEBUG::2013-11-19 21:38:17,475::task::495::TaskManager.Task::(_debug) Task 86040e83-6102-4055-ab64-3ce65dd99598: moving from state init -> state preparing >Thread-15911614::DEBUG::2013-11-19 21:38:17,476::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911614::DEBUG::2013-11-19 21:38:17,476::task::495::TaskManager.Task::(_debug) Task 86040e83-6102-4055-ab64-3ce65dd99598: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911614::DEBUG::2013-11-19 21:38:17,481::task::495::TaskManager.Task::(_debug) Task 86040e83-6102-4055-ab64-3ce65dd99598: moving from state preparing -> state finished >Thread-15911614::DEBUG::2013-11-19 21:38:17,482::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911614::DEBUG::2013-11-19 21:38:17,482::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911614::DEBUG::2013-11-19 21:38:17,483::task::495::TaskManager.Task::(_debug) Task 86040e83-6102-4055-ab64-3ce65dd99598: ref 0 aborting False >Thread-15911614::INFO::2013-11-19 21:38:17,484::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911615::DEBUG::2013-11-19 21:38:17,492::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911615::INFO::2013-11-19 21:38:17,493::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911615::DEBUG::2013-11-19 21:38:17,494::task::495::TaskManager.Task::(_debug) Task ee2442e8-5e97-4aec-ba85-bf1180626a34: moving from state init -> state preparing >Thread-15911615::DEBUG::2013-11-19 21:38:17,495::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`72e11759-2a2a-4bff-a2c7-2d29e43ea2ac`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911615::DEBUG::2013-11-19 21:38:17,496::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911615::DEBUG::2013-11-19 21:38:17,497::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911615::DEBUG::2013-11-19 21:38:17,498::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`72e11759-2a2a-4bff-a2c7-2d29e43ea2ac`::Granted request >Thread-15911615::DEBUG::2013-11-19 21:38:17,507::task::495::TaskManager.Task::(_debug) Task ee2442e8-5e97-4aec-ba85-bf1180626a34: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911615::DEBUG::2013-11-19 21:38:17,508::task::495::TaskManager.Task::(_debug) Task ee2442e8-5e97-4aec-ba85-bf1180626a34: ref 1 aborting False >Thread-15911615::WARNING::2013-11-19 21:38:17,510::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911615::DEBUG::2013-11-19 21:38:17,511::task::495::TaskManager.Task::(_debug) Task ee2442e8-5e97-4aec-ba85-bf1180626a34: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911615::DEBUG::2013-11-19 21:38:17,512::task::495::TaskManager.Task::(_debug) Task ee2442e8-5e97-4aec-ba85-bf1180626a34: moving from state preparing -> state finished >Thread-15911615::DEBUG::2013-11-19 21:38:17,513::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911615::DEBUG::2013-11-19 21:38:17,513::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911615::DEBUG::2013-11-19 21:38:17,514::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911615::DEBUG::2013-11-19 21:38:17,515::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911615::DEBUG::2013-11-19 21:38:17,515::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911615::DEBUG::2013-11-19 21:38:17,516::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911615::DEBUG::2013-11-19 21:38:17,516::task::495::TaskManager.Task::(_debug) Task ee2442e8-5e97-4aec-ba85-bf1180626a34: ref 0 aborting False >Thread-15911615::INFO::2013-11-19 21:38:17,517::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:38:19,382::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:19,909::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0691234 s, 14.8 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:38:21,615::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:38:21,924::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:22,469::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0702838 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15911618::INFO::2013-11-19 21:38:23,084::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911618::DEBUG::2013-11-19 21:38:23,085::task::495::TaskManager.Task::(_debug) Task 55fa5182-4b64-4d70-a4cd-09cbd3a94388: moving from state init -> state preparing >Thread-15911618::DEBUG::2013-11-19 21:38:23,086::task::495::TaskManager.Task::(_debug) Task 55fa5182-4b64-4d70-a4cd-09cbd3a94388: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000472068786621', 'lastCheck': 1384897093.950063, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147795677185', 'lastCheck': 1384897095.5166919, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897101.6159141, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000579833984375', 'lastCheck': 1384897092.945426, 'valid': True, 'code': 0}} >Thread-15911618::DEBUG::2013-11-19 21:38:23,094::task::495::TaskManager.Task::(_debug) Task 55fa5182-4b64-4d70-a4cd-09cbd3a94388: moving from state preparing -> state finished >Thread-15911618::DEBUG::2013-11-19 21:38:23,095::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911618::DEBUG::2013-11-19 21:38:23,095::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911618::DEBUG::2013-11-19 21:38:23,096::task::495::TaskManager.Task::(_debug) Task 55fa5182-4b64-4d70-a4cd-09cbd3a94388: ref 0 aborting False >Thread-15911618::INFO::2013-11-19 21:38:23,099::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000472068786621', 'lastCheck': 1384897093.950063, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147795677185', 'lastCheck': 1384897095.5166919, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897101.6159141, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000579833984375', 'lastCheck': 1384897092.945426, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:38:24,484::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:25,003::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0695959 s, 14.7 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:38:27,018::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:27,566::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0723218 s, 14.2 MB/s\n'; <rc> = 0 >Thread-15911622::DEBUG::2013-11-19 21:38:27,655::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911622::INFO::2013-11-19 21:38:27,656::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911622::DEBUG::2013-11-19 21:38:27,657::task::495::TaskManager.Task::(_debug) Task de92b221-539d-4566-8530-b1bd803a4c03: moving from state init -> state preparing >Thread-15911622::DEBUG::2013-11-19 21:38:27,657::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911622::DEBUG::2013-11-19 21:38:27,658::task::495::TaskManager.Task::(_debug) Task de92b221-539d-4566-8530-b1bd803a4c03: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911622::DEBUG::2013-11-19 21:38:27,659::task::495::TaskManager.Task::(_debug) Task de92b221-539d-4566-8530-b1bd803a4c03: moving from state preparing -> state finished >Thread-15911622::DEBUG::2013-11-19 21:38:27,659::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911622::DEBUG::2013-11-19 21:38:27,660::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911622::DEBUG::2013-11-19 21:38:27,660::task::495::TaskManager.Task::(_debug) Task de92b221-539d-4566-8530-b1bd803a4c03: ref 0 aborting False >Thread-15911622::INFO::2013-11-19 21:38:27,661::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911623::DEBUG::2013-11-19 21:38:27,667::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911623::INFO::2013-11-19 21:38:27,668::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911623::DEBUG::2013-11-19 21:38:27,668::task::495::TaskManager.Task::(_debug) Task 939426b8-3427-4444-b5fc-5f2163b86c5c: moving from state init -> state preparing >Thread-15911623::DEBUG::2013-11-19 21:38:27,669::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d80f834b-b6cd-4aac-83fc-11f52569fce5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911623::DEBUG::2013-11-19 21:38:27,670::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911623::DEBUG::2013-11-19 21:38:27,671::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911623::DEBUG::2013-11-19 21:38:27,672::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d80f834b-b6cd-4aac-83fc-11f52569fce5`::Granted request >Thread-15911623::DEBUG::2013-11-19 21:38:27,673::task::495::TaskManager.Task::(_debug) Task 939426b8-3427-4444-b5fc-5f2163b86c5c: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911623::DEBUG::2013-11-19 21:38:27,673::task::495::TaskManager.Task::(_debug) Task 939426b8-3427-4444-b5fc-5f2163b86c5c: ref 1 aborting False >Thread-15911623::WARNING::2013-11-19 21:38:27,675::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911623::DEBUG::2013-11-19 21:38:27,676::task::495::TaskManager.Task::(_debug) Task 939426b8-3427-4444-b5fc-5f2163b86c5c: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911623::DEBUG::2013-11-19 21:38:27,677::task::495::TaskManager.Task::(_debug) Task 939426b8-3427-4444-b5fc-5f2163b86c5c: moving from state preparing -> state finished >Thread-15911623::DEBUG::2013-11-19 21:38:27,678::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911623::DEBUG::2013-11-19 21:38:27,678::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911623::DEBUG::2013-11-19 21:38:27,679::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911623::DEBUG::2013-11-19 21:38:27,680::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911623::DEBUG::2013-11-19 21:38:27,680::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911623::DEBUG::2013-11-19 21:38:27,681::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911623::DEBUG::2013-11-19 21:38:27,681::task::495::TaskManager.Task::(_debug) Task 939426b8-3427-4444-b5fc-5f2163b86c5c: ref 0 aborting False >Thread-15911623::INFO::2013-11-19 21:38:27,682::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:38:29,581::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:30,116::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0698918 s, 14.7 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:38:31,827::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:38:32,131::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:32,659::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0701147 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15911626::INFO::2013-11-19 21:38:33,367::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911626::DEBUG::2013-11-19 21:38:33,367::task::495::TaskManager.Task::(_debug) Task f05e481a-523f-48b1-9055-1b8f42bc07a1: moving from state init -> state preparing >Thread-15911626::DEBUG::2013-11-19 21:38:33,368::task::495::TaskManager.Task::(_debug) Task f05e481a-523f-48b1-9055-1b8f42bc07a1: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000432014465332', 'lastCheck': 1384897104.174964, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00143790245056', 'lastCheck': 1384897105.70068, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897111.8288031, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000478982925415', 'lastCheck': 1384897113.357759, 'valid': True, 'code': 0}} >Thread-15911626::DEBUG::2013-11-19 21:38:33,374::task::495::TaskManager.Task::(_debug) Task f05e481a-523f-48b1-9055-1b8f42bc07a1: moving from state preparing -> state finished >Thread-15911626::DEBUG::2013-11-19 21:38:33,375::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911626::DEBUG::2013-11-19 21:38:33,375::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911626::DEBUG::2013-11-19 21:38:33,376::task::495::TaskManager.Task::(_debug) Task f05e481a-523f-48b1-9055-1b8f42bc07a1: ref 0 aborting False >Thread-15911626::INFO::2013-11-19 21:38:33,378::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000432014465332', 'lastCheck': 1384897104.174964, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00143790245056', 'lastCheck': 1384897105.70068, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897111.8288031, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000478982925415', 'lastCheck': 1384897113.357759, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:38:34,675::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:35,215::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0702658 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:38:37,230::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:37,744::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0696683 s, 14.7 MB/s\n'; <rc> = 0 >Thread-15911630::DEBUG::2013-11-19 21:38:37,891::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911630::INFO::2013-11-19 21:38:37,892::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911630::DEBUG::2013-11-19 21:38:37,893::task::495::TaskManager.Task::(_debug) Task 95edae5c-b7d8-48f0-8f17-b0cc8ffbf0f1: moving from state init -> state preparing >Thread-15911630::DEBUG::2013-11-19 21:38:37,894::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911630::DEBUG::2013-11-19 21:38:37,894::task::495::TaskManager.Task::(_debug) Task 95edae5c-b7d8-48f0-8f17-b0cc8ffbf0f1: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911630::DEBUG::2013-11-19 21:38:37,895::task::495::TaskManager.Task::(_debug) Task 95edae5c-b7d8-48f0-8f17-b0cc8ffbf0f1: moving from state preparing -> state finished >Thread-15911630::DEBUG::2013-11-19 21:38:37,896::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911630::DEBUG::2013-11-19 21:38:37,896::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911630::DEBUG::2013-11-19 21:38:37,897::task::495::TaskManager.Task::(_debug) Task 95edae5c-b7d8-48f0-8f17-b0cc8ffbf0f1: ref 0 aborting False >Thread-15911630::INFO::2013-11-19 21:38:37,897::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911631::DEBUG::2013-11-19 21:38:37,903::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911631::INFO::2013-11-19 21:38:37,904::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911631::DEBUG::2013-11-19 21:38:37,905::task::495::TaskManager.Task::(_debug) Task 68a69219-678b-482f-a8bf-3657ed6b2029: moving from state init -> state preparing >Thread-15911631::DEBUG::2013-11-19 21:38:37,908::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`18e0c9b0-64dd-4bd3-949d-2f0c9ae9527b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911631::DEBUG::2013-11-19 21:38:37,909::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911631::DEBUG::2013-11-19 21:38:37,911::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911631::DEBUG::2013-11-19 21:38:37,912::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`18e0c9b0-64dd-4bd3-949d-2f0c9ae9527b`::Granted request >Thread-15911631::DEBUG::2013-11-19 21:38:37,913::task::495::TaskManager.Task::(_debug) Task 68a69219-678b-482f-a8bf-3657ed6b2029: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911631::DEBUG::2013-11-19 21:38:37,914::task::495::TaskManager.Task::(_debug) Task 68a69219-678b-482f-a8bf-3657ed6b2029: ref 1 aborting False >Thread-15911631::WARNING::2013-11-19 21:38:37,916::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911631::DEBUG::2013-11-19 21:38:37,917::task::495::TaskManager.Task::(_debug) Task 68a69219-678b-482f-a8bf-3657ed6b2029: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911631::DEBUG::2013-11-19 21:38:37,918::task::495::TaskManager.Task::(_debug) Task 68a69219-678b-482f-a8bf-3657ed6b2029: moving from state preparing -> state finished >Thread-15911631::DEBUG::2013-11-19 21:38:37,918::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911631::DEBUG::2013-11-19 21:38:37,919::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911631::DEBUG::2013-11-19 21:38:37,919::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911631::DEBUG::2013-11-19 21:38:37,920::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911631::DEBUG::2013-11-19 21:38:37,921::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911631::DEBUG::2013-11-19 21:38:37,921::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911631::DEBUG::2013-11-19 21:38:37,922::task::495::TaskManager.Task::(_debug) Task 68a69219-678b-482f-a8bf-3657ed6b2029: ref 0 aborting False >Thread-15911631::INFO::2013-11-19 21:38:37,922::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:38:39,758::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:40,302::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0932622 s, 11.0 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:38:41,986::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:38:42,317::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:42,844::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0702656 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15911634::INFO::2013-11-19 21:38:43,529::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911634::DEBUG::2013-11-19 21:38:43,530::task::495::TaskManager.Task::(_debug) Task 1c16cf1f-e0c5-4b94-aa0a-fbb122c25f52: moving from state init -> state preparing >Thread-15911634::DEBUG::2013-11-19 21:38:43,531::task::495::TaskManager.Task::(_debug) Task 1c16cf1f-e0c5-4b94-aa0a-fbb122c25f52: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000480890274048', 'lastCheck': 1384897114.3604419, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146198272705', 'lastCheck': 1384897115.9116211, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897121.9870801, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000478982925415', 'lastCheck': 1384897113.357759, 'valid': True, 'code': 0}} >Thread-15911634::DEBUG::2013-11-19 21:38:43,532::task::495::TaskManager.Task::(_debug) Task 1c16cf1f-e0c5-4b94-aa0a-fbb122c25f52: moving from state preparing -> state finished >Thread-15911634::DEBUG::2013-11-19 21:38:43,532::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911634::DEBUG::2013-11-19 21:38:43,533::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911634::DEBUG::2013-11-19 21:38:43,534::task::495::TaskManager.Task::(_debug) Task 1c16cf1f-e0c5-4b94-aa0a-fbb122c25f52: ref 0 aborting False >Thread-15911634::INFO::2013-11-19 21:38:43,537::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000480890274048', 'lastCheck': 1384897114.3604419, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146198272705', 'lastCheck': 1384897115.9116211, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897121.9870801, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000478982925415', 'lastCheck': 1384897113.357759, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:38:44,860::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:45,376::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.069632 s, 14.7 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:38:47,391::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:47,961::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092682 s, 11.0 MB/s\n'; <rc> = 0 >Thread-15911638::DEBUG::2013-11-19 21:38:48,035::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911638::INFO::2013-11-19 21:38:48,036::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911638::DEBUG::2013-11-19 21:38:48,037::task::495::TaskManager.Task::(_debug) Task 24310a4a-f35b-487a-83b9-838675708daf: moving from state init -> state preparing >Thread-15911638::DEBUG::2013-11-19 21:38:48,038::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911638::DEBUG::2013-11-19 21:38:48,038::task::495::TaskManager.Task::(_debug) Task 24310a4a-f35b-487a-83b9-838675708daf: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911638::DEBUG::2013-11-19 21:38:48,039::task::495::TaskManager.Task::(_debug) Task 24310a4a-f35b-487a-83b9-838675708daf: moving from state preparing -> state finished >Thread-15911638::DEBUG::2013-11-19 21:38:48,040::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911638::DEBUG::2013-11-19 21:38:48,040::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911638::DEBUG::2013-11-19 21:38:48,041::task::495::TaskManager.Task::(_debug) Task 24310a4a-f35b-487a-83b9-838675708daf: ref 0 aborting False >Thread-15911638::INFO::2013-11-19 21:38:48,041::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911639::DEBUG::2013-11-19 21:38:48,047::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911639::INFO::2013-11-19 21:38:48,048::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911639::DEBUG::2013-11-19 21:38:48,049::task::495::TaskManager.Task::(_debug) Task ad658633-005b-47e4-9d76-060379e09e66: moving from state init -> state preparing >Thread-15911639::DEBUG::2013-11-19 21:38:48,050::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c8705395-8c55-4c3c-b842-bbef989c6079`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911639::DEBUG::2013-11-19 21:38:48,050::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911639::DEBUG::2013-11-19 21:38:48,051::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911639::DEBUG::2013-11-19 21:38:48,052::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c8705395-8c55-4c3c-b842-bbef989c6079`::Granted request >Thread-15911639::DEBUG::2013-11-19 21:38:48,053::task::495::TaskManager.Task::(_debug) Task ad658633-005b-47e4-9d76-060379e09e66: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911639::DEBUG::2013-11-19 21:38:48,053::task::495::TaskManager.Task::(_debug) Task ad658633-005b-47e4-9d76-060379e09e66: ref 1 aborting False >Thread-15911639::WARNING::2013-11-19 21:38:48,056::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911639::DEBUG::2013-11-19 21:38:48,056::task::495::TaskManager.Task::(_debug) Task ad658633-005b-47e4-9d76-060379e09e66: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911639::DEBUG::2013-11-19 21:38:48,057::task::495::TaskManager.Task::(_debug) Task ad658633-005b-47e4-9d76-060379e09e66: moving from state preparing -> state finished >Thread-15911639::DEBUG::2013-11-19 21:38:48,058::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911639::DEBUG::2013-11-19 21:38:48,059::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911639::DEBUG::2013-11-19 21:38:48,059::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911639::DEBUG::2013-11-19 21:38:48,060::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911639::DEBUG::2013-11-19 21:38:48,061::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911639::DEBUG::2013-11-19 21:38:48,061::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911639::DEBUG::2013-11-19 21:38:48,063::task::495::TaskManager.Task::(_debug) Task ad658633-005b-47e4-9d76-060379e09e66: ref 0 aborting False >Thread-15911639::INFO::2013-11-19 21:38:48,063::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:38:49,977::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:50,498::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0702064 s, 14.6 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:38:52,193::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:38:52,513::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:53,038::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0703057 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15911642::INFO::2013-11-19 21:38:53,696::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911642::DEBUG::2013-11-19 21:38:53,697::task::495::TaskManager.Task::(_debug) Task b6741232-ad8a-409e-a5ed-6c59b9d6ffd5: moving from state init -> state preparing >Thread-15911642::DEBUG::2013-11-19 21:38:53,697::task::495::TaskManager.Task::(_debug) Task b6741232-ad8a-409e-a5ed-6c59b9d6ffd5: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000468969345093', 'lastCheck': 1384897124.5587959, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146889686584', 'lastCheck': 1384897126.065712, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897132.1946211, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0010199546814', 'lastCheck': 1384897123.5402679, 'valid': True, 'code': 0}} >Thread-15911642::DEBUG::2013-11-19 21:38:53,698::task::495::TaskManager.Task::(_debug) Task b6741232-ad8a-409e-a5ed-6c59b9d6ffd5: moving from state preparing -> state finished >Thread-15911642::DEBUG::2013-11-19 21:38:53,699::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911642::DEBUG::2013-11-19 21:38:53,699::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911642::DEBUG::2013-11-19 21:38:53,700::task::495::TaskManager.Task::(_debug) Task b6741232-ad8a-409e-a5ed-6c59b9d6ffd5: ref 0 aborting False >Thread-15911642::INFO::2013-11-19 21:38:53,701::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000468969345093', 'lastCheck': 1384897124.5587959, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146889686584', 'lastCheck': 1384897126.065712, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897132.1946211, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0010199546814', 'lastCheck': 1384897123.5402679, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:38:55,053::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:55,601::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0707513 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:38:57,615::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:38:58,144::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.070456 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15911646::DEBUG::2013-11-19 21:38:58,171::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911646::INFO::2013-11-19 21:38:58,172::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911646::DEBUG::2013-11-19 21:38:58,173::task::495::TaskManager.Task::(_debug) Task 28d670b0-062b-4f77-9c70-637663158907: moving from state init -> state preparing >Thread-15911646::DEBUG::2013-11-19 21:38:58,174::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911646::DEBUG::2013-11-19 21:38:58,174::task::495::TaskManager.Task::(_debug) Task 28d670b0-062b-4f77-9c70-637663158907: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911646::DEBUG::2013-11-19 21:38:58,175::task::495::TaskManager.Task::(_debug) Task 28d670b0-062b-4f77-9c70-637663158907: moving from state preparing -> state finished >Thread-15911646::DEBUG::2013-11-19 21:38:58,175::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911646::DEBUG::2013-11-19 21:38:58,176::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911646::DEBUG::2013-11-19 21:38:58,177::task::495::TaskManager.Task::(_debug) Task 28d670b0-062b-4f77-9c70-637663158907: ref 0 aborting False >Thread-15911646::INFO::2013-11-19 21:38:58,177::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911647::DEBUG::2013-11-19 21:38:58,187::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911647::INFO::2013-11-19 21:38:58,188::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911647::DEBUG::2013-11-19 21:38:58,189::task::495::TaskManager.Task::(_debug) Task 97b0512e-f85e-4ac5-8eb5-8ac744c78bc0: moving from state init -> state preparing >Thread-15911647::DEBUG::2013-11-19 21:38:58,190::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2710bc18-f98c-4574-8f71-af717ec01372`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911647::DEBUG::2013-11-19 21:38:58,190::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911647::DEBUG::2013-11-19 21:38:58,191::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911647::DEBUG::2013-11-19 21:38:58,192::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2710bc18-f98c-4574-8f71-af717ec01372`::Granted request >Thread-15911647::DEBUG::2013-11-19 21:38:58,193::task::495::TaskManager.Task::(_debug) Task 97b0512e-f85e-4ac5-8eb5-8ac744c78bc0: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911647::DEBUG::2013-11-19 21:38:58,193::task::495::TaskManager.Task::(_debug) Task 97b0512e-f85e-4ac5-8eb5-8ac744c78bc0: ref 1 aborting False >Thread-15911647::WARNING::2013-11-19 21:38:58,195::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911647::DEBUG::2013-11-19 21:38:58,196::task::495::TaskManager.Task::(_debug) Task 97b0512e-f85e-4ac5-8eb5-8ac744c78bc0: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911647::DEBUG::2013-11-19 21:38:58,197::task::495::TaskManager.Task::(_debug) Task 97b0512e-f85e-4ac5-8eb5-8ac744c78bc0: moving from state preparing -> state finished >Thread-15911647::DEBUG::2013-11-19 21:38:58,198::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911647::DEBUG::2013-11-19 21:38:58,199::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911647::DEBUG::2013-11-19 21:38:58,199::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911647::DEBUG::2013-11-19 21:38:58,200::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911647::DEBUG::2013-11-19 21:38:58,200::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911647::DEBUG::2013-11-19 21:38:58,201::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911647::DEBUG::2013-11-19 21:38:58,202::task::495::TaskManager.Task::(_debug) Task 97b0512e-f85e-4ac5-8eb5-8ac744c78bc0: ref 0 aborting False >Thread-15911647::INFO::2013-11-19 21:38:58,202::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133593825280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:39:00,159::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:00,717::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0742684 s, 13.8 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:39:02,394::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:39:02,732::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:03,252::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0700733 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15911650::INFO::2013-11-19 21:39:03,861::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911650::DEBUG::2013-11-19 21:39:03,862::task::495::TaskManager.Task::(_debug) Task 21ae5293-2afe-47f7-9ad4-7eb692e7e58a: moving from state init -> state preparing >Thread-15911650::DEBUG::2013-11-19 21:39:03,862::task::495::TaskManager.Task::(_debug) Task 21ae5293-2afe-47f7-9ad4-7eb692e7e58a: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000375032424927', 'lastCheck': 1384897134.7402501, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00218987464905', 'lastCheck': 1384897136.29813, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897142.3955081, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000626087188721', 'lastCheck': 1384897133.73611, 'valid': True, 'code': 0}} >Thread-15911650::DEBUG::2013-11-19 21:39:03,868::task::495::TaskManager.Task::(_debug) Task 21ae5293-2afe-47f7-9ad4-7eb692e7e58a: moving from state preparing -> state finished >Thread-15911650::DEBUG::2013-11-19 21:39:03,869::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911650::DEBUG::2013-11-19 21:39:03,869::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911650::DEBUG::2013-11-19 21:39:03,870::task::495::TaskManager.Task::(_debug) Task 21ae5293-2afe-47f7-9ad4-7eb692e7e58a: ref 0 aborting False >Thread-15911650::INFO::2013-11-19 21:39:03,871::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000375032424927', 'lastCheck': 1384897134.7402501, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00218987464905', 'lastCheck': 1384897136.29813, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897142.3955081, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000626087188721', 'lastCheck': 1384897133.73611, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:39:05,267::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:05,814::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0708154 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:39:07,830::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:08,364::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0702079 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15911654::DEBUG::2013-11-19 21:39:09,037::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911654::INFO::2013-11-19 21:39:09,038::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911654::DEBUG::2013-11-19 21:39:09,039::task::495::TaskManager.Task::(_debug) Task 7e03dc26-7808-44f4-9d49-3cccd5518b04: moving from state init -> state preparing >Thread-15911654::DEBUG::2013-11-19 21:39:09,040::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911654::DEBUG::2013-11-19 21:39:09,040::task::495::TaskManager.Task::(_debug) Task 7e03dc26-7808-44f4-9d49-3cccd5518b04: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911654::DEBUG::2013-11-19 21:39:09,041::task::495::TaskManager.Task::(_debug) Task 7e03dc26-7808-44f4-9d49-3cccd5518b04: moving from state preparing -> state finished >Thread-15911654::DEBUG::2013-11-19 21:39:09,041::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911654::DEBUG::2013-11-19 21:39:09,042::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911654::DEBUG::2013-11-19 21:39:09,043::task::495::TaskManager.Task::(_debug) Task 7e03dc26-7808-44f4-9d49-3cccd5518b04: ref 0 aborting False >Thread-15911654::INFO::2013-11-19 21:39:09,043::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911655::DEBUG::2013-11-19 21:39:09,056::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911655::INFO::2013-11-19 21:39:09,057::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911655::DEBUG::2013-11-19 21:39:09,058::task::495::TaskManager.Task::(_debug) Task c17cdaa8-918b-46df-956c-5fbc9f859c8f: moving from state init -> state preparing >Thread-15911655::DEBUG::2013-11-19 21:39:09,059::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fe563f78-4cf6-4edf-be88-bd503a13b822`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911655::DEBUG::2013-11-19 21:39:09,060::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911655::DEBUG::2013-11-19 21:39:09,060::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911655::DEBUG::2013-11-19 21:39:09,061::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fe563f78-4cf6-4edf-be88-bd503a13b822`::Granted request >Thread-15911655::DEBUG::2013-11-19 21:39:09,062::task::495::TaskManager.Task::(_debug) Task c17cdaa8-918b-46df-956c-5fbc9f859c8f: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911655::DEBUG::2013-11-19 21:39:09,063::task::495::TaskManager.Task::(_debug) Task c17cdaa8-918b-46df-956c-5fbc9f859c8f: ref 1 aborting False >Thread-15911655::WARNING::2013-11-19 21:39:09,065::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911655::DEBUG::2013-11-19 21:39:09,067::task::495::TaskManager.Task::(_debug) Task c17cdaa8-918b-46df-956c-5fbc9f859c8f: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133970264064', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911655::DEBUG::2013-11-19 21:39:09,068::task::495::TaskManager.Task::(_debug) Task c17cdaa8-918b-46df-956c-5fbc9f859c8f: moving from state preparing -> state finished >Thread-15911655::DEBUG::2013-11-19 21:39:09,068::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911655::DEBUG::2013-11-19 21:39:09,069::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911655::DEBUG::2013-11-19 21:39:09,070::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911655::DEBUG::2013-11-19 21:39:09,070::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911655::DEBUG::2013-11-19 21:39:09,071::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911655::DEBUG::2013-11-19 21:39:09,071::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911655::DEBUG::2013-11-19 21:39:09,072::task::495::TaskManager.Task::(_debug) Task c17cdaa8-918b-46df-956c-5fbc9f859c8f: ref 0 aborting False >Thread-15911655::INFO::2013-11-19 21:39:09,073::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133970264064', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:39:10,379::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:10,913::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0701843 s, 14.6 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:39:12,619::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:39:12,929::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:13,472::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0705942 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15911658::INFO::2013-11-19 21:39:14,086::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911658::DEBUG::2013-11-19 21:39:14,087::task::495::TaskManager.Task::(_debug) Task feb0cce4-e1ae-4175-b44e-467ed44f8765: moving from state init -> state preparing >Thread-15911658::DEBUG::2013-11-19 21:39:14,098::task::495::TaskManager.Task::(_debug) Task feb0cce4-e1ae-4175-b44e-467ed44f8765: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000389099121094', 'lastCheck': 1384897144.9570501, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0014750957489', 'lastCheck': 1384897146.5063679, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897152.6211431, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000453948974609', 'lastCheck': 1384897143.9518919, 'valid': True, 'code': 0}} >Thread-15911658::DEBUG::2013-11-19 21:39:14,105::task::495::TaskManager.Task::(_debug) Task feb0cce4-e1ae-4175-b44e-467ed44f8765: moving from state preparing -> state finished >Thread-15911658::DEBUG::2013-11-19 21:39:14,105::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911658::DEBUG::2013-11-19 21:39:14,106::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911658::DEBUG::2013-11-19 21:39:14,106::task::495::TaskManager.Task::(_debug) Task feb0cce4-e1ae-4175-b44e-467ed44f8765: ref 0 aborting False >Thread-15911658::INFO::2013-11-19 21:39:14,107::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000389099121094', 'lastCheck': 1384897144.9570501, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0014750957489', 'lastCheck': 1384897146.5063679, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897152.6211431, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000453948974609', 'lastCheck': 1384897143.9518919, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:39:15,487::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:16,014::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0705622 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:39:18,029::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:18,574::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0705349 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15911662::DEBUG::2013-11-19 21:39:19,193::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911662::INFO::2013-11-19 21:39:19,194::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911662::DEBUG::2013-11-19 21:39:19,195::task::495::TaskManager.Task::(_debug) Task de884029-eabe-4e28-a808-358292ce349d: moving from state init -> state preparing >Thread-15911662::DEBUG::2013-11-19 21:39:19,196::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911662::DEBUG::2013-11-19 21:39:19,197::task::495::TaskManager.Task::(_debug) Task de884029-eabe-4e28-a808-358292ce349d: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911662::DEBUG::2013-11-19 21:39:19,197::task::495::TaskManager.Task::(_debug) Task de884029-eabe-4e28-a808-358292ce349d: moving from state preparing -> state finished >Thread-15911662::DEBUG::2013-11-19 21:39:19,198::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911662::DEBUG::2013-11-19 21:39:19,198::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911662::DEBUG::2013-11-19 21:39:19,199::task::495::TaskManager.Task::(_debug) Task de884029-eabe-4e28-a808-358292ce349d: ref 0 aborting False >Thread-15911662::INFO::2013-11-19 21:39:19,199::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911663::DEBUG::2013-11-19 21:39:19,205::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911663::INFO::2013-11-19 21:39:19,206::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911663::DEBUG::2013-11-19 21:39:19,207::task::495::TaskManager.Task::(_debug) Task 43817885-0a6a-46c1-938e-996a34bac0f0: moving from state init -> state preparing >Thread-15911663::DEBUG::2013-11-19 21:39:19,208::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`868d70d8-9b97-4680-9f38-6785ec70b9f7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911663::DEBUG::2013-11-19 21:39:19,208::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911663::DEBUG::2013-11-19 21:39:19,209::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911663::DEBUG::2013-11-19 21:39:19,210::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`868d70d8-9b97-4680-9f38-6785ec70b9f7`::Granted request >Thread-15911663::DEBUG::2013-11-19 21:39:19,211::task::495::TaskManager.Task::(_debug) Task 43817885-0a6a-46c1-938e-996a34bac0f0: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911663::DEBUG::2013-11-19 21:39:19,211::task::495::TaskManager.Task::(_debug) Task 43817885-0a6a-46c1-938e-996a34bac0f0: ref 1 aborting False >Thread-15911663::WARNING::2013-11-19 21:39:19,214::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911663::DEBUG::2013-11-19 21:39:19,214::task::495::TaskManager.Task::(_debug) Task 43817885-0a6a-46c1-938e-996a34bac0f0: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133842337792', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911663::DEBUG::2013-11-19 21:39:19,215::task::495::TaskManager.Task::(_debug) Task 43817885-0a6a-46c1-938e-996a34bac0f0: moving from state preparing -> state finished >Thread-15911663::DEBUG::2013-11-19 21:39:19,216::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911663::DEBUG::2013-11-19 21:39:19,217::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911663::DEBUG::2013-11-19 21:39:19,217::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911663::DEBUG::2013-11-19 21:39:19,218::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911663::DEBUG::2013-11-19 21:39:19,218::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911663::DEBUG::2013-11-19 21:39:19,219::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911663::DEBUG::2013-11-19 21:39:19,220::task::495::TaskManager.Task::(_debug) Task 43817885-0a6a-46c1-938e-996a34bac0f0: ref 0 aborting False >Thread-15911663::INFO::2013-11-19 21:39:19,220::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133842337792', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:39:20,590::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:21,117::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0700825 s, 14.6 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:39:22,803::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:39:23,132::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:23,652::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0695724 s, 14.7 MB/s\n'; <rc> = 0 >Thread-15911666::INFO::2013-11-19 21:39:24,286::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911666::DEBUG::2013-11-19 21:39:24,287::task::495::TaskManager.Task::(_debug) Task 51f57657-f5cc-4782-85b8-94603d939b1b: moving from state init -> state preparing >Thread-15911666::DEBUG::2013-11-19 21:39:24,297::task::495::TaskManager.Task::(_debug) Task 51f57657-f5cc-4782-85b8-94603d939b1b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000333070755005', 'lastCheck': 1384897155.1538999, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152492523193', 'lastCheck': 1384897156.7083681, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897162.804759, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000536203384399', 'lastCheck': 1384897154.1495919, 'valid': True, 'code': 0}} >Thread-15911666::DEBUG::2013-11-19 21:39:24,298::task::495::TaskManager.Task::(_debug) Task 51f57657-f5cc-4782-85b8-94603d939b1b: moving from state preparing -> state finished >Thread-15911666::DEBUG::2013-11-19 21:39:24,298::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911666::DEBUG::2013-11-19 21:39:24,299::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911666::DEBUG::2013-11-19 21:39:24,299::task::495::TaskManager.Task::(_debug) Task 51f57657-f5cc-4782-85b8-94603d939b1b: ref 0 aborting False >Thread-15911666::INFO::2013-11-19 21:39:24,300::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000333070755005', 'lastCheck': 1384897155.1538999, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152492523193', 'lastCheck': 1384897156.7083681, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897162.804759, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000536203384399', 'lastCheck': 1384897154.1495919, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:39:25,667::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:26,212::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0706747 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:39:28,228::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:28,753::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0710031 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15911670::DEBUG::2013-11-19 21:39:29,346::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911670::INFO::2013-11-19 21:39:29,348::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911670::DEBUG::2013-11-19 21:39:29,356::task::495::TaskManager.Task::(_debug) Task c6c4c91c-47f1-4e21-9ed9-18c4f2d753ee: moving from state init -> state preparing >Thread-15911670::DEBUG::2013-11-19 21:39:29,357::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911670::DEBUG::2013-11-19 21:39:29,357::task::495::TaskManager.Task::(_debug) Task c6c4c91c-47f1-4e21-9ed9-18c4f2d753ee: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911670::DEBUG::2013-11-19 21:39:29,358::task::495::TaskManager.Task::(_debug) Task c6c4c91c-47f1-4e21-9ed9-18c4f2d753ee: moving from state preparing -> state finished >Thread-15911670::DEBUG::2013-11-19 21:39:29,358::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911670::DEBUG::2013-11-19 21:39:29,359::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911670::DEBUG::2013-11-19 21:39:29,359::task::495::TaskManager.Task::(_debug) Task c6c4c91c-47f1-4e21-9ed9-18c4f2d753ee: ref 0 aborting False >Thread-15911670::INFO::2013-11-19 21:39:29,360::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911671::DEBUG::2013-11-19 21:39:29,366::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911671::INFO::2013-11-19 21:39:29,367::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911671::DEBUG::2013-11-19 21:39:29,367::task::495::TaskManager.Task::(_debug) Task b7410507-5454-441e-a9eb-be07049e27da: moving from state init -> state preparing >Thread-15911671::DEBUG::2013-11-19 21:39:29,369::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f1a0ccee-1b7a-4942-b5fb-5980f5421e45`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911671::DEBUG::2013-11-19 21:39:29,369::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911671::DEBUG::2013-11-19 21:39:29,370::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911671::DEBUG::2013-11-19 21:39:29,371::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f1a0ccee-1b7a-4942-b5fb-5980f5421e45`::Granted request >Thread-15911671::DEBUG::2013-11-19 21:39:29,372::task::495::TaskManager.Task::(_debug) Task b7410507-5454-441e-a9eb-be07049e27da: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911671::DEBUG::2013-11-19 21:39:29,373::task::495::TaskManager.Task::(_debug) Task b7410507-5454-441e-a9eb-be07049e27da: ref 1 aborting False >Thread-15911671::WARNING::2013-11-19 21:39:29,375::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911671::DEBUG::2013-11-19 21:39:29,376::task::495::TaskManager.Task::(_debug) Task b7410507-5454-441e-a9eb-be07049e27da: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133804589056', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911671::DEBUG::2013-11-19 21:39:29,377::task::495::TaskManager.Task::(_debug) Task b7410507-5454-441e-a9eb-be07049e27da: moving from state preparing -> state finished >Thread-15911671::DEBUG::2013-11-19 21:39:29,377::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911671::DEBUG::2013-11-19 21:39:29,378::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911671::DEBUG::2013-11-19 21:39:29,378::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911671::DEBUG::2013-11-19 21:39:29,379::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911671::DEBUG::2013-11-19 21:39:29,380::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911671::DEBUG::2013-11-19 21:39:29,380::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911671::DEBUG::2013-11-19 21:39:29,381::task::495::TaskManager.Task::(_debug) Task b7410507-5454-441e-a9eb-be07049e27da: ref 0 aborting False >Thread-15911671::INFO::2013-11-19 21:39:29,381::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133804589056', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:39:30,768::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:31,314::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0705928 s, 14.5 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:39:33,005::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:39:33,330::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:33,855::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0744026 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15911674::INFO::2013-11-19 21:39:34,499::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911674::DEBUG::2013-11-19 21:39:34,500::task::495::TaskManager.Task::(_debug) Task 98d62e51-14c3-4309-941d-140b4ffb05e8: moving from state init -> state preparing >Thread-15911674::DEBUG::2013-11-19 21:39:34,511::task::495::TaskManager.Task::(_debug) Task 98d62e51-14c3-4309-941d-140b4ffb05e8: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000287055969238', 'lastCheck': 1384897165.3577609, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148582458496', 'lastCheck': 1384897166.896399, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897173.006453, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000479936599731', 'lastCheck': 1384897164.354795, 'valid': True, 'code': 0}} >Thread-15911674::DEBUG::2013-11-19 21:39:34,512::task::495::TaskManager.Task::(_debug) Task 98d62e51-14c3-4309-941d-140b4ffb05e8: moving from state preparing -> state finished >Thread-15911674::DEBUG::2013-11-19 21:39:34,513::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911674::DEBUG::2013-11-19 21:39:34,513::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911674::DEBUG::2013-11-19 21:39:34,514::task::495::TaskManager.Task::(_debug) Task 98d62e51-14c3-4309-941d-140b4ffb05e8: ref 0 aborting False >Thread-15911674::INFO::2013-11-19 21:39:34,515::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000287055969238', 'lastCheck': 1384897165.3577609, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148582458496', 'lastCheck': 1384897166.896399, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897173.006453, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000479936599731', 'lastCheck': 1384897164.354795, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:39:35,871::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:36,382::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0694911 s, 14.7 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:39:38,397::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:38,932::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0701873 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15911678::DEBUG::2013-11-19 21:39:39,523::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911678::INFO::2013-11-19 21:39:39,524::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911678::DEBUG::2013-11-19 21:39:39,534::task::495::TaskManager.Task::(_debug) Task 9b1d5a60-2722-40eb-b5b6-98d37090a666: moving from state init -> state preparing >Thread-15911678::DEBUG::2013-11-19 21:39:39,535::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911678::DEBUG::2013-11-19 21:39:39,536::task::495::TaskManager.Task::(_debug) Task 9b1d5a60-2722-40eb-b5b6-98d37090a666: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911678::DEBUG::2013-11-19 21:39:39,536::task::495::TaskManager.Task::(_debug) Task 9b1d5a60-2722-40eb-b5b6-98d37090a666: moving from state preparing -> state finished >Thread-15911678::DEBUG::2013-11-19 21:39:39,537::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911678::DEBUG::2013-11-19 21:39:39,537::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911678::DEBUG::2013-11-19 21:39:39,538::task::495::TaskManager.Task::(_debug) Task 9b1d5a60-2722-40eb-b5b6-98d37090a666: ref 0 aborting False >Thread-15911678::INFO::2013-11-19 21:39:39,539::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911679::DEBUG::2013-11-19 21:39:39,550::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911679::INFO::2013-11-19 21:39:39,551::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911679::DEBUG::2013-11-19 21:39:39,552::task::495::TaskManager.Task::(_debug) Task 824e2afb-d628-4688-88df-e090356c6f9f: moving from state init -> state preparing >Thread-15911679::DEBUG::2013-11-19 21:39:39,553::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9215f267-ddf3-41e9-8367-51338108095a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911679::DEBUG::2013-11-19 21:39:39,554::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911679::DEBUG::2013-11-19 21:39:39,554::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911679::DEBUG::2013-11-19 21:39:39,555::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9215f267-ddf3-41e9-8367-51338108095a`::Granted request >Thread-15911679::DEBUG::2013-11-19 21:39:39,556::task::495::TaskManager.Task::(_debug) Task 824e2afb-d628-4688-88df-e090356c6f9f: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911679::DEBUG::2013-11-19 21:39:39,557::task::495::TaskManager.Task::(_debug) Task 824e2afb-d628-4688-88df-e090356c6f9f: ref 1 aborting False >Thread-15911679::WARNING::2013-11-19 21:39:39,559::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911679::DEBUG::2013-11-19 21:39:39,560::task::495::TaskManager.Task::(_debug) Task 824e2afb-d628-4688-88df-e090356c6f9f: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133801443328', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911679::DEBUG::2013-11-19 21:39:39,561::task::495::TaskManager.Task::(_debug) Task 824e2afb-d628-4688-88df-e090356c6f9f: moving from state preparing -> state finished >Thread-15911679::DEBUG::2013-11-19 21:39:39,561::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911679::DEBUG::2013-11-19 21:39:39,562::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911679::DEBUG::2013-11-19 21:39:39,563::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911679::DEBUG::2013-11-19 21:39:39,563::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911679::DEBUG::2013-11-19 21:39:39,564::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911679::DEBUG::2013-11-19 21:39:39,564::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911679::DEBUG::2013-11-19 21:39:39,565::task::495::TaskManager.Task::(_debug) Task 824e2afb-d628-4688-88df-e090356c6f9f: ref 0 aborting False >Thread-15911679::INFO::2013-11-19 21:39:39,566::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133801443328', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:39:40,947::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:41,479::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0707925 s, 14.5 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:39:43,165::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:39:43,494::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:44,009::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.06958 s, 14.7 MB/s\n'; <rc> = 0 >Thread-15911682::INFO::2013-11-19 21:39:44,658::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911682::DEBUG::2013-11-19 21:39:44,659::task::495::TaskManager.Task::(_debug) Task ee95603b-097b-447c-836d-3a4a21c0387e: moving from state init -> state preparing >Thread-15911682::DEBUG::2013-11-19 21:39:44,666::task::495::TaskManager.Task::(_debug) Task ee95603b-097b-447c-836d-3a4a21c0387e: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000414133071899', 'lastCheck': 1384897175.5449901, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142288208008', 'lastCheck': 1384897177.0786681, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897183.1659491, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000552892684937', 'lastCheck': 1384897174.5397, 'valid': True, 'code': 0}} >Thread-15911682::DEBUG::2013-11-19 21:39:44,681::task::495::TaskManager.Task::(_debug) Task ee95603b-097b-447c-836d-3a4a21c0387e: moving from state preparing -> state finished >Thread-15911682::DEBUG::2013-11-19 21:39:44,681::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911682::DEBUG::2013-11-19 21:39:44,682::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911682::DEBUG::2013-11-19 21:39:44,682::task::495::TaskManager.Task::(_debug) Task ee95603b-097b-447c-836d-3a4a21c0387e: ref 0 aborting False >Thread-15911682::INFO::2013-11-19 21:39:44,683::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000414133071899', 'lastCheck': 1384897175.5449901, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142288208008', 'lastCheck': 1384897177.0786681, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897183.1659491, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000552892684937', 'lastCheck': 1384897174.5397, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:39:46,025::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:46,583::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0945474 s, 10.8 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:39:48,599::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:49,123::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0699839 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:39:51,139::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15911687::DEBUG::2013-11-19 21:39:51,375::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911687::INFO::2013-11-19 21:39:51,377::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911687::DEBUG::2013-11-19 21:39:51,378::task::495::TaskManager.Task::(_debug) Task 1031e0e4-8771-413a-a451-867a1e552512: moving from state init -> state preparing >Thread-15911687::DEBUG::2013-11-19 21:39:51,379::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911687::DEBUG::2013-11-19 21:39:51,380::task::495::TaskManager.Task::(_debug) Task 1031e0e4-8771-413a-a451-867a1e552512: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911687::DEBUG::2013-11-19 21:39:51,381::task::495::TaskManager.Task::(_debug) Task 1031e0e4-8771-413a-a451-867a1e552512: moving from state preparing -> state finished >Thread-15911687::DEBUG::2013-11-19 21:39:51,381::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911687::DEBUG::2013-11-19 21:39:51,382::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911687::DEBUG::2013-11-19 21:39:51,383::task::495::TaskManager.Task::(_debug) Task 1031e0e4-8771-413a-a451-867a1e552512: ref 0 aborting False >Thread-15911687::INFO::2013-11-19 21:39:51,383::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911688::DEBUG::2013-11-19 21:39:51,395::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911688::INFO::2013-11-19 21:39:51,396::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911688::DEBUG::2013-11-19 21:39:51,397::task::495::TaskManager.Task::(_debug) Task 294f0695-5f71-4c5d-b221-53914532f235: moving from state init -> state preparing >Thread-15911688::DEBUG::2013-11-19 21:39:51,398::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3f116f03-09dc-4920-8474-ff6807372bda`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911688::DEBUG::2013-11-19 21:39:51,399::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911688::DEBUG::2013-11-19 21:39:51,400::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911688::DEBUG::2013-11-19 21:39:51,401::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3f116f03-09dc-4920-8474-ff6807372bda`::Granted request >Thread-15911688::DEBUG::2013-11-19 21:39:51,402::task::495::TaskManager.Task::(_debug) Task 294f0695-5f71-4c5d-b221-53914532f235: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911688::DEBUG::2013-11-19 21:39:51,403::task::495::TaskManager.Task::(_debug) Task 294f0695-5f71-4c5d-b221-53914532f235: ref 1 aborting False >Thread-15911688::WARNING::2013-11-19 21:39:51,405::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911688::DEBUG::2013-11-19 21:39:51,406::task::495::TaskManager.Task::(_debug) Task 294f0695-5f71-4c5d-b221-53914532f235: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133809831936', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911688::DEBUG::2013-11-19 21:39:51,407::task::495::TaskManager.Task::(_debug) Task 294f0695-5f71-4c5d-b221-53914532f235: moving from state preparing -> state finished >Thread-15911688::DEBUG::2013-11-19 21:39:51,408::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911688::DEBUG::2013-11-19 21:39:51,408::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911688::DEBUG::2013-11-19 21:39:51,409::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911688::DEBUG::2013-11-19 21:39:51,410::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911688::DEBUG::2013-11-19 21:39:51,410::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911688::DEBUG::2013-11-19 21:39:51,411::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911688::DEBUG::2013-11-19 21:39:51,412::task::495::TaskManager.Task::(_debug) Task 294f0695-5f71-4c5d-b221-53914532f235: ref 0 aborting False >Thread-15911688::INFO::2013-11-19 21:39:51,412::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133809831936', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:39:51,682::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0704483 s, 14.5 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:39:53,372::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:39:53,697::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:54,233::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0745386 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15911690::INFO::2013-11-19 21:39:54,882::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911690::DEBUG::2013-11-19 21:39:54,883::task::495::TaskManager.Task::(_debug) Task 694c8009-8a74-468c-a436-6ba2338cd86f: moving from state init -> state preparing >Thread-15911690::DEBUG::2013-11-19 21:39:54,884::task::495::TaskManager.Task::(_debug) Task 694c8009-8a74-468c-a436-6ba2338cd86f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000414133071899', 'lastCheck': 1384897185.7120061, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150799751282', 'lastCheck': 1384897187.25246, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897193.3728571, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000556945800781', 'lastCheck': 1384897184.7166281, 'valid': True, 'code': 0}} >Thread-15911690::DEBUG::2013-11-19 21:39:54,885::task::495::TaskManager.Task::(_debug) Task 694c8009-8a74-468c-a436-6ba2338cd86f: moving from state preparing -> state finished >Thread-15911690::DEBUG::2013-11-19 21:39:54,885::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911690::DEBUG::2013-11-19 21:39:54,886::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911690::DEBUG::2013-11-19 21:39:54,886::task::495::TaskManager.Task::(_debug) Task 694c8009-8a74-468c-a436-6ba2338cd86f: ref 0 aborting False >Thread-15911690::INFO::2013-11-19 21:39:54,887::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000414133071899', 'lastCheck': 1384897185.7120061, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150799751282', 'lastCheck': 1384897187.25246, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897193.3728571, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000556945800781', 'lastCheck': 1384897184.7166281, 'valid': True, 'code': 0}} >Thread-42838::DEBUG::2013-11-19 21:39:55,910::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:56,249::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-42838::DEBUG::2013-11-19 21:39:56,610::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:39:56,774::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0704507 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:39:58,789::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:39:59,333::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0707952 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:40:01,348::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15911695::DEBUG::2013-11-19 21:40:01,600::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911695::INFO::2013-11-19 21:40:01,607::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911695::DEBUG::2013-11-19 21:40:01,608::task::495::TaskManager.Task::(_debug) Task cf2b4ef0-d921-48a0-893a-84afe6656ae1: moving from state init -> state preparing >Thread-15911695::DEBUG::2013-11-19 21:40:01,609::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911695::DEBUG::2013-11-19 21:40:01,610::task::495::TaskManager.Task::(_debug) Task cf2b4ef0-d921-48a0-893a-84afe6656ae1: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911695::DEBUG::2013-11-19 21:40:01,611::task::495::TaskManager.Task::(_debug) Task cf2b4ef0-d921-48a0-893a-84afe6656ae1: moving from state preparing -> state finished >Thread-15911695::DEBUG::2013-11-19 21:40:01,611::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911695::DEBUG::2013-11-19 21:40:01,612::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911695::DEBUG::2013-11-19 21:40:01,612::task::495::TaskManager.Task::(_debug) Task cf2b4ef0-d921-48a0-893a-84afe6656ae1: ref 0 aborting False >Thread-15911695::INFO::2013-11-19 21:40:01,613::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911696::DEBUG::2013-11-19 21:40:01,625::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911696::INFO::2013-11-19 21:40:01,626::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911696::DEBUG::2013-11-19 21:40:01,627::task::495::TaskManager.Task::(_debug) Task 96a28e5e-c3be-4d80-b750-cf210486cd8d: moving from state init -> state preparing >Thread-15911696::DEBUG::2013-11-19 21:40:01,628::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a5939890-f5cf-417d-854b-faca580b5560`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911696::DEBUG::2013-11-19 21:40:01,629::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911696::DEBUG::2013-11-19 21:40:01,630::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911696::DEBUG::2013-11-19 21:40:01,630::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a5939890-f5cf-417d-854b-faca580b5560`::Granted request >Thread-15911696::DEBUG::2013-11-19 21:40:01,632::task::495::TaskManager.Task::(_debug) Task 96a28e5e-c3be-4d80-b750-cf210486cd8d: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911696::DEBUG::2013-11-19 21:40:01,632::task::495::TaskManager.Task::(_debug) Task 96a28e5e-c3be-4d80-b750-cf210486cd8d: ref 1 aborting False >Thread-15911696::WARNING::2013-11-19 21:40:01,635::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911696::DEBUG::2013-11-19 21:40:01,636::task::495::TaskManager.Task::(_debug) Task 96a28e5e-c3be-4d80-b750-cf210486cd8d: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133811929088', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911696::DEBUG::2013-11-19 21:40:01,637::task::495::TaskManager.Task::(_debug) Task 96a28e5e-c3be-4d80-b750-cf210486cd8d: moving from state preparing -> state finished >Thread-15911696::DEBUG::2013-11-19 21:40:01,638::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911696::DEBUG::2013-11-19 21:40:01,638::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911696::DEBUG::2013-11-19 21:40:01,639::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911696::DEBUG::2013-11-19 21:40:01,640::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911696::DEBUG::2013-11-19 21:40:01,640::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911696::DEBUG::2013-11-19 21:40:01,641::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911696::DEBUG::2013-11-19 21:40:01,641::task::495::TaskManager.Task::(_debug) Task 96a28e5e-c3be-4d80-b750-cf210486cd8d: ref 0 aborting False >Thread-15911696::INFO::2013-11-19 21:40:01,642::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133811929088', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Dummy-15911465::DEBUG::2013-11-19 21:40:01,893::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0702969 s, 14.6 MB/s\n'; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:40:03,596::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:40:03,908::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:40:04,451::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0703995 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15911698::INFO::2013-11-19 21:40:05,042::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911698::DEBUG::2013-11-19 21:40:05,043::task::495::TaskManager.Task::(_debug) Task e091f210-9f0e-4b61-9362-e76bc557f95e: moving from state init -> state preparing >Thread-15911698::DEBUG::2013-11-19 21:40:05,044::task::495::TaskManager.Task::(_debug) Task e091f210-9f0e-4b61-9362-e76bc557f95e: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000407934188843', 'lastCheck': 1384897196.6160581, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0021800994873', 'lastCheck': 1384897197.4660299, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897203.597055, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000550985336304', 'lastCheck': 1384897194.909667, 'valid': True, 'code': 0}} >Thread-15911698::DEBUG::2013-11-19 21:40:05,054::task::495::TaskManager.Task::(_debug) Task e091f210-9f0e-4b61-9362-e76bc557f95e: moving from state preparing -> state finished >Thread-15911698::DEBUG::2013-11-19 21:40:05,055::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911698::DEBUG::2013-11-19 21:40:05,055::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911698::DEBUG::2013-11-19 21:40:05,056::task::495::TaskManager.Task::(_debug) Task e091f210-9f0e-4b61-9362-e76bc557f95e: ref 0 aborting False >Thread-15911698::INFO::2013-11-19 21:40:05,056::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000407934188843', 'lastCheck': 1384897196.6160581, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0021800994873', 'lastCheck': 1384897197.4660299, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897203.597055, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000550985336304', 'lastCheck': 1384897194.909667, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:40:06,466::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:40:06,992::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0701655 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:40:09,006::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:40:09,537::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0700583 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:40:11,552::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:40:12,100::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0713147 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15911703::DEBUG::2013-11-19 21:40:12,195::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911703::INFO::2013-11-19 21:40:12,196::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911703::DEBUG::2013-11-19 21:40:12,197::task::495::TaskManager.Task::(_debug) Task 916e831c-c6fc-491e-82b5-b842f59ab446: moving from state init -> state preparing >Thread-15911703::DEBUG::2013-11-19 21:40:12,198::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911703::DEBUG::2013-11-19 21:40:12,198::task::495::TaskManager.Task::(_debug) Task 916e831c-c6fc-491e-82b5-b842f59ab446: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911703::DEBUG::2013-11-19 21:40:12,199::task::495::TaskManager.Task::(_debug) Task 916e831c-c6fc-491e-82b5-b842f59ab446: moving from state preparing -> state finished >Thread-15911703::DEBUG::2013-11-19 21:40:12,199::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911703::DEBUG::2013-11-19 21:40:12,200::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911703::DEBUG::2013-11-19 21:40:12,200::task::495::TaskManager.Task::(_debug) Task 916e831c-c6fc-491e-82b5-b842f59ab446: ref 0 aborting False >Thread-15911703::INFO::2013-11-19 21:40:12,201::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911704::DEBUG::2013-11-19 21:40:12,207::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911704::INFO::2013-11-19 21:40:12,208::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911704::DEBUG::2013-11-19 21:40:12,208::task::495::TaskManager.Task::(_debug) Task abbce858-a49b-45e5-8b28-d47549b46481: moving from state init -> state preparing >Thread-15911704::DEBUG::2013-11-19 21:40:12,210::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c4bfc036-f5e7-46ad-b748-f9501104c94c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15911704::DEBUG::2013-11-19 21:40:12,210::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15911704::DEBUG::2013-11-19 21:40:12,211::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15911704::DEBUG::2013-11-19 21:40:12,212::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c4bfc036-f5e7-46ad-b748-f9501104c94c`::Granted request >Thread-15911704::DEBUG::2013-11-19 21:40:12,213::task::495::TaskManager.Task::(_debug) Task abbce858-a49b-45e5-8b28-d47549b46481: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15911704::DEBUG::2013-11-19 21:40:12,213::task::495::TaskManager.Task::(_debug) Task abbce858-a49b-45e5-8b28-d47549b46481: ref 1 aborting False >Thread-15911704::WARNING::2013-11-19 21:40:12,215::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911704::DEBUG::2013-11-19 21:40:12,216::task::495::TaskManager.Task::(_debug) Task abbce858-a49b-45e5-8b28-d47549b46481: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133811929088', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-15911704::DEBUG::2013-11-19 21:40:12,217::task::495::TaskManager.Task::(_debug) Task abbce858-a49b-45e5-8b28-d47549b46481: moving from state preparing -> state finished >Thread-15911704::DEBUG::2013-11-19 21:40:12,218::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911704::DEBUG::2013-11-19 21:40:12,218::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911704::DEBUG::2013-11-19 21:40:12,219::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911704::DEBUG::2013-11-19 21:40:12,220::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911704::DEBUG::2013-11-19 21:40:12,220::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911704::DEBUG::2013-11-19 21:40:12,221::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911704::DEBUG::2013-11-19 21:40:12,221::task::495::TaskManager.Task::(_debug) Task abbce858-a49b-45e5-8b28-d47549b46481: ref 0 aborting False >Thread-15911704::INFO::2013-11-19 21:40:12,222::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 285}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '133811929088', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'alerts': []}}} >Thread-1024767::ERROR::2013-11-19 21:40:13,789::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Dummy-15911465::DEBUG::2013-11-19 21:40:14,116::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:40:14,639::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0705105 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15911706::INFO::2013-11-19 21:40:15,295::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911706::DEBUG::2013-11-19 21:40:15,296::task::495::TaskManager.Task::(_debug) Task 17934326-f1fe-4f62-9658-59611c9286c4: moving from state init -> state preparing >Thread-15911706::DEBUG::2013-11-19 21:40:15,297::task::495::TaskManager.Task::(_debug) Task 17934326-f1fe-4f62-9658-59611c9286c4: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000596046447754', 'lastCheck': 1384897206.684478, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148606300354', 'lastCheck': 1384897207.687403, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897213.7906649, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000538110733032', 'lastCheck': 1384897205.133152, 'valid': True, 'code': 0}} >Thread-15911706::DEBUG::2013-11-19 21:40:15,297::task::495::TaskManager.Task::(_debug) Task 17934326-f1fe-4f62-9658-59611c9286c4: moving from state preparing -> state finished >Thread-15911706::DEBUG::2013-11-19 21:40:15,298::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911706::DEBUG::2013-11-19 21:40:15,299::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911706::DEBUG::2013-11-19 21:40:15,299::task::495::TaskManager.Task::(_debug) Task 17934326-f1fe-4f62-9658-59611c9286c4: ref 0 aborting False >Thread-15911706::INFO::2013-11-19 21:40:15,300::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000596046447754', 'lastCheck': 1384897206.684478, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148606300354', 'lastCheck': 1384897207.687403, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897213.7906649, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000538110733032', 'lastCheck': 1384897205.133152, 'valid': True, 'code': 0}} >Dummy-15911465::DEBUG::2013-11-19 21:40:16,654::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15911465::DEBUG::2013-11-19 21:40:17,110::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0450239 s, 22.7 MB/s\n'; <rc> = 0 >Dummy-15911465::DEBUG::2013-11-19 21:40:19,130::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15911710::DEBUG::2013-11-19 21:40:19,641::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911710::INFO::2013-11-19 21:40:19,644::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksStatuses, args: () >Thread-15911710::DEBUG::2013-11-19 21:40:19,644::task::495::TaskManager.Task::(_debug) Task 711c83ea-155c-453d-9404-c8069898bba6: moving from state init -> state preparing >Thread-15911710::DEBUG::2013-11-19 21:40:19,645::taskManager::92::TaskManager::(getAllTasksStatuses) Entry. >Thread-15911710::DEBUG::2013-11-19 21:40:19,647::taskManager::97::TaskManager::(getAllTasksStatuses) Return: {} >Thread-15911710::DEBUG::2013-11-19 21:40:19,648::task::495::TaskManager.Task::(_debug) Task 711c83ea-155c-453d-9404-c8069898bba6: finished: {'allTasksStatus': {}} >Thread-15911710::DEBUG::2013-11-19 21:40:19,649::task::495::TaskManager.Task::(_debug) Task 711c83ea-155c-453d-9404-c8069898bba6: moving from state preparing -> state finished >Thread-15911710::DEBUG::2013-11-19 21:40:19,650::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911710::DEBUG::2013-11-19 21:40:19,652::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911710::DEBUG::2013-11-19 21:40:19,653::task::495::TaskManager.Task::(_debug) Task 711c83ea-155c-453d-9404-c8069898bba6: ref 0 aborting False >Thread-15911710::INFO::2013-11-19 21:40:19,654::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksStatuses, Return response: {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}} >Dummy-15911465::DEBUG::2013-11-19 21:40:19,695::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918885 s, 11.1 MB/s\n'; <rc> = 0 >Thread-15911711::DEBUG::2013-11-19 21:40:19,813::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911711::INFO::2013-11-19 21:40:19,814::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: deactivateStorageDomain, args: ( sdUUID=5d7b5178-3790-472c-a13b-b481eb27a61e spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 msdUUID=00000000-0000-0000-0000-000000000000 masterVersion=1) >Thread-15911711::DEBUG::2013-11-19 21:40:19,814::task::495::TaskManager.Task::(_debug) Task 8ac9ff13-f628-40fa-a83e-a0a57719328a: moving from state init -> state preparing >Thread-15911711::DEBUG::2013-11-19 21:40:19,816::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a3a47e79-d594-4803-b2b1-6d6b2d53afb4`::Request was made in '/usr/share/vdsm/storage/spm.py' line '1112' at 'public_deactivateStorageDomain' >Thread-15911711::DEBUG::2013-11-19 21:40:19,816::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >Thread-15911711::DEBUG::2013-11-19 21:40:19,817::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >Thread-15911711::DEBUG::2013-11-19 21:40:19,818::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a3a47e79-d594-4803-b2b1-6d6b2d53afb4`::Granted request >Thread-15911711::DEBUG::2013-11-19 21:40:19,819::task::495::TaskManager.Task::(_debug) Task 8ac9ff13-f628-40fa-a83e-a0a57719328a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (exclusive) >Thread-15911711::DEBUG::2013-11-19 21:40:19,819::task::495::TaskManager.Task::(_debug) Task 8ac9ff13-f628-40fa-a83e-a0a57719328a: ref 1 aborting False >Thread-15911711::DEBUG::2013-11-19 21:40:19,820::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`d0c9aece-3b3a-4892-92d4-5c54c0a80a50`::Request was made in '/usr/share/vdsm/storage/spm.py' line '1113' at 'public_deactivateStorageDomain' >Thread-15911711::DEBUG::2013-11-19 21:40:19,821::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' for lock type 'exclusive' >Thread-15911711::DEBUG::2013-11-19 21:40:19,822::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' is free. Now locking as 'exclusive' (1 active user) >Thread-15911711::DEBUG::2013-11-19 21:40:19,822::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`d0c9aece-3b3a-4892-92d4-5c54c0a80a50`::Granted request >Thread-15911711::DEBUG::2013-11-19 21:40:19,823::task::495::TaskManager.Task::(_debug) Task 8ac9ff13-f628-40fa-a83e-a0a57719328a: _resourcesAcquired: Storage.5d7b5178-3790-472c-a13b-b481eb27a61e (exclusive) >Thread-15911711::DEBUG::2013-11-19 21:40:19,824::task::495::TaskManager.Task::(_debug) Task 8ac9ff13-f628-40fa-a83e-a0a57719328a: ref 1 aborting False >Thread-15911711::INFO::2013-11-19 21:40:19,824::sp::775::Storage.StoragePool::(deactivateSD) sdUUID=5d7b5178-3790-472c-a13b-b481eb27a61e spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 new_msdUUID=00000000-0000-0000-0000-000000000000 >Thread-15911711::INFO::2013-11-19 21:40:19,825::fileSD::176::Storage.StorageDomain::(validate) sdUUID=5d7b5178-3790-472c-a13b-b481eb27a61e >Thread-15911711::DEBUG::2013-11-19 21:40:19,826::persistentDict::204::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=[] >Thread-15911711::WARNING::2013-11-19 21:40:19,827::persistentDict::226::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >Thread-15911711::ERROR::2013-11-19 21:40:19,827::task::868::TaskManager.Task::(_setError) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 876, in _run > File "/usr/share/vdsm/storage/spm.py", line 115, in run > File "/usr/share/vdsm/storage/spm.py", line 1115, in public_deactivateStorageDomain > File "/usr/share/vdsm/storage/sp.py", line 798, in deactivateSD > File "/usr/share/vdsm/storage/sd.py", line 666, in isBackup > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15911711::DEBUG::2013-11-19 21:40:19,836::task::495::TaskManager.Task::(_debug) Task 8ac9ff13-f628-40fa-a83e-a0a57719328a: Task._run: 8ac9ff13-f628-40fa-a83e-a0a57719328a ('5d7b5178-3790-472c-a13b-b481eb27a61e', 'c4e5889a-348d-11e2-83fe-9f72369eab30', '00000000-0000-0000-0000-000000000000', 1) {} failed - stopping task >Thread-15911711::DEBUG::2013-11-19 21:40:19,837::task::495::TaskManager.Task::(_debug) Task 8ac9ff13-f628-40fa-a83e-a0a57719328a: stopping in state preparing (force False) >Thread-15911711::DEBUG::2013-11-19 21:40:19,837::task::495::TaskManager.Task::(_debug) Task 8ac9ff13-f628-40fa-a83e-a0a57719328a: ref 1 aborting True >Thread-15911711::INFO::2013-11-19 21:40:19,838::task::1171::TaskManager.Task::(prepare) aborting: Task is aborted: "'CLASS'" - code 100 >Thread-15911711::DEBUG::2013-11-19 21:40:19,838::task::495::TaskManager.Task::(_debug) Task 8ac9ff13-f628-40fa-a83e-a0a57719328a: Prepare: aborted: 'CLASS' >Thread-15911711::DEBUG::2013-11-19 21:40:19,839::task::495::TaskManager.Task::(_debug) Task 8ac9ff13-f628-40fa-a83e-a0a57719328a: ref 0 aborting True >Thread-15911711::DEBUG::2013-11-19 21:40:19,839::task::495::TaskManager.Task::(_debug) Task 8ac9ff13-f628-40fa-a83e-a0a57719328a: Task._doAbort: force False >Thread-15911711::DEBUG::2013-11-19 21:40:19,840::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911711::DEBUG::2013-11-19 21:40:19,841::task::495::TaskManager.Task::(_debug) Task 8ac9ff13-f628-40fa-a83e-a0a57719328a: moving from state preparing -> state aborting >Thread-15911711::DEBUG::2013-11-19 21:40:19,841::task::495::TaskManager.Task::(_debug) Task 8ac9ff13-f628-40fa-a83e-a0a57719328a: _aborting: recover policy none >Thread-15911711::DEBUG::2013-11-19 21:40:19,842::task::495::TaskManager.Task::(_debug) Task 8ac9ff13-f628-40fa-a83e-a0a57719328a: moving from state aborting -> state failed >Thread-15911711::DEBUG::2013-11-19 21:40:19,842::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>, 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e': < ResourceRef 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e', isValid: 'True' obj: 'None'>} >Thread-15911711::DEBUG::2013-11-19 21:40:19,843::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911711::DEBUG::2013-11-19 21:40:19,843::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911711::DEBUG::2013-11-19 21:40:19,844::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911711::DEBUG::2013-11-19 21:40:19,845::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911711::DEBUG::2013-11-19 21:40:19,845::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911711::DEBUG::2013-11-19 21:40:19,846::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' >Thread-15911711::DEBUG::2013-11-19 21:40:19,846::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' (0 active users) >Thread-15911711::DEBUG::2013-11-19 21:40:19,847::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e' is free, finding out if anyone is waiting for it. >Thread-15911711::DEBUG::2013-11-19 21:40:19,847::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.5d7b5178-3790-472c-a13b-b481eb27a61e', Clearing records. >Thread-15911711::ERROR::2013-11-19 21:40:19,848::dispatcher::106::Storage.Dispatcher.Protect::(run) 'CLASS' >Thread-15911711::ERROR::2013-11-19 21:40:19,849::dispatcher::107::Storage.Dispatcher.Protect::(run) Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 96, in run > File "/usr/share/vdsm/storage/task.py", line 1178, in prepare >KeyError: 'CLASS' > >Thread-15911712::DEBUG::2013-11-19 21:40:19,880::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911712::INFO::2013-11-19 21:40:19,881::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911712::DEBUG::2013-11-19 21:40:19,881::task::495::TaskManager.Task::(_debug) Task 8a7a83cc-688c-4dd2-a325-c81b39a5eb0f: moving from state init -> state preparing >Thread-15911712::DEBUG::2013-11-19 21:40:19,882::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=285 spmId=2 >Thread-15911712::DEBUG::2013-11-19 21:40:19,883::task::495::TaskManager.Task::(_debug) Task 8a7a83cc-688c-4dd2-a325-c81b39a5eb0f: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911712::DEBUG::2013-11-19 21:40:19,883::task::495::TaskManager.Task::(_debug) Task 8a7a83cc-688c-4dd2-a325-c81b39a5eb0f: moving from state preparing -> state finished >Thread-15911712::DEBUG::2013-11-19 21:40:19,884::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911712::DEBUG::2013-11-19 21:40:19,884::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911712::DEBUG::2013-11-19 21:40:19,885::task::495::TaskManager.Task::(_debug) Task 8a7a83cc-688c-4dd2-a325-c81b39a5eb0f: ref 0 aborting False >Thread-15911712::INFO::2013-11-19 21:40:19,885::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 285}} >Thread-15911713::DEBUG::2013-11-19 21:40:19,903::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911713::INFO::2013-11-19 21:40:19,908::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksStatuses, args: () >Thread-15911713::DEBUG::2013-11-19 21:40:19,910::task::495::TaskManager.Task::(_debug) Task 37e2980c-7b77-44ec-801c-595371bee316: moving from state init -> state preparing >Thread-15911713::DEBUG::2013-11-19 21:40:19,913::taskManager::92::TaskManager::(getAllTasksStatuses) Entry. >Thread-15911713::DEBUG::2013-11-19 21:40:19,915::taskManager::97::TaskManager::(getAllTasksStatuses) Return: {} >Thread-15911713::DEBUG::2013-11-19 21:40:19,916::task::495::TaskManager.Task::(_debug) Task 37e2980c-7b77-44ec-801c-595371bee316: finished: {'allTasksStatus': {}} >Thread-15911713::DEBUG::2013-11-19 21:40:19,916::task::495::TaskManager.Task::(_debug) Task 37e2980c-7b77-44ec-801c-595371bee316: moving from state preparing -> state finished >Thread-15911713::DEBUG::2013-11-19 21:40:19,917::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911713::DEBUG::2013-11-19 21:40:19,917::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911713::DEBUG::2013-11-19 21:40:19,918::task::495::TaskManager.Task::(_debug) Task 37e2980c-7b77-44ec-801c-595371bee316: ref 0 aborting False >Thread-15911713::INFO::2013-11-19 21:40:19,919::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksStatuses, Return response: {'status': {'message': 'OK', 'code': 0}, 'allTasksStatus': {}} >Thread-15911714::DEBUG::2013-11-19 21:40:19,924::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15911714::INFO::2013-11-19 21:40:19,925::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: spmStop, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15911714::DEBUG::2013-11-19 21:40:19,925::task::495::TaskManager.Task::(_debug) Task d1c0b138-4d14-4d13-81d5-17d287e303d5: moving from state init -> state preparing >Thread-15911714::DEBUG::2013-11-19 21:40:19,926::spm::547::Storage.SPM::(public_spmStop) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 >Thread-15911714::DEBUG::2013-11-19 21:40:19,926::taskManager::105::TaskManager::(getAllTasks) Entry. >Thread-15911714::DEBUG::2013-11-19 21:40:19,927::taskManager::110::TaskManager::(getAllTasks) Return: {} >Thread-15911714::DEBUG::2013-11-19 21:40:19,928::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c6ae2c6a-fbfb-4191-833f-d4f2ed7ee2be`::Request was made in '/usr/share/vdsm/storage/spm.py' line '573' at 'public_spmStop' >Thread-15911714::DEBUG::2013-11-19 21:40:19,928::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >Thread-15911714::DEBUG::2013-11-19 21:40:19,929::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >Thread-15911714::DEBUG::2013-11-19 21:40:19,930::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c6ae2c6a-fbfb-4191-833f-d4f2ed7ee2be`::Granted request >Thread-15911714::DEBUG::2013-11-19 21:40:19,931::task::495::TaskManager.Task::(_debug) Task d1c0b138-4d14-4d13-81d5-17d287e303d5: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (exclusive) >Thread-15911714::DEBUG::2013-11-19 21:40:19,931::task::495::TaskManager.Task::(_debug) Task d1c0b138-4d14-4d13-81d5-17d287e303d5: ref 1 aborting False >Thread-15911714::DEBUG::2013-11-19 21:40:19,932::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a4315281-a910-4449-8ad0-8b33d2cec479`::Request was made in '/usr/share/vdsm/storage/spm.py' line '584' at '_stop' >Thread-15911714::DEBUG::2013-11-19 21:40:19,933::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >Thread-15911714::DEBUG::2013-11-19 21:40:19,933::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >Thread-15911714::DEBUG::2013-11-19 21:40:19,934::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a4315281-a910-4449-8ad0-8b33d2cec479`::Granted request >Thread-15911714::DEBUG::2013-11-19 21:40:19,945::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`000c26ed-f9bf-4449-94f4-6f72df1d0563`::Request was made in '/usr/share/vdsm/storage/spm.py' line '601' at '_stop' >Thread-15911714::DEBUG::2013-11-19 21:40:19,945::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' for lock type 'exclusive' >Thread-15911714::DEBUG::2013-11-19 21:40:19,946::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' is free. Now locking as 'exclusive' (1 active user) >Thread-15911714::DEBUG::2013-11-19 21:40:19,947::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e`ReqID=`000c26ed-f9bf-4449-94f4-6f72df1d0563`::Granted request >Thread-15911714::DEBUG::2013-11-19 21:40:19,948::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' >Thread-15911714::DEBUG::2013-11-19 21:40:19,949::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' (0 active users) >Thread-15911714::DEBUG::2013-11-19 21:40:19,949::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e' is free, finding out if anyone is waiting for it. >Thread-15911714::DEBUG::2013-11-19 21:40:19,950::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_5d7b5178-3790-472c-a13b-b481eb27a61e', Clearing records. >Thread-15911714::DEBUG::2013-11-19 21:40:19,951::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911714::DEBUG::2013-11-19 21:40:19,951::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911714::DEBUG::2013-11-19 21:40:19,952::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911714::DEBUG::2013-11-19 21:40:19,952::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911714::DEBUG::2013-11-19 21:40:19,953::taskManager::118::TaskManager::(unloadTasks) Entry. >Thread-15911714::DEBUG::2013-11-19 21:40:19,954::taskManager::122::TaskManager::(unloadTasks) Return >Thread-15911714::DEBUG::2013-11-19 21:40:19,956::spm::195::Storage.SPM::(__cleanupMasterMount) unmounting /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master >Thread-15911714::DEBUG::2013-11-19 21:40:20,096::fileUtils::112::Storage.Misc.excCmd::(umount) '/usr/bin/sudo -n /bin/umount -f /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15911714::DEBUG::2013-11-19 21:40:20,442::fileUtils::112::Storage.Misc.excCmd::(umount) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911714::DEBUG::2013-11-19 21:40:20,445::spm::290::Storage.SPM::(__cleanupSPM) cleaning up SPM: c4e5889a-348d-11e2-83fe-9f72369eab30 >Thread-15911714::DEBUG::2013-11-19 21:40:20,446::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >Thread-15911714::DEBUG::2013-11-19 21:40:20,447::persistentDict::146::Storage.PersistentDict::(transaction) Flushing changes >Thread-15911714::DEBUG::2013-11-19 21:40:20,448::persistentDict::265::Storage.PersistentDict::(flush) about to write lines (VGTagMetadataRW)=[u'CLASS=Data', u'DESCRIPTION=OS', u'IOOPTIMEOUTSEC=10', u'LEASERETRIES=3', u'LEASETIMESEC=60', u'LOCKPOLICY=', u'LOCKRENEWALINTERVALSEC=5', u'LOGBLKSIZE=512', u'MASTER_VERSION=1', u'PHYBLKSIZE=512', u'POOL_DESCRIPTION=Default', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_ID=-1', u'POOL_SPM_LVER=285', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'ROLE=Master', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'TYPE=FCP', u'VERSION=2', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', '_SHA_CKSUM=90a91eb94d806053a3001427d0dfd2783a6d5bb4'] >Thread-15911714::DEBUG::2013-11-19 21:40:20,449::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15911714::DEBUG::2013-11-19 21:40:20,450::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15911714::DEBUG::2013-11-19 21:40:20,451::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15911714::DEBUG::2013-11-19 21:40:20,451::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15911714::DEBUG::2013-11-19 21:40:20,452::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15911714::DEBUG::2013-11-19 21:40:20,453::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15911714::DEBUG::2013-11-19 21:40:21,090::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911714::DEBUG::2013-11-19 21:40:21,094::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15911714::DEBUG::2013-11-19 21:40:21,096::blockSD::145::storage.Metadata.VGTagMetadataRW::(writelines) Updating metadata adding=MDT_POOL_SPM_ID=-1, MDT__SHA_CKSUM=90a91eb94d806053a3001427d0dfd2783a6d5bb4 removing=MDT__SHA_CKSUM=901e51f2c0f3aefcf8d2e501fb4543e2352ea1bc, MDT_POOL_SPM_ID=2 >Thread-15911714::DEBUG::2013-11-19 21:40:21,097::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --deltag MDT_POOL_SPM_ID=2 --deltag MDT__SHA_CKSUM=901e51f2c0f3aefcf8d2e501fb4543e2352ea1bc --addtag MDT__SHA_CKSUM=90a91eb94d806053a3001427d0dfd2783a6d5bb4 --addtag MDT_POOL_SPM_ID=-1 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15911465::INFO::2013-11-19 21:40:21,710::storage_mailbox::673::Storage.MailBox.SpmMailMonitor::(run) SPM_MailMonitor - Incoming mail monitoring thread stopped >Thread-15911714::DEBUG::2013-11-19 21:40:21,749::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911714::DEBUG::2013-11-19 21:40:21,751::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15911714::DEBUG::2013-11-19 21:40:21,751::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15911714::DEBUG::2013-11-19 21:40:21,752::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >Thread-15911714::INFO::2013-11-19 21:40:21,753::safelease::83::ClusterLock::(release) Releasing cluster lock for domain 6b892067-ac7e-4039-a7be-5b7387525b16 >Thread-15911714::DEBUG::2013-11-19 21:40:21,754::safelease::84::Storage.Misc.excCmd::(release) '/usr/libexec/vdsm/spmstop.sh 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd /usr/libexec/vdsm/) >Thread-15911714::DEBUG::2013-11-19 21:40:23,224::safelease::84::Storage.Misc.excCmd::(release) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911714::DEBUG::2013-11-19 21:40:23,226::safelease::87::ClusterLock::(release) Cluster lock released successfully >Thread-15911714::DEBUG::2013-11-19 21:40:23,227::task::495::TaskManager.Task::(_debug) Task d1c0b138-4d14-4d13-81d5-17d287e303d5: finished: None >Thread-15911714::DEBUG::2013-11-19 21:40:23,227::task::495::TaskManager.Task::(_debug) Task d1c0b138-4d14-4d13-81d5-17d287e303d5: moving from state preparing -> state finished >Thread-15911714::DEBUG::2013-11-19 21:40:23,228::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15911714::DEBUG::2013-11-19 21:40:23,228::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911714::DEBUG::2013-11-19 21:40:23,229::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15911714::DEBUG::2013-11-19 21:40:23,230::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15911714::DEBUG::2013-11-19 21:40:23,231::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15911714::DEBUG::2013-11-19 21:40:23,231::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15911714::DEBUG::2013-11-19 21:40:23,232::task::495::TaskManager.Task::(_debug) Task d1c0b138-4d14-4d13-81d5-17d287e303d5: ref 0 aborting False >Thread-15911714::INFO::2013-11-19 21:40:23,233::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: spmStop, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:40:23,966::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911717::INFO::2013-11-19 21:40:25,415::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911717::DEBUG::2013-11-19 21:40:25,416::task::495::TaskManager.Task::(_debug) Task 67bda3b9-ee27-4629-a306-270cc0fbbf0b: moving from state init -> state preparing >Thread-15911717::WARNING::2013-11-19 21:40:25,417::hsm::1895::Storage.HSM::(public_repoStats) repoStats detected invalid master:6b892067-ac7e-4039-a7be-5b7387525b16 {'finish': 1384897216.911788, 'mdasize': '134479872', 'mdathreshold': True, 'mdavalid': True, 'diskfree': '1142595518464', 'disktotal': '3595827150848', 'result': {'delay': '0.000508069992065', 'lastCheck': 1384897216.911788, 'valid': True, 'code': 0}, 'masterValidate': {'mount': True, 'valid': True}, 'mdafree': '67205632'} >Thread-15911717::DEBUG::2013-11-19 21:40:25,417::task::495::TaskManager.Task::(_debug) Task 67bda3b9-ee27-4629-a306-270cc0fbbf0b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000508069992065', 'lastCheck': 1384897216.911788, 'valid': True, 'code': 355}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146198272705', 'lastCheck': 1384897217.9052651, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897223.9675469, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000401973724365', 'lastCheck': 1384897225.364517, 'valid': True, 'code': 0}} >Thread-15911717::DEBUG::2013-11-19 21:40:25,428::task::495::TaskManager.Task::(_debug) Task 67bda3b9-ee27-4629-a306-270cc0fbbf0b: moving from state preparing -> state finished >Thread-15911717::DEBUG::2013-11-19 21:40:25,429::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911717::DEBUG::2013-11-19 21:40:25,429::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911717::DEBUG::2013-11-19 21:40:25,430::task::495::TaskManager.Task::(_debug) Task 67bda3b9-ee27-4629-a306-270cc0fbbf0b: ref 0 aborting False >Thread-15911717::INFO::2013-11-19 21:40:25,431::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000508069992065', 'lastCheck': 1384897216.911788, 'valid': True, 'code': 355}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146198272705', 'lastCheck': 1384897217.9052651, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897223.9675469, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000401973724365', 'lastCheck': 1384897225.364517, 'valid': True, 'code': 0}} >Thread-42838::DEBUG::2013-11-19 21:40:26,964::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-42838::DEBUG::2013-11-19 21:40:26,965::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 21:40:27,554::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-42838::DEBUG::2013-11-19 21:40:27,558::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1024767::ERROR::2013-11-19 21:40:34,186::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911723::INFO::2013-11-19 21:40:35,578::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911723::DEBUG::2013-11-19 21:40:35,579::task::495::TaskManager.Task::(_debug) Task 98788fe2-be2d-4d56-abf4-92907deab943: moving from state init -> state preparing >Thread-15911723::DEBUG::2013-11-19 21:40:35,580::task::495::TaskManager.Task::(_debug) Task 98788fe2-be2d-4d56-abf4-92907deab943: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000333070755005', 'lastCheck': 1384897227.560931, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00130605697632', 'lastCheck': 1384897227.969564, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897234.1881471, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000515937805176', 'lastCheck': 1384897235.375993, 'valid': True, 'code': 0}} >Thread-15911723::DEBUG::2013-11-19 21:40:35,585::task::495::TaskManager.Task::(_debug) Task 98788fe2-be2d-4d56-abf4-92907deab943: moving from state preparing -> state finished >Thread-15911723::DEBUG::2013-11-19 21:40:35,585::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911723::DEBUG::2013-11-19 21:40:35,586::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911723::DEBUG::2013-11-19 21:40:35,587::task::495::TaskManager.Task::(_debug) Task 98788fe2-be2d-4d56-abf4-92907deab943: ref 0 aborting False >Thread-15911723::INFO::2013-11-19 21:40:35,587::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000333070755005', 'lastCheck': 1384897227.560931, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00130605697632', 'lastCheck': 1384897227.969564, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897234.1881471, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000515937805176', 'lastCheck': 1384897235.375993, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:40:44,200::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911729::INFO::2013-11-19 21:40:45,689::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911729::DEBUG::2013-11-19 21:40:45,690::task::495::TaskManager.Task::(_debug) Task 6747af37-85c5-45de-aa15-397ae797f486: moving from state init -> state preparing >Thread-15911729::DEBUG::2013-11-19 21:40:45,691::task::495::TaskManager.Task::(_debug) Task 6747af37-85c5-45de-aa15-397ae797f486: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000402927398682', 'lastCheck': 1384897237.574053, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00143504142761', 'lastCheck': 1384897237.9851861, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897244.201607, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000426054000854', 'lastCheck': 1384897245.387563, 'valid': True, 'code': 0}} >Thread-15911729::DEBUG::2013-11-19 21:40:45,697::task::495::TaskManager.Task::(_debug) Task 6747af37-85c5-45de-aa15-397ae797f486: moving from state preparing -> state finished >Thread-15911729::DEBUG::2013-11-19 21:40:45,697::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911729::DEBUG::2013-11-19 21:40:45,698::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911729::DEBUG::2013-11-19 21:40:45,698::task::495::TaskManager.Task::(_debug) Task 6747af37-85c5-45de-aa15-397ae797f486: ref 0 aborting False >Thread-15911729::INFO::2013-11-19 21:40:45,699::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000402927398682', 'lastCheck': 1384897237.574053, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00143504142761', 'lastCheck': 1384897237.9851861, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897244.201607, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000426054000854', 'lastCheck': 1384897245.387563, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:40:54,214::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911735::INFO::2013-11-19 21:40:55,846::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911735::DEBUG::2013-11-19 21:40:55,846::task::495::TaskManager.Task::(_debug) Task 89ffaf2f-9202-4e27-bb27-98b9387e7b94: moving from state init -> state preparing >Thread-15911735::DEBUG::2013-11-19 21:40:55,847::task::495::TaskManager.Task::(_debug) Task 89ffaf2f-9202-4e27-bb27-98b9387e7b94: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000283002853394', 'lastCheck': 1384897247.586884, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00138282775879', 'lastCheck': 1384897247.999289, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897254.215116, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000448942184448', 'lastCheck': 1384897255.3991621, 'valid': True, 'code': 0}} >Thread-15911735::DEBUG::2013-11-19 21:40:55,856::task::495::TaskManager.Task::(_debug) Task 89ffaf2f-9202-4e27-bb27-98b9387e7b94: moving from state preparing -> state finished >Thread-15911735::DEBUG::2013-11-19 21:40:55,857::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911735::DEBUG::2013-11-19 21:40:55,858::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911735::DEBUG::2013-11-19 21:40:55,858::task::495::TaskManager.Task::(_debug) Task 89ffaf2f-9202-4e27-bb27-98b9387e7b94: ref 0 aborting False >Thread-15911735::INFO::2013-11-19 21:40:55,859::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000283002853394', 'lastCheck': 1384897247.586884, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00138282775879', 'lastCheck': 1384897247.999289, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897254.215116, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000448942184448', 'lastCheck': 1384897255.3991621, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:41:04,227::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911741::INFO::2013-11-19 21:41:06,233::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911741::DEBUG::2013-11-19 21:41:06,234::task::495::TaskManager.Task::(_debug) Task 6e52a5a9-5519-427c-af78-f596a06c3024: moving from state init -> state preparing >Thread-15911741::DEBUG::2013-11-19 21:41:06,235::task::495::TaskManager.Task::(_debug) Task 6e52a5a9-5519-427c-af78-f596a06c3024: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000331878662109', 'lastCheck': 1384897257.5987539, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00206112861633', 'lastCheck': 1384897258.0155859, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897264.2284961, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000365972518921', 'lastCheck': 1384897265.4106901, 'valid': True, 'code': 0}} >Thread-15911741::DEBUG::2013-11-19 21:41:06,243::task::495::TaskManager.Task::(_debug) Task 6e52a5a9-5519-427c-af78-f596a06c3024: moving from state preparing -> state finished >Thread-15911741::DEBUG::2013-11-19 21:41:06,243::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911741::DEBUG::2013-11-19 21:41:06,244::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911741::DEBUG::2013-11-19 21:41:06,245::task::495::TaskManager.Task::(_debug) Task 6e52a5a9-5519-427c-af78-f596a06c3024: ref 0 aborting False >Thread-15911741::INFO::2013-11-19 21:41:06,246::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000331878662109', 'lastCheck': 1384897257.5987539, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00206112861633', 'lastCheck': 1384897258.0155859, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897264.2284961, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000365972518921', 'lastCheck': 1384897265.4106901, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:41:14,241::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911747::INFO::2013-11-19 21:41:17,290::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911747::DEBUG::2013-11-19 21:41:17,290::task::495::TaskManager.Task::(_debug) Task 9aac206d-f5c0-42da-8df8-401652ad34b8: moving from state init -> state preparing >Thread-15911747::DEBUG::2013-11-19 21:41:17,291::task::495::TaskManager.Task::(_debug) Task 9aac206d-f5c0-42da-8df8-401652ad34b8: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00109004974365', 'lastCheck': 1384897267.613065, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0494320392609', 'lastCheck': 1384897268.079035, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897274.2418351, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000361204147339', 'lastCheck': 1384897275.422224, 'valid': True, 'code': 0}} >Thread-15911747::DEBUG::2013-11-19 21:41:17,300::task::495::TaskManager.Task::(_debug) Task 9aac206d-f5c0-42da-8df8-401652ad34b8: moving from state preparing -> state finished >Thread-15911747::DEBUG::2013-11-19 21:41:17,300::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911747::DEBUG::2013-11-19 21:41:17,301::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911747::DEBUG::2013-11-19 21:41:17,301::task::495::TaskManager.Task::(_debug) Task 9aac206d-f5c0-42da-8df8-401652ad34b8: ref 0 aborting False >Thread-15911747::INFO::2013-11-19 21:41:17,306::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00109004974365', 'lastCheck': 1384897267.613065, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0494320392609', 'lastCheck': 1384897268.079035, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897274.2418351, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000361204147339', 'lastCheck': 1384897275.422224, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:41:24,254::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911753::INFO::2013-11-19 21:41:27,447::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911753::DEBUG::2013-11-19 21:41:27,447::task::495::TaskManager.Task::(_debug) Task 2b899450-9168-4b0d-b984-cbfd9aabdcab: moving from state init -> state preparing >Thread-15911753::DEBUG::2013-11-19 21:41:27,448::task::495::TaskManager.Task::(_debug) Task 2b899450-9168-4b0d-b984-cbfd9aabdcab: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000324964523315', 'lastCheck': 1384897277.6257949, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00133109092712', 'lastCheck': 1384897278.0942831, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897284.255214, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000363111495972', 'lastCheck': 1384897285.4337831, 'valid': True, 'code': 0}} >Thread-15911753::DEBUG::2013-11-19 21:41:27,449::task::495::TaskManager.Task::(_debug) Task 2b899450-9168-4b0d-b984-cbfd9aabdcab: moving from state preparing -> state finished >Thread-15911753::DEBUG::2013-11-19 21:41:27,449::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911753::DEBUG::2013-11-19 21:41:27,450::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911753::DEBUG::2013-11-19 21:41:27,451::task::495::TaskManager.Task::(_debug) Task 2b899450-9168-4b0d-b984-cbfd9aabdcab: ref 0 aborting False >Thread-15911753::INFO::2013-11-19 21:41:27,457::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000324964523315', 'lastCheck': 1384897277.6257949, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00133109092712', 'lastCheck': 1384897278.0942831, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897284.255214, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000363111495972', 'lastCheck': 1384897285.4337831, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:41:34,268::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911759::INFO::2013-11-19 21:41:37,601::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911759::DEBUG::2013-11-19 21:41:37,605::task::495::TaskManager.Task::(_debug) Task 50747220-a0bb-42d4-81c6-4e1392b05369: moving from state init -> state preparing >Thread-15911759::DEBUG::2013-11-19 21:41:37,607::task::495::TaskManager.Task::(_debug) Task 50747220-a0bb-42d4-81c6-4e1392b05369: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000346183776855', 'lastCheck': 1384897287.638407, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142002105713', 'lastCheck': 1384897288.109844, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897294.2691131, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000498056411743', 'lastCheck': 1384897295.4455531, 'valid': True, 'code': 0}} >Thread-15911759::DEBUG::2013-11-19 21:41:37,629::task::495::TaskManager.Task::(_debug) Task 50747220-a0bb-42d4-81c6-4e1392b05369: moving from state preparing -> state finished >Thread-15911759::DEBUG::2013-11-19 21:41:37,631::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911759::DEBUG::2013-11-19 21:41:37,632::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911759::DEBUG::2013-11-19 21:41:37,632::task::495::TaskManager.Task::(_debug) Task 50747220-a0bb-42d4-81c6-4e1392b05369: ref 0 aborting False >Thread-15911759::INFO::2013-11-19 21:41:37,638::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000346183776855', 'lastCheck': 1384897287.638407, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142002105713', 'lastCheck': 1384897288.109844, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897294.2691131, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000498056411743', 'lastCheck': 1384897295.4455531, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:41:44,281::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911765::INFO::2013-11-19 21:41:47,756::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911765::DEBUG::2013-11-19 21:41:47,757::task::495::TaskManager.Task::(_debug) Task 47205ac4-e502-484a-b7b7-c8019e172b96: moving from state init -> state preparing >Thread-15911765::DEBUG::2013-11-19 21:41:47,757::task::495::TaskManager.Task::(_debug) Task 47205ac4-e502-484a-b7b7-c8019e172b96: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000342130661011', 'lastCheck': 1384897307.6631291, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00136780738831', 'lastCheck': 1384897298.125643, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897304.2824919, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000363826751709', 'lastCheck': 1384897305.4571559, 'valid': True, 'code': 0}} >Thread-15911765::DEBUG::2013-11-19 21:41:47,769::task::495::TaskManager.Task::(_debug) Task 47205ac4-e502-484a-b7b7-c8019e172b96: moving from state preparing -> state finished >Thread-15911765::DEBUG::2013-11-19 21:41:47,769::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911765::DEBUG::2013-11-19 21:41:47,770::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911765::DEBUG::2013-11-19 21:41:47,770::task::495::TaskManager.Task::(_debug) Task 47205ac4-e502-484a-b7b7-c8019e172b96: ref 0 aborting False >Thread-15911765::INFO::2013-11-19 21:41:47,776::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000342130661011', 'lastCheck': 1384897307.6631291, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00136780738831', 'lastCheck': 1384897298.125643, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897304.2824919, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000363826751709', 'lastCheck': 1384897305.4571559, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:41:54,295::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911771::INFO::2013-11-19 21:41:57,905::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911771::DEBUG::2013-11-19 21:41:57,906::task::495::TaskManager.Task::(_debug) Task 1aa36bf0-1e54-460a-8c93-f8bafcf11506: moving from state init -> state preparing >Thread-15911771::DEBUG::2013-11-19 21:41:57,906::task::495::TaskManager.Task::(_debug) Task 1aa36bf0-1e54-460a-8c93-f8bafcf11506: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000350952148438', 'lastCheck': 1384897317.6756811, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0013530254364', 'lastCheck': 1384897308.1410539, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897314.295814, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000368118286133', 'lastCheck': 1384897315.4687469, 'valid': True, 'code': 0}} >Thread-15911771::DEBUG::2013-11-19 21:41:57,917::task::495::TaskManager.Task::(_debug) Task 1aa36bf0-1e54-460a-8c93-f8bafcf11506: moving from state preparing -> state finished >Thread-15911771::DEBUG::2013-11-19 21:41:57,917::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911771::DEBUG::2013-11-19 21:41:57,918::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911771::DEBUG::2013-11-19 21:41:57,918::task::495::TaskManager.Task::(_debug) Task 1aa36bf0-1e54-460a-8c93-f8bafcf11506: ref 0 aborting False >Thread-15911771::INFO::2013-11-19 21:41:57,919::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000350952148438', 'lastCheck': 1384897317.6756811, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0013530254364', 'lastCheck': 1384897308.1410539, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897314.295814, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000368118286133', 'lastCheck': 1384897315.4687469, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:42:04,308::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911777::INFO::2013-11-19 21:42:09,667::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911777::DEBUG::2013-11-19 21:42:09,668::task::495::TaskManager.Task::(_debug) Task 77deb013-fada-4bdf-88e8-6ea60cbe6548: moving from state init -> state preparing >Thread-15911777::DEBUG::2013-11-19 21:42:09,669::task::495::TaskManager.Task::(_debug) Task 77deb013-fada-4bdf-88e8-6ea60cbe6548: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00036883354187', 'lastCheck': 1384897327.6883531, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139307975769', 'lastCheck': 1384897328.1722939, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897324.30955, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000443935394287', 'lastCheck': 1384897325.4805, 'valid': True, 'code': 0}} >Thread-15911777::DEBUG::2013-11-19 21:42:09,677::task::495::TaskManager.Task::(_debug) Task 77deb013-fada-4bdf-88e8-6ea60cbe6548: moving from state preparing -> state finished >Thread-15911777::DEBUG::2013-11-19 21:42:09,678::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911777::DEBUG::2013-11-19 21:42:09,679::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911777::DEBUG::2013-11-19 21:42:09,679::task::495::TaskManager.Task::(_debug) Task 77deb013-fada-4bdf-88e8-6ea60cbe6548: ref 0 aborting False >Thread-15911777::INFO::2013-11-19 21:42:09,680::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00036883354187', 'lastCheck': 1384897327.6883531, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139307975769', 'lastCheck': 1384897328.1722939, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897324.30955, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000443935394287', 'lastCheck': 1384897325.4805, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:42:14,322::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911783::INFO::2013-11-19 21:42:19,938::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911783::DEBUG::2013-11-19 21:42:19,938::task::495::TaskManager.Task::(_debug) Task 7645550b-6d8a-4b79-bb18-5801c0686217: moving from state init -> state preparing >Thread-15911783::DEBUG::2013-11-19 21:42:19,954::task::495::TaskManager.Task::(_debug) Task 7645550b-6d8a-4b79-bb18-5801c0686217: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00035285949707', 'lastCheck': 1384897337.700597, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00138115882874', 'lastCheck': 1384897338.1876769, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897334.3231189, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000401020050049', 'lastCheck': 1384897335.4921689, 'valid': True, 'code': 0}} >Thread-15911783::DEBUG::2013-11-19 21:42:19,966::task::495::TaskManager.Task::(_debug) Task 7645550b-6d8a-4b79-bb18-5801c0686217: moving from state preparing -> state finished >Thread-15911783::DEBUG::2013-11-19 21:42:19,966::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911783::DEBUG::2013-11-19 21:42:19,967::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911783::DEBUG::2013-11-19 21:42:19,967::task::495::TaskManager.Task::(_debug) Task 7645550b-6d8a-4b79-bb18-5801c0686217: ref 0 aborting False >Thread-15911783::INFO::2013-11-19 21:42:19,971::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00035285949707', 'lastCheck': 1384897337.700597, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00138115882874', 'lastCheck': 1384897338.1876769, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897334.3231189, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000401020050049', 'lastCheck': 1384897335.4921689, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:42:24,336::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 21:42:25,503::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 21:42:26,207::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911789::INFO::2013-11-19 21:42:30,067::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911789::DEBUG::2013-11-19 21:42:30,068::task::495::TaskManager.Task::(_debug) Task 24900dba-b140-421b-b45c-f2c0e36fe3b4: moving from state init -> state preparing >Thread-15911789::DEBUG::2013-11-19 21:42:30,068::task::495::TaskManager.Task::(_debug) Task 24900dba-b140-421b-b45c-f2c0e36fe3b4: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000453948974609', 'lastCheck': 1384897347.746563, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146317481995', 'lastCheck': 1384897348.2034869, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897344.3370881, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000348091125488', 'lastCheck': 1384897346.2093611, 'valid': True, 'code': 0}} >Thread-15911789::DEBUG::2013-11-19 21:42:30,069::task::495::TaskManager.Task::(_debug) Task 24900dba-b140-421b-b45c-f2c0e36fe3b4: moving from state preparing -> state finished >Thread-15911789::DEBUG::2013-11-19 21:42:30,070::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911789::DEBUG::2013-11-19 21:42:30,070::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911789::DEBUG::2013-11-19 21:42:30,071::task::495::TaskManager.Task::(_debug) Task 24900dba-b140-421b-b45c-f2c0e36fe3b4: ref 0 aborting False >Thread-15911789::INFO::2013-11-19 21:42:30,072::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000453948974609', 'lastCheck': 1384897347.746563, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146317481995', 'lastCheck': 1384897348.2034869, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897344.3370881, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000348091125488', 'lastCheck': 1384897346.2093611, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:42:34,349::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911795::INFO::2013-11-19 21:42:40,169::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911795::DEBUG::2013-11-19 21:42:40,170::task::495::TaskManager.Task::(_debug) Task d733679b-ad49-4d5d-b943-8d8beec92835: moving from state init -> state preparing >Thread-15911795::DEBUG::2013-11-19 21:42:40,171::task::495::TaskManager.Task::(_debug) Task d733679b-ad49-4d5d-b943-8d8beec92835: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000780820846558', 'lastCheck': 1384897357.7631941, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00143909454346', 'lastCheck': 1384897358.2191319, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897354.350445, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00037693977356', 'lastCheck': 1384897356.2208469, 'valid': True, 'code': 0}} >Thread-15911795::DEBUG::2013-11-19 21:42:40,171::task::495::TaskManager.Task::(_debug) Task d733679b-ad49-4d5d-b943-8d8beec92835: moving from state preparing -> state finished >Thread-15911795::DEBUG::2013-11-19 21:42:40,172::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911795::DEBUG::2013-11-19 21:42:40,173::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911795::DEBUG::2013-11-19 21:42:40,173::task::495::TaskManager.Task::(_debug) Task d733679b-ad49-4d5d-b943-8d8beec92835: ref 0 aborting False >Thread-15911795::INFO::2013-11-19 21:42:40,174::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000780820846558', 'lastCheck': 1384897357.7631941, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00143909454346', 'lastCheck': 1384897358.2191319, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897354.350445, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00037693977356', 'lastCheck': 1384897356.2208469, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:42:44,363::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911801::INFO::2013-11-19 21:42:50,324::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911801::DEBUG::2013-11-19 21:42:50,325::task::495::TaskManager.Task::(_debug) Task 7c0632c9-77db-4014-bf09-016f24d29584: moving from state init -> state preparing >Thread-15911801::DEBUG::2013-11-19 21:42:50,326::task::495::TaskManager.Task::(_debug) Task 7c0632c9-77db-4014-bf09-016f24d29584: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000335931777954', 'lastCheck': 1384897367.776026, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139713287354', 'lastCheck': 1384897368.2342999, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897364.3641069, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000427961349487', 'lastCheck': 1384897366.231375, 'valid': True, 'code': 0}} >Thread-15911801::DEBUG::2013-11-19 21:42:50,336::task::495::TaskManager.Task::(_debug) Task 7c0632c9-77db-4014-bf09-016f24d29584: moving from state preparing -> state finished >Thread-15911801::DEBUG::2013-11-19 21:42:50,337::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911801::DEBUG::2013-11-19 21:42:50,338::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911801::DEBUG::2013-11-19 21:42:50,339::task::495::TaskManager.Task::(_debug) Task 7c0632c9-77db-4014-bf09-016f24d29584: ref 0 aborting False >Thread-15911801::INFO::2013-11-19 21:42:50,344::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000335931777954', 'lastCheck': 1384897367.776026, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139713287354', 'lastCheck': 1384897368.2342999, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897364.3641069, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000427961349487', 'lastCheck': 1384897366.231375, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:42:54,376::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911807::INFO::2013-11-19 21:43:00,505::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911807::DEBUG::2013-11-19 21:43:00,506::task::495::TaskManager.Task::(_debug) Task b6d29b79-487f-4814-b86c-1a60e8ed2014: moving from state init -> state preparing >Thread-15911807::DEBUG::2013-11-19 21:43:00,507::task::495::TaskManager.Task::(_debug) Task b6d29b79-487f-4814-b86c-1a60e8ed2014: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000265121459961', 'lastCheck': 1384897377.7888069, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00211405754089', 'lastCheck': 1384897378.2504871, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897374.377619, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000397205352783', 'lastCheck': 1384897376.242481, 'valid': True, 'code': 0}} >Thread-15911807::DEBUG::2013-11-19 21:43:00,514::task::495::TaskManager.Task::(_debug) Task b6d29b79-487f-4814-b86c-1a60e8ed2014: moving from state preparing -> state finished >Thread-15911807::DEBUG::2013-11-19 21:43:00,515::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911807::DEBUG::2013-11-19 21:43:00,515::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911807::DEBUG::2013-11-19 21:43:00,516::task::495::TaskManager.Task::(_debug) Task b6d29b79-487f-4814-b86c-1a60e8ed2014: ref 0 aborting False >Thread-15911807::INFO::2013-11-19 21:43:00,516::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000265121459961', 'lastCheck': 1384897377.7888069, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00211405754089', 'lastCheck': 1384897378.2504871, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897374.377619, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000397205352783', 'lastCheck': 1384897376.242481, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:43:04,390::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911813::INFO::2013-11-19 21:43:10,624::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911813::DEBUG::2013-11-19 21:43:10,625::task::495::TaskManager.Task::(_debug) Task e27d59a6-f41a-43e4-af04-d755461e9090: moving from state init -> state preparing >Thread-15911813::DEBUG::2013-11-19 21:43:10,625::task::495::TaskManager.Task::(_debug) Task e27d59a6-f41a-43e4-af04-d755461e9090: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000328063964844', 'lastCheck': 1384897387.801584, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139093399048', 'lastCheck': 1384897388.2647991, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897384.391139, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000361919403076', 'lastCheck': 1384897386.2542601, 'valid': True, 'code': 0}} >Thread-15911813::DEBUG::2013-11-19 21:43:10,626::task::495::TaskManager.Task::(_debug) Task e27d59a6-f41a-43e4-af04-d755461e9090: moving from state preparing -> state finished >Thread-15911813::DEBUG::2013-11-19 21:43:10,627::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911813::DEBUG::2013-11-19 21:43:10,627::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911813::DEBUG::2013-11-19 21:43:10,628::task::495::TaskManager.Task::(_debug) Task e27d59a6-f41a-43e4-af04-d755461e9090: ref 0 aborting False >Thread-15911813::INFO::2013-11-19 21:43:10,629::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000328063964844', 'lastCheck': 1384897387.801584, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139093399048', 'lastCheck': 1384897388.2647991, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897384.391139, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000361919403076', 'lastCheck': 1384897386.2542601, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:43:14,403::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911819::INFO::2013-11-19 21:43:20,739::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911819::DEBUG::2013-11-19 21:43:20,740::task::495::TaskManager.Task::(_debug) Task 2237a240-fc94-42f0-88a4-be31b39400d7: moving from state init -> state preparing >Thread-15911819::DEBUG::2013-11-19 21:43:20,741::task::495::TaskManager.Task::(_debug) Task 2237a240-fc94-42f0-88a4-be31b39400d7: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000323057174683', 'lastCheck': 1384897397.814194, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00140595436096', 'lastCheck': 1384897398.2798901, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897394.404515, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000301122665405', 'lastCheck': 1384897396.2657509, 'valid': True, 'code': 0}} >Thread-15911819::DEBUG::2013-11-19 21:43:20,748::task::495::TaskManager.Task::(_debug) Task 2237a240-fc94-42f0-88a4-be31b39400d7: moving from state preparing -> state finished >Thread-15911819::DEBUG::2013-11-19 21:43:20,748::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911819::DEBUG::2013-11-19 21:43:20,749::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911819::DEBUG::2013-11-19 21:43:20,749::task::495::TaskManager.Task::(_debug) Task 2237a240-fc94-42f0-88a4-be31b39400d7: ref 0 aborting False >Thread-15911819::INFO::2013-11-19 21:43:20,754::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000323057174683', 'lastCheck': 1384897397.814194, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00140595436096', 'lastCheck': 1384897398.2798901, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897394.404515, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000301122665405', 'lastCheck': 1384897396.2657509, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:43:24,417::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911825::INFO::2013-11-19 21:43:30,871::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911825::DEBUG::2013-11-19 21:43:30,872::task::495::TaskManager.Task::(_debug) Task cbe2c401-410c-4398-9518-41138fe91265: moving from state init -> state preparing >Thread-15911825::DEBUG::2013-11-19 21:43:30,873::task::495::TaskManager.Task::(_debug) Task cbe2c401-410c-4398-9518-41138fe91265: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000292778015137', 'lastCheck': 1384897407.826875, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0013439655304', 'lastCheck': 1384897408.2949741, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897404.4179709, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000377893447876', 'lastCheck': 1384897406.2773399, 'valid': True, 'code': 0}} >Thread-15911825::DEBUG::2013-11-19 21:43:30,878::task::495::TaskManager.Task::(_debug) Task cbe2c401-410c-4398-9518-41138fe91265: moving from state preparing -> state finished >Thread-15911825::DEBUG::2013-11-19 21:43:30,879::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911825::DEBUG::2013-11-19 21:43:30,879::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911825::DEBUG::2013-11-19 21:43:30,880::task::495::TaskManager.Task::(_debug) Task cbe2c401-410c-4398-9518-41138fe91265: ref 0 aborting False >Thread-15911825::INFO::2013-11-19 21:43:30,885::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000292778015137', 'lastCheck': 1384897407.826875, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0013439655304', 'lastCheck': 1384897408.2949741, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897404.4179709, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000377893447876', 'lastCheck': 1384897406.2773399, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:43:34,431::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911831::INFO::2013-11-19 21:43:40,995::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911831::DEBUG::2013-11-19 21:43:40,996::task::495::TaskManager.Task::(_debug) Task 2bc1ac6a-55da-448e-ba3b-eb98b44ba5d1: moving from state init -> state preparing >Thread-15911831::DEBUG::2013-11-19 21:43:40,997::task::495::TaskManager.Task::(_debug) Task 2bc1ac6a-55da-448e-ba3b-eb98b44ba5d1: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000269889831543', 'lastCheck': 1384897417.839484, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00145602226257', 'lastCheck': 1384897418.3104081, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897414.4318049, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000456809997559', 'lastCheck': 1384897416.2890429, 'valid': True, 'code': 0}} >Thread-15911831::DEBUG::2013-11-19 21:43:40,998::task::495::TaskManager.Task::(_debug) Task 2bc1ac6a-55da-448e-ba3b-eb98b44ba5d1: moving from state preparing -> state finished >Thread-15911831::DEBUG::2013-11-19 21:43:40,998::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911831::DEBUG::2013-11-19 21:43:40,999::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911831::DEBUG::2013-11-19 21:43:41,000::task::495::TaskManager.Task::(_debug) Task 2bc1ac6a-55da-448e-ba3b-eb98b44ba5d1: ref 0 aborting False >Thread-15911831::INFO::2013-11-19 21:43:41,008::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000269889831543', 'lastCheck': 1384897417.839484, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00145602226257', 'lastCheck': 1384897418.3104081, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897414.4318049, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000456809997559', 'lastCheck': 1384897416.2890429, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:43:44,444::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911837::INFO::2013-11-19 21:43:51,099::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911837::DEBUG::2013-11-19 21:43:51,099::task::495::TaskManager.Task::(_debug) Task ce564904-2f04-474a-bd0f-1dfa3a32a839: moving from state init -> state preparing >Thread-15911837::DEBUG::2013-11-19 21:43:51,100::task::495::TaskManager.Task::(_debug) Task ce564904-2f04-474a-bd0f-1dfa3a32a839: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00029182434082', 'lastCheck': 1384897427.8520401, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00132393836975', 'lastCheck': 1384897428.3255551, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897424.4452541, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000329971313477', 'lastCheck': 1384897426.300627, 'valid': True, 'code': 0}} >Thread-15911837::DEBUG::2013-11-19 21:43:51,101::task::495::TaskManager.Task::(_debug) Task ce564904-2f04-474a-bd0f-1dfa3a32a839: moving from state preparing -> state finished >Thread-15911837::DEBUG::2013-11-19 21:43:51,102::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911837::DEBUG::2013-11-19 21:43:51,103::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911837::DEBUG::2013-11-19 21:43:51,103::task::495::TaskManager.Task::(_debug) Task ce564904-2f04-474a-bd0f-1dfa3a32a839: ref 0 aborting False >Thread-15911837::INFO::2013-11-19 21:43:51,104::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00029182434082', 'lastCheck': 1384897427.8520401, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00132393836975', 'lastCheck': 1384897428.3255551, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897424.4452541, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000329971313477', 'lastCheck': 1384897426.300627, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:43:54,458::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911843::INFO::2013-11-19 21:44:01,358::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911843::DEBUG::2013-11-19 21:44:01,359::task::495::TaskManager.Task::(_debug) Task 78b94a99-6bc4-4efb-95b8-c884e8cbf401: moving from state init -> state preparing >Thread-15911843::DEBUG::2013-11-19 21:44:01,359::task::495::TaskManager.Task::(_debug) Task 78b94a99-6bc4-4efb-95b8-c884e8cbf401: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00026798248291', 'lastCheck': 1384897437.864686, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00208711624146', 'lastCheck': 1384897438.342711, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897434.458694, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000418901443481', 'lastCheck': 1384897436.3123901, 'valid': True, 'code': 0}} >Thread-15911843::DEBUG::2013-11-19 21:44:01,370::task::495::TaskManager.Task::(_debug) Task 78b94a99-6bc4-4efb-95b8-c884e8cbf401: moving from state preparing -> state finished >Thread-15911843::DEBUG::2013-11-19 21:44:01,370::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911843::DEBUG::2013-11-19 21:44:01,371::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911843::DEBUG::2013-11-19 21:44:01,371::task::495::TaskManager.Task::(_debug) Task 78b94a99-6bc4-4efb-95b8-c884e8cbf401: ref 0 aborting False >Thread-15911843::INFO::2013-11-19 21:44:01,374::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00026798248291', 'lastCheck': 1384897437.864686, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00208711624146', 'lastCheck': 1384897438.342711, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384897434.458694, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000418901443481', 'lastCheck': 1384897436.3123901, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 21:44:04,471::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:44:14,485::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:44:24,498::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:44:34,512::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:44:44,525::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:44:54,538::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 21:44:57,937::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 21:44:58,609::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:45:04,552::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:45:14,566::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:45:24,579::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:45:34,593::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:45:44,607::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:45:54,620::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:46:04,634::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:46:14,648::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:46:24,661::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:46:34,675::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:46:44,689::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:46:54,702::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:47:04,716::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:47:14,729::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:47:24,743::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 21:47:26,556::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 21:47:27,218::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:47:34,778::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:47:44,792::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:47:54,806::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:48:04,820::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:48:14,834::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:48:24,847::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:48:34,861::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:48:44,875::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:48:54,888::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:49:04,902::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:49:14,915::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:49:24,929::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:49:34,943::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:49:44,957::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:49:54,970::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 21:49:59,005::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 21:49:59,650::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:50:04,985::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:50:14,999::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:50:25,013::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:50:35,027::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:50:45,041::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:50:55,055::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:51:05,068::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:51:15,082::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:51:25,096::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:51:35,110::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:51:45,124::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:51:55,138::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:52:05,151::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:52:15,165::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:52:25,179::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 21:52:27,567::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 21:52:28,221::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:52:35,193::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:52:45,206::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:52:55,220::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:53:05,234::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:53:15,247::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:53:25,261::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:53:35,274::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:53:45,287::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:53:55,300::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:54:05,314::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:54:15,327::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:54:25,340::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:54:35,354::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:54:45,368::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:54:55,381::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 21:55:00,032::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 21:55:00,680::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:55:05,395::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:55:15,409::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:55:25,422::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:55:35,435::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:55:45,448::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:55:55,462::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:56:05,476::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:56:15,489::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:56:25,502::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:56:35,516::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:56:45,529::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:56:55,543::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:57:05,556::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:57:15,569::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:57:25,583::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 21:57:28,576::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 21:57:29,203::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 21:57:35,790::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:57:45,804::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:57:55,817::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:58:05,830::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:58:15,844::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:58:25,857::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:58:35,872::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:58:45,886::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:58:55,899::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:59:05,912::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:59:15,925::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:59:25,938::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:59:35,952::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:59:45,965::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 21:59:55,980::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 22:00:01,064::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 22:00:01,726::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:00:05,994::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:00:16,007::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:00:26,021::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:00:36,035::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:00:46,048::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:00:56,061::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:01:06,075::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:01:16,089::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:01:26,102::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:01:36,116::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:01:46,130::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:01:56,143::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:02:06,157::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:02:16,170::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:02:26,184::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 22:02:29,558::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 22:02:30,272::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:02:36,197::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:02:46,210::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:02:56,224::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:03:06,237::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:03:16,250::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:03:26,264::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:03:36,277::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:03:46,291::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:03:56,305::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:04:06,319::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:04:16,332::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:04:26,346::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:04:36,360::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:04:46,373::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:04:56,387::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 22:05:02,113::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 22:05:02,794::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:05:06,401::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:05:16,414::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:05:26,428::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:05:36,443::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:05:46,457::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:05:56,470::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:06:06,484::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:06:16,498::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:06:26,511::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:06:36,525::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:06:46,538::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:06:56,552::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:07:06,565::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:07:16,579::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:07:26,592::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 22:07:30,618::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 22:07:31,265::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:07:36,606::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:07:46,620::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:07:56,634::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:08:06,647::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:08:16,661::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:08:26,675::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:08:36,688::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:08:46,702::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:08:56,716::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:09:06,729::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:09:16,743::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:09:26,757::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:09:36,771::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:09:46,785::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:09:56,798::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 22:10:03,180::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 22:10:03,821::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:10:06,813::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:10:16,827::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911847::INFO::2013-11-19 22:10:24,510::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911847::DEBUG::2013-11-19 22:10:24,511::task::495::TaskManager.Task::(_debug) Task 14271aee-57d1-483e-97e6-a7e7aee6e90f: moving from state init -> state preparing >Thread-15911847::DEBUG::2013-11-19 22:10:24,519::task::495::TaskManager.Task::(_debug) Task 14271aee-57d1-483e-97e6-a7e7aee6e90f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000361204147339', 'lastCheck': 1384899023.850594, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142097473145', 'lastCheck': 1384899021.4222579, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899016.827668, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000737905502319', 'lastCheck': 1384899021.4634449, 'valid': True, 'code': 0}} >Thread-15911847::DEBUG::2013-11-19 22:10:24,526::task::495::TaskManager.Task::(_debug) Task 14271aee-57d1-483e-97e6-a7e7aee6e90f: moving from state preparing -> state finished >Thread-15911847::DEBUG::2013-11-19 22:10:24,527::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911847::DEBUG::2013-11-19 22:10:24,528::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911847::DEBUG::2013-11-19 22:10:24,528::task::495::TaskManager.Task::(_debug) Task 14271aee-57d1-483e-97e6-a7e7aee6e90f: ref 0 aborting False >Thread-15911847::INFO::2013-11-19 22:10:24,531::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000361204147339', 'lastCheck': 1384899023.850594, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142097473145', 'lastCheck': 1384899021.4222579, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899016.827668, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000737905502319', 'lastCheck': 1384899021.4634449, 'valid': True, 'code': 0}} >Thread-15911848::DEBUG::2013-11-19 22:10:24,564::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVdsCapabilities with () {} >Thread-15911848::DEBUG::2013-11-19 22:10:25,000::utils::579::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-img-rhev' (cwd None) >Thread-15911848::DEBUG::2013-11-19 22:10:25,418::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911848::DEBUG::2013-11-19 22:10:25,420::utils::579::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-kvm-rhev' (cwd None) >Thread-15911848::DEBUG::2013-11-19 22:10:25,863::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911848::DEBUG::2013-11-19 22:10:25,865::utils::579::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" vdsm' (cwd None) >Thread-15911848::DEBUG::2013-11-19 22:10:26,321::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911848::DEBUG::2013-11-19 22:10:26,323::utils::579::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" spice-server' (cwd None) >Thread-15911848::DEBUG::2013-11-19 22:10:26,750::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911848::DEBUG::2013-11-19 22:10:26,751::utils::579::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" libvirt' (cwd None) >Thread-1024767::ERROR::2013-11-19 22:10:26,997::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911848::DEBUG::2013-11-19 22:10:27,212::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15911848::DEBUG::2013-11-19 22:10:27,215::clientIF::59::vds::(wrapper) return getVdsCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:c936ebb147a8'}], 'FC': []}, 'packages2': {'kernel': {'release': '279.11.1.el6.x86_64', 'buildtime': 1348297826.0, 'version': '2.6.32'}, 'spice-server': {'release': '10.el6', 'buildtime': '1337611492', 'version': '0.10.1'}, 'vdsm': {'release': '113.4.el6_3', 'buildtime': '1347534072', 'version': '4.9'}, 'qemu-kvm': {'release': '2.295.el6_3.2', 'buildtime': '1346673796', 'version': '0.12.1.2'}, 'libvirt': {'release': '21.el6_3.5', 'buildtime': '1348687577', 'version': '0.9.10'}, 'qemu-img': {'release': '2.295.el6_3.2', 'buildtime': '1346673796', 'version': '0.12.1.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz', 'hooks': {'before_vm_start': {'50_vhostmd': {'md5': '1635e45584995d9ad2629939cb0bcdbf'}}, 'after_vm_destroy': {'50_vhostmd': {'md5': '7e7509430304099f38e4a9de966af090'}}, 'before_vm_dehibernate': {'50_vhostmd': {'md5': '1635e45584995d9ad2629939cb0bcdbf'}}, 'before_vm_migrate_destination': {'50_vhostmd': {'md5': '1635e45584995d9ad2629939cb0bcdbf'}}}, 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'addr': '192.168.100.133', 'cfg': {'IPADDR': '192.168.100.133', 'GATEWAY': '192.168.100.1', 'DELAY': '0', 'NETMASK': '255.255.128.0', 'STP': 'no', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ports': ['bond0'], 'netmask': '255.255.128.0', 'stp': 'off', 'gateway': '192.168.100.1'}, 'DMZ': {'addr': '172.16.1.133', 'cfg': {'IPADDR': '172.16.1.133', 'DELAY': '0', 'NETMASK': '255.255.255.0', 'STP': 'no', 'DEVICE': 'DMZ', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ports': ['eth0'], 'netmask': '255.255.255.0', 'stp': 'off', 'gateway': '0.0.0.0'}}, 'uuid': '30343536-3138-5A43-4A32-333630373442_AC:16:2D:76:EB:F0', 'lastClientIface': 'rhevm', 'nics': {'eth3': {'hwaddr': 'AC:16:2D:76:EB:F2', 'netmask': '', 'speed': 1000, 'addr': '', 'permhwaddr': 'AC:16:2D:76:EB:F3'}, 'eth2': {'hwaddr': 'AC:16:2D:76:EB:F2', 'netmask': '', 'speed': 1000, 'addr': '', 'permhwaddr': 'AC:16:2D:76:EB:F2'}, 'eth1': {'hwaddr': 'AC:16:2D:76:EB:F1', 'netmask': '', 'speed': 0, 'addr': ''}, 'eth0': {'hwaddr': 'AC:16:2D:76:EB:F0', 'netmask': '', 'speed': 1000, 'addr': ''}}, 'software_revision': '113.4', 'management_ip': '192.168.100.133', 'clusterLevels': ['3.0'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,dca,sse4_1,sse4_2,x2apic,popcnt,aes,xsave,avx,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_486,model_pentium,model_pentium2,model_pentium3,model_pentiumpro,model_qemu32,model_cpu64-rhel5,model_cpu64-rhel6,model_kvm32,model_coreduo,model_kvm64,model_core2duo,model_n270,model_Conroe,model_Penryn,model_Nehalem,model_Westmere,model_SandyBridge,model_Opteron_G1', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:c936ebb147a8', 'memSize': '64392', 'reservedMem': '256', 'bondings': {'bond4': {'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'netmask': '', 'addr': '', 'slaves': []}, 'bond0': {'hwaddr': 'AC:16:2D:76:EB:F2', 'cfg': {'DEVICE': 'bond0', 'BRIDGE': 'rhevm', 'BOOTPROTO': 'none', 'ONBOOT': 'yes', 'BONDING_OPTS': 'mode=1 miimon=100'}, 'netmask': '', 'addr': '', 'slaves': ['eth2', 'eth3']}, 'bond1': {'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'netmask': '', 'addr': '', 'slaves': []}, 'bond2': {'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'netmask': '', 'addr': '', 'slaves': []}, 'bond3': {'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'netmask': '', 'addr': '', 'slaves': []}}, 'software_version': '4.9', 'cpuSpeed': '1995.126', 'cpuSockets': '2', 'vlans': {}, 'cpuCores': '12', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'version_name': 'Snow Man', 'emulatedMachines': [u'rhel6.3.0', u'pc', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '20121012.0.el6_3', 'version': '6.3', 'name': 'RHEV Hypervisor'}, 'lastClient': '192.168.100.135'}} >Thread-1024767::ERROR::2013-11-19 22:10:37,012::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911854::INFO::2013-11-19 22:10:37,583::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911854::DEBUG::2013-11-19 22:10:37,584::task::495::TaskManager.Task::(_debug) Task 9d4fff69-7262-4d57-9f74-ac44ccaf3f60: moving from state init -> state preparing >Thread-15911854::DEBUG::2013-11-19 22:10:37,585::task::495::TaskManager.Task::(_debug) Task 9d4fff69-7262-4d57-9f74-ac44ccaf3f60: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000437021255493', 'lastCheck': 1384899034.0024991, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00153803825378', 'lastCheck': 1384899031.6465781, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899037.013145, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000581979751587', 'lastCheck': 1384899031.6420341, 'valid': True, 'code': 0}} >Thread-15911854::DEBUG::2013-11-19 22:10:37,593::task::495::TaskManager.Task::(_debug) Task 9d4fff69-7262-4d57-9f74-ac44ccaf3f60: moving from state preparing -> state finished >Thread-15911854::DEBUG::2013-11-19 22:10:37,594::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911854::DEBUG::2013-11-19 22:10:37,594::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911854::DEBUG::2013-11-19 22:10:37,595::task::495::TaskManager.Task::(_debug) Task 9d4fff69-7262-4d57-9f74-ac44ccaf3f60: ref 0 aborting False >Thread-15911854::INFO::2013-11-19 22:10:37,595::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000437021255493', 'lastCheck': 1384899034.0024991, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00153803825378', 'lastCheck': 1384899031.6465781, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899037.013145, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000581979751587', 'lastCheck': 1384899031.6420341, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:10:47,025::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911860::INFO::2013-11-19 22:10:48,160::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911860::DEBUG::2013-11-19 22:10:48,160::task::495::TaskManager.Task::(_debug) Task 0ef391c0-feb6-40f0-bf80-6cb26d28a8ad: moving from state init -> state preparing >Thread-15911860::DEBUG::2013-11-19 22:10:48,161::task::495::TaskManager.Task::(_debug) Task 0ef391c0-feb6-40f0-bf80-6cb26d28a8ad: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000416040420532', 'lastCheck': 1384899044.0154419, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152993202209', 'lastCheck': 1384899041.662467, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899047.026602, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000422954559326', 'lastCheck': 1384899041.6535621, 'valid': True, 'code': 0}} >Thread-15911860::DEBUG::2013-11-19 22:10:48,170::task::495::TaskManager.Task::(_debug) Task 0ef391c0-feb6-40f0-bf80-6cb26d28a8ad: moving from state preparing -> state finished >Thread-15911860::DEBUG::2013-11-19 22:10:48,171::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911860::DEBUG::2013-11-19 22:10:48,171::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911860::DEBUG::2013-11-19 22:10:48,172::task::495::TaskManager.Task::(_debug) Task 0ef391c0-feb6-40f0-bf80-6cb26d28a8ad: ref 0 aborting False >Thread-15911860::INFO::2013-11-19 22:10:48,173::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000416040420532', 'lastCheck': 1384899044.0154419, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152993202209', 'lastCheck': 1384899041.662467, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899047.026602, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000422954559326', 'lastCheck': 1384899041.6535621, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:10:57,039::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911866::INFO::2013-11-19 22:10:58,351::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911866::DEBUG::2013-11-19 22:10:58,353::task::495::TaskManager.Task::(_debug) Task 1b5c8efe-97a7-4293-8775-db0a84abe06e: moving from state init -> state preparing >Thread-15911866::DEBUG::2013-11-19 22:10:58,355::task::495::TaskManager.Task::(_debug) Task 1b5c8efe-97a7-4293-8775-db0a84abe06e: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000334024429321', 'lastCheck': 1384899054.0281579, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148105621338', 'lastCheck': 1384899051.6782041, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899057.040169, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000379085540771', 'lastCheck': 1384899051.6648729, 'valid': True, 'code': 0}} >Thread-15911866::DEBUG::2013-11-19 22:10:58,368::task::495::TaskManager.Task::(_debug) Task 1b5c8efe-97a7-4293-8775-db0a84abe06e: moving from state preparing -> state finished >Thread-15911866::DEBUG::2013-11-19 22:10:58,370::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911866::DEBUG::2013-11-19 22:10:58,371::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911866::DEBUG::2013-11-19 22:10:58,372::task::495::TaskManager.Task::(_debug) Task 1b5c8efe-97a7-4293-8775-db0a84abe06e: ref 0 aborting False >Thread-15911866::INFO::2013-11-19 22:10:58,372::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000334024429321', 'lastCheck': 1384899054.0281579, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148105621338', 'lastCheck': 1384899051.6782041, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899057.040169, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000379085540771', 'lastCheck': 1384899051.6648729, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:11:07,053::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911872::INFO::2013-11-19 22:11:08,568::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911872::DEBUG::2013-11-19 22:11:08,569::task::495::TaskManager.Task::(_debug) Task 764ac90a-ae2d-4a4b-bfed-a32db3c1529c: moving from state init -> state preparing >Thread-15911872::DEBUG::2013-11-19 22:11:08,570::task::495::TaskManager.Task::(_debug) Task 764ac90a-ae2d-4a4b-bfed-a32db3c1529c: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00036096572876', 'lastCheck': 1384899064.0409529, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.023845911026', 'lastCheck': 1384899061.7165401, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899067.053674, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000464916229248', 'lastCheck': 1384899061.6762969, 'valid': True, 'code': 0}} >Thread-15911872::DEBUG::2013-11-19 22:11:08,581::task::495::TaskManager.Task::(_debug) Task 764ac90a-ae2d-4a4b-bfed-a32db3c1529c: moving from state preparing -> state finished >Thread-15911872::DEBUG::2013-11-19 22:11:08,581::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911872::DEBUG::2013-11-19 22:11:08,582::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911872::DEBUG::2013-11-19 22:11:08,582::task::495::TaskManager.Task::(_debug) Task 764ac90a-ae2d-4a4b-bfed-a32db3c1529c: ref 0 aborting False >Thread-15911872::INFO::2013-11-19 22:11:08,583::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00036096572876', 'lastCheck': 1384899064.0409529, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.023845911026', 'lastCheck': 1384899061.7165401, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899067.053674, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000464916229248', 'lastCheck': 1384899061.6762969, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:11:17,066::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911878::INFO::2013-11-19 22:11:18,745::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911878::DEBUG::2013-11-19 22:11:18,746::task::495::TaskManager.Task::(_debug) Task a4eb19d0-5836-4f24-b9e9-8ca481cca8c0: moving from state init -> state preparing >Thread-15911878::DEBUG::2013-11-19 22:11:18,746::task::495::TaskManager.Task::(_debug) Task a4eb19d0-5836-4f24-b9e9-8ca481cca8c0: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000389099121094', 'lastCheck': 1384899074.0537479, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00166797637939', 'lastCheck': 1384899071.73228, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899077.067194, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00043511390686', 'lastCheck': 1384899071.6877, 'valid': True, 'code': 0}} >Thread-15911878::DEBUG::2013-11-19 22:11:18,762::task::495::TaskManager.Task::(_debug) Task a4eb19d0-5836-4f24-b9e9-8ca481cca8c0: moving from state preparing -> state finished >Thread-15911878::DEBUG::2013-11-19 22:11:18,763::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911878::DEBUG::2013-11-19 22:11:18,763::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911878::DEBUG::2013-11-19 22:11:18,764::task::495::TaskManager.Task::(_debug) Task a4eb19d0-5836-4f24-b9e9-8ca481cca8c0: ref 0 aborting False >Thread-15911878::INFO::2013-11-19 22:11:18,769::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000389099121094', 'lastCheck': 1384899074.0537479, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00166797637939', 'lastCheck': 1384899071.73228, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899077.067194, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00043511390686', 'lastCheck': 1384899071.6877, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:11:27,080::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911884::INFO::2013-11-19 22:11:28,893::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911884::DEBUG::2013-11-19 22:11:28,894::task::495::TaskManager.Task::(_debug) Task 9343605a-7b6c-4f22-8078-b2d80f380842: moving from state init -> state preparing >Thread-15911884::DEBUG::2013-11-19 22:11:28,894::task::495::TaskManager.Task::(_debug) Task 9343605a-7b6c-4f22-8078-b2d80f380842: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000363826751709', 'lastCheck': 1384899084.066514, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139093399048', 'lastCheck': 1384899081.7479241, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899087.0807371, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000458002090454', 'lastCheck': 1384899081.699137, 'valid': True, 'code': 0}} >Thread-15911884::DEBUG::2013-11-19 22:11:28,907::task::495::TaskManager.Task::(_debug) Task 9343605a-7b6c-4f22-8078-b2d80f380842: moving from state preparing -> state finished >Thread-15911884::DEBUG::2013-11-19 22:11:28,907::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911884::DEBUG::2013-11-19 22:11:28,908::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911884::DEBUG::2013-11-19 22:11:28,908::task::495::TaskManager.Task::(_debug) Task 9343605a-7b6c-4f22-8078-b2d80f380842: ref 0 aborting False >Thread-15911884::INFO::2013-11-19 22:11:28,909::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000363826751709', 'lastCheck': 1384899084.066514, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139093399048', 'lastCheck': 1384899081.7479241, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899087.0807371, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000458002090454', 'lastCheck': 1384899081.699137, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:11:37,093::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911890::INFO::2013-11-19 22:11:39,039::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911890::DEBUG::2013-11-19 22:11:39,039::task::495::TaskManager.Task::(_debug) Task d0b99a9b-c854-4de9-8305-e7e38153dee8: moving from state init -> state preparing >Thread-15911890::DEBUG::2013-11-19 22:11:39,057::task::495::TaskManager.Task::(_debug) Task d0b99a9b-c854-4de9-8305-e7e38153dee8: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000380039215088', 'lastCheck': 1384899094.0793469, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00135087966919', 'lastCheck': 1384899091.763551, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899097.0946281, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00042200088501', 'lastCheck': 1384899091.710542, 'valid': True, 'code': 0}} >Thread-15911890::DEBUG::2013-11-19 22:11:39,063::task::495::TaskManager.Task::(_debug) Task d0b99a9b-c854-4de9-8305-e7e38153dee8: moving from state preparing -> state finished >Thread-15911890::DEBUG::2013-11-19 22:11:39,064::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911890::DEBUG::2013-11-19 22:11:39,064::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911890::DEBUG::2013-11-19 22:11:39,065::task::495::TaskManager.Task::(_debug) Task d0b99a9b-c854-4de9-8305-e7e38153dee8: ref 0 aborting False >Thread-15911890::INFO::2013-11-19 22:11:39,069::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000380039215088', 'lastCheck': 1384899094.0793469, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00135087966919', 'lastCheck': 1384899091.763551, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899097.0946281, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00042200088501', 'lastCheck': 1384899091.710542, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:11:47,107::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911896::INFO::2013-11-19 22:11:49,214::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911896::DEBUG::2013-11-19 22:11:49,215::task::495::TaskManager.Task::(_debug) Task 48ee9297-c5bf-4cb6-9301-c2318cd056ba: moving from state init -> state preparing >Thread-15911896::DEBUG::2013-11-19 22:11:49,215::task::495::TaskManager.Task::(_debug) Task 48ee9297-c5bf-4cb6-9301-c2318cd056ba: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000313997268677', 'lastCheck': 1384899104.0921569, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00143098831177', 'lastCheck': 1384899101.7794211, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899107.108434, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00046181678772', 'lastCheck': 1384899101.7220521, 'valid': True, 'code': 0}} >Thread-15911896::DEBUG::2013-11-19 22:11:49,222::task::495::TaskManager.Task::(_debug) Task 48ee9297-c5bf-4cb6-9301-c2318cd056ba: moving from state preparing -> state finished >Thread-15911896::DEBUG::2013-11-19 22:11:49,223::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911896::DEBUG::2013-11-19 22:11:49,224::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911896::DEBUG::2013-11-19 22:11:49,224::task::495::TaskManager.Task::(_debug) Task 48ee9297-c5bf-4cb6-9301-c2318cd056ba: ref 0 aborting False >Thread-15911896::INFO::2013-11-19 22:11:49,227::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000313997268677', 'lastCheck': 1384899104.0921569, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00143098831177', 'lastCheck': 1384899101.7794211, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899107.108434, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00046181678772', 'lastCheck': 1384899101.7220521, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:11:57,121::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911902::INFO::2013-11-19 22:11:59,527::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911902::DEBUG::2013-11-19 22:11:59,528::task::495::TaskManager.Task::(_debug) Task d94339b9-fa3a-49d8-8380-ee69f1942961: moving from state init -> state preparing >Thread-15911902::DEBUG::2013-11-19 22:11:59,528::task::495::TaskManager.Task::(_debug) Task d94339b9-fa3a-49d8-8380-ee69f1942961: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000380039215088', 'lastCheck': 1384899114.105006, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00133895874023', 'lastCheck': 1384899111.795001, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899117.1221471, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000370025634766', 'lastCheck': 1384899111.733439, 'valid': True, 'code': 0}} >Thread-15911902::DEBUG::2013-11-19 22:11:59,539::task::495::TaskManager.Task::(_debug) Task d94339b9-fa3a-49d8-8380-ee69f1942961: moving from state preparing -> state finished >Thread-15911902::DEBUG::2013-11-19 22:11:59,540::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911902::DEBUG::2013-11-19 22:11:59,540::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911902::DEBUG::2013-11-19 22:11:59,541::task::495::TaskManager.Task::(_debug) Task d94339b9-fa3a-49d8-8380-ee69f1942961: ref 0 aborting False >Thread-15911902::INFO::2013-11-19 22:11:59,544::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000380039215088', 'lastCheck': 1384899114.105006, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00133895874023', 'lastCheck': 1384899111.795001, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899117.1221471, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000370025634766', 'lastCheck': 1384899111.733439, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:12:07,134::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911908::INFO::2013-11-19 22:12:09,681::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911908::DEBUG::2013-11-19 22:12:09,682::task::495::TaskManager.Task::(_debug) Task f3ad77ac-f656-44f7-a19f-f68ea5f5d722: moving from state init -> state preparing >Thread-15911908::DEBUG::2013-11-19 22:12:09,682::task::495::TaskManager.Task::(_debug) Task f3ad77ac-f656-44f7-a19f-f68ea5f5d722: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000478029251099', 'lastCheck': 1384899124.118196, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00212097167969', 'lastCheck': 1384899121.81181, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899127.135011, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000392913818359', 'lastCheck': 1384899121.7448759, 'valid': True, 'code': 0}} >Thread-15911908::DEBUG::2013-11-19 22:12:09,690::task::495::TaskManager.Task::(_debug) Task f3ad77ac-f656-44f7-a19f-f68ea5f5d722: moving from state preparing -> state finished >Thread-15911908::DEBUG::2013-11-19 22:12:09,690::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911908::DEBUG::2013-11-19 22:12:09,691::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911908::DEBUG::2013-11-19 22:12:09,691::task::495::TaskManager.Task::(_debug) Task f3ad77ac-f656-44f7-a19f-f68ea5f5d722: ref 0 aborting False >Thread-15911908::INFO::2013-11-19 22:12:09,692::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000478029251099', 'lastCheck': 1384899124.118196, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00212097167969', 'lastCheck': 1384899121.81181, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899127.135011, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000392913818359', 'lastCheck': 1384899121.7448759, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:12:17,151::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911914::INFO::2013-11-19 22:12:19,853::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911914::DEBUG::2013-11-19 22:12:19,854::task::495::TaskManager.Task::(_debug) Task 9a1c691c-0bd1-4c8d-afb4-12ac058301fd: moving from state init -> state preparing >Thread-15911914::DEBUG::2013-11-19 22:12:19,854::task::495::TaskManager.Task::(_debug) Task 9a1c691c-0bd1-4c8d-afb4-12ac058301fd: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00041389465332', 'lastCheck': 1384899134.1315229, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146389007568', 'lastCheck': 1384899131.8273871, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899137.1518691, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00037693977356', 'lastCheck': 1384899131.7563429, 'valid': True, 'code': 0}} >Thread-15911914::DEBUG::2013-11-19 22:12:19,860::task::495::TaskManager.Task::(_debug) Task 9a1c691c-0bd1-4c8d-afb4-12ac058301fd: moving from state preparing -> state finished >Thread-15911914::DEBUG::2013-11-19 22:12:19,861::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911914::DEBUG::2013-11-19 22:12:19,862::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911914::DEBUG::2013-11-19 22:12:19,862::task::495::TaskManager.Task::(_debug) Task 9a1c691c-0bd1-4c8d-afb4-12ac058301fd: ref 0 aborting False >Thread-15911914::INFO::2013-11-19 22:12:19,867::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00041389465332', 'lastCheck': 1384899134.1315229, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146389007568', 'lastCheck': 1384899131.8273871, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899137.1518691, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00037693977356', 'lastCheck': 1384899131.7563429, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:12:27,164::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911920::INFO::2013-11-19 22:12:30,060::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911920::DEBUG::2013-11-19 22:12:30,061::task::495::TaskManager.Task::(_debug) Task a480542f-aed7-4f9f-b8e4-03e53b967d9b: moving from state init -> state preparing >Thread-15911920::DEBUG::2013-11-19 22:12:30,062::task::495::TaskManager.Task::(_debug) Task a480542f-aed7-4f9f-b8e4-03e53b967d9b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000303030014038', 'lastCheck': 1384899144.1441309, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148105621338', 'lastCheck': 1384899141.8429649, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899147.165333, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000361204147339', 'lastCheck': 1384899141.767771, 'valid': True, 'code': 0}} >Thread-15911920::DEBUG::2013-11-19 22:12:30,072::task::495::TaskManager.Task::(_debug) Task a480542f-aed7-4f9f-b8e4-03e53b967d9b: moving from state preparing -> state finished >Thread-15911920::DEBUG::2013-11-19 22:12:30,072::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911920::DEBUG::2013-11-19 22:12:30,073::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911920::DEBUG::2013-11-19 22:12:30,073::task::495::TaskManager.Task::(_debug) Task a480542f-aed7-4f9f-b8e4-03e53b967d9b: ref 0 aborting False >Thread-15911920::INFO::2013-11-19 22:12:30,074::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000303030014038', 'lastCheck': 1384899144.1441309, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148105621338', 'lastCheck': 1384899141.8429649, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899147.165333, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000361204147339', 'lastCheck': 1384899141.767771, 'valid': True, 'code': 0}} >Thread-42840::DEBUG::2013-11-19 22:12:31,779::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 22:12:32,458::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:12:37,178::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911926::INFO::2013-11-19 22:12:40,226::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911926::DEBUG::2013-11-19 22:12:40,227::task::495::TaskManager.Task::(_debug) Task 3cbfbf30-d530-42b5-8ec2-5d987e4b22e1: moving from state init -> state preparing >Thread-15911926::DEBUG::2013-11-19 22:12:40,228::task::495::TaskManager.Task::(_debug) Task 3cbfbf30-d530-42b5-8ec2-5d987e4b22e1: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000408887863159', 'lastCheck': 1384899154.157372, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137615203857', 'lastCheck': 1384899152.0079639, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899157.1797559, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000333070755005', 'lastCheck': 1384899152.4606471, 'valid': True, 'code': 0}} >Thread-15911926::DEBUG::2013-11-19 22:12:40,234::task::495::TaskManager.Task::(_debug) Task 3cbfbf30-d530-42b5-8ec2-5d987e4b22e1: moving from state preparing -> state finished >Thread-15911926::DEBUG::2013-11-19 22:12:40,235::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911926::DEBUG::2013-11-19 22:12:40,235::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911926::DEBUG::2013-11-19 22:12:40,236::task::495::TaskManager.Task::(_debug) Task 3cbfbf30-d530-42b5-8ec2-5d987e4b22e1: ref 0 aborting False >Thread-15911926::INFO::2013-11-19 22:12:40,237::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000408887863159', 'lastCheck': 1384899154.157372, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137615203857', 'lastCheck': 1384899152.0079639, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899157.1797559, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000333070755005', 'lastCheck': 1384899152.4606471, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:12:47,192::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911932::INFO::2013-11-19 22:12:50,419::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911932::DEBUG::2013-11-19 22:12:50,420::task::495::TaskManager.Task::(_debug) Task dc75ed79-322a-45a4-bad2-8cbf4f8b52f5: moving from state init -> state preparing >Thread-15911932::DEBUG::2013-11-19 22:12:50,420::task::495::TaskManager.Task::(_debug) Task dc75ed79-322a-45a4-bad2-8cbf4f8b52f5: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000321865081787', 'lastCheck': 1384899164.1702571, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154685974121', 'lastCheck': 1384899162.02426, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899167.193222, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000554084777832', 'lastCheck': 1384899162.472271, 'valid': True, 'code': 0}} >Thread-15911932::DEBUG::2013-11-19 22:12:50,431::task::495::TaskManager.Task::(_debug) Task dc75ed79-322a-45a4-bad2-8cbf4f8b52f5: moving from state preparing -> state finished >Thread-15911932::DEBUG::2013-11-19 22:12:50,432::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911932::DEBUG::2013-11-19 22:12:50,433::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911932::DEBUG::2013-11-19 22:12:50,433::task::495::TaskManager.Task::(_debug) Task dc75ed79-322a-45a4-bad2-8cbf4f8b52f5: ref 0 aborting False >Thread-15911932::INFO::2013-11-19 22:12:50,434::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000321865081787', 'lastCheck': 1384899164.1702571, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154685974121', 'lastCheck': 1384899162.02426, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899167.193222, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000554084777832', 'lastCheck': 1384899162.472271, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:12:57,205::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911938::INFO::2013-11-19 22:13:00,569::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911938::DEBUG::2013-11-19 22:13:00,570::task::495::TaskManager.Task::(_debug) Task 5aa2b9f4-980a-4213-80e3-6ad78d099c69: moving from state init -> state preparing >Thread-15911938::DEBUG::2013-11-19 22:13:00,571::task::495::TaskManager.Task::(_debug) Task 5aa2b9f4-980a-4213-80e3-6ad78d099c69: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000407934188843', 'lastCheck': 1384899174.1829889, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139498710632', 'lastCheck': 1384899172.0396459, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899177.206548, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000410079956055', 'lastCheck': 1384899172.4825909, 'valid': True, 'code': 0}} >Thread-15911938::DEBUG::2013-11-19 22:13:00,583::task::495::TaskManager.Task::(_debug) Task 5aa2b9f4-980a-4213-80e3-6ad78d099c69: moving from state preparing -> state finished >Thread-15911938::DEBUG::2013-11-19 22:13:00,584::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911938::DEBUG::2013-11-19 22:13:00,584::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911938::DEBUG::2013-11-19 22:13:00,585::task::495::TaskManager.Task::(_debug) Task 5aa2b9f4-980a-4213-80e3-6ad78d099c69: ref 0 aborting False >Thread-15911938::INFO::2013-11-19 22:13:00,586::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000407934188843', 'lastCheck': 1384899174.1829889, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139498710632', 'lastCheck': 1384899172.0396459, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899177.206548, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000410079956055', 'lastCheck': 1384899172.4825909, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:13:07,219::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911944::INFO::2013-11-19 22:13:10,857::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911944::DEBUG::2013-11-19 22:13:10,858::task::495::TaskManager.Task::(_debug) Task c72b8ad6-2771-4b99-bfcc-0178f4281772: moving from state init -> state preparing >Thread-15911944::DEBUG::2013-11-19 22:13:10,858::task::495::TaskManager.Task::(_debug) Task c72b8ad6-2771-4b99-bfcc-0178f4281772: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000325918197632', 'lastCheck': 1384899184.1956429, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00226497650146', 'lastCheck': 1384899182.0561559, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899187.22001, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000416040420532', 'lastCheck': 1384899182.4938991, 'valid': True, 'code': 0}} >Thread-15911944::DEBUG::2013-11-19 22:13:10,859::task::495::TaskManager.Task::(_debug) Task c72b8ad6-2771-4b99-bfcc-0178f4281772: moving from state preparing -> state finished >Thread-15911944::DEBUG::2013-11-19 22:13:10,860::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911944::DEBUG::2013-11-19 22:13:10,860::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911944::DEBUG::2013-11-19 22:13:10,861::task::495::TaskManager.Task::(_debug) Task c72b8ad6-2771-4b99-bfcc-0178f4281772: ref 0 aborting False >Thread-15911944::INFO::2013-11-19 22:13:10,862::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000325918197632', 'lastCheck': 1384899184.1956429, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00226497650146', 'lastCheck': 1384899182.0561559, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899187.22001, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000416040420532', 'lastCheck': 1384899182.4938991, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:13:17,233::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911950::INFO::2013-11-19 22:13:21,150::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911950::DEBUG::2013-11-19 22:13:21,151::task::495::TaskManager.Task::(_debug) Task a8a49307-2165-4976-886c-2a27a72ba179: moving from state init -> state preparing >Thread-15911950::DEBUG::2013-11-19 22:13:21,152::task::495::TaskManager.Task::(_debug) Task a8a49307-2165-4976-886c-2a27a72ba179: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000347852706909', 'lastCheck': 1384899194.2083759, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00133109092712', 'lastCheck': 1384899192.071286, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899197.2336969, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000395059585571', 'lastCheck': 1384899192.50489, 'valid': True, 'code': 0}} >Thread-15911950::DEBUG::2013-11-19 22:13:21,156::task::495::TaskManager.Task::(_debug) Task a8a49307-2165-4976-886c-2a27a72ba179: moving from state preparing -> state finished >Thread-15911950::DEBUG::2013-11-19 22:13:21,157::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911950::DEBUG::2013-11-19 22:13:21,157::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911950::DEBUG::2013-11-19 22:13:21,158::task::495::TaskManager.Task::(_debug) Task a8a49307-2165-4976-886c-2a27a72ba179: ref 0 aborting False >Thread-15911950::INFO::2013-11-19 22:13:21,162::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000347852706909', 'lastCheck': 1384899194.2083759, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00133109092712', 'lastCheck': 1384899192.071286, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899197.2336969, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000395059585571', 'lastCheck': 1384899192.50489, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:13:27,246::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911956::INFO::2013-11-19 22:13:31,378::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911956::DEBUG::2013-11-19 22:13:31,379::task::495::TaskManager.Task::(_debug) Task be82b700-5c86-4f15-80e7-8f57c541f7e3: moving from state init -> state preparing >Thread-15911956::DEBUG::2013-11-19 22:13:31,381::task::495::TaskManager.Task::(_debug) Task be82b700-5c86-4f15-80e7-8f57c541f7e3: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000309944152832', 'lastCheck': 1384899204.2210031, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146412849426', 'lastCheck': 1384899202.0867479, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899207.2472041, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00043797492981', 'lastCheck': 1384899202.5165651, 'valid': True, 'code': 0}} >Thread-15911956::DEBUG::2013-11-19 22:13:31,389::task::495::TaskManager.Task::(_debug) Task be82b700-5c86-4f15-80e7-8f57c541f7e3: moving from state preparing -> state finished >Thread-15911956::DEBUG::2013-11-19 22:13:31,391::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911956::DEBUG::2013-11-19 22:13:31,391::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911956::DEBUG::2013-11-19 22:13:31,392::task::495::TaskManager.Task::(_debug) Task be82b700-5c86-4f15-80e7-8f57c541f7e3: ref 0 aborting False >Thread-15911956::INFO::2013-11-19 22:13:31,396::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000309944152832', 'lastCheck': 1384899204.2210031, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146412849426', 'lastCheck': 1384899202.0867479, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899207.2472041, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00043797492981', 'lastCheck': 1384899202.5165651, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:13:37,260::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911962::INFO::2013-11-19 22:13:41,548::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911962::DEBUG::2013-11-19 22:13:41,549::task::495::TaskManager.Task::(_debug) Task f3690dd7-a504-4806-a9e5-d2fdec55e834: moving from state init -> state preparing >Thread-15911962::DEBUG::2013-11-19 22:13:41,549::task::495::TaskManager.Task::(_debug) Task f3690dd7-a504-4806-a9e5-d2fdec55e834: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000406980514526', 'lastCheck': 1384899214.234071, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00143694877625', 'lastCheck': 1384899212.1022091, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899217.260875, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000401973724365', 'lastCheck': 1384899212.528116, 'valid': True, 'code': 0}} >Thread-15911962::DEBUG::2013-11-19 22:13:41,556::task::495::TaskManager.Task::(_debug) Task f3690dd7-a504-4806-a9e5-d2fdec55e834: moving from state preparing -> state finished >Thread-15911962::DEBUG::2013-11-19 22:13:41,557::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911962::DEBUG::2013-11-19 22:13:41,558::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911962::DEBUG::2013-11-19 22:13:41,558::task::495::TaskManager.Task::(_debug) Task f3690dd7-a504-4806-a9e5-d2fdec55e834: ref 0 aborting False >Thread-15911962::INFO::2013-11-19 22:13:41,559::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000406980514526', 'lastCheck': 1384899214.234071, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00143694877625', 'lastCheck': 1384899212.1022091, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899217.260875, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000401973724365', 'lastCheck': 1384899212.528116, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:13:47,273::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911968::INFO::2013-11-19 22:13:51,706::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911968::DEBUG::2013-11-19 22:13:51,706::task::495::TaskManager.Task::(_debug) Task 6a10380e-280b-4739-8df1-20f16d7a989b: moving from state init -> state preparing >Thread-15911968::DEBUG::2013-11-19 22:13:51,707::task::495::TaskManager.Task::(_debug) Task 6a10380e-280b-4739-8df1-20f16d7a989b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000365972518921', 'lastCheck': 1384899224.2470429, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00133895874023', 'lastCheck': 1384899222.117789, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899227.2744, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000395059585571', 'lastCheck': 1384899222.5395119, 'valid': True, 'code': 0}} >Thread-15911968::DEBUG::2013-11-19 22:13:51,718::task::495::TaskManager.Task::(_debug) Task 6a10380e-280b-4739-8df1-20f16d7a989b: moving from state preparing -> state finished >Thread-15911968::DEBUG::2013-11-19 22:13:51,718::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911968::DEBUG::2013-11-19 22:13:51,719::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911968::DEBUG::2013-11-19 22:13:51,719::task::495::TaskManager.Task::(_debug) Task 6a10380e-280b-4739-8df1-20f16d7a989b: ref 0 aborting False >Thread-15911968::INFO::2013-11-19 22:13:51,722::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000365972518921', 'lastCheck': 1384899224.2470429, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00133895874023', 'lastCheck': 1384899222.117789, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899227.2744, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000395059585571', 'lastCheck': 1384899222.5395119, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:13:57,287::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911974::INFO::2013-11-19 22:14:01,876::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911974::DEBUG::2013-11-19 22:14:01,876::task::495::TaskManager.Task::(_debug) Task dd52bec4-13ff-492a-bcf7-a715b40c6ab7: moving from state init -> state preparing >Thread-15911974::DEBUG::2013-11-19 22:14:01,877::task::495::TaskManager.Task::(_debug) Task dd52bec4-13ff-492a-bcf7-a715b40c6ab7: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000384092330933', 'lastCheck': 1384899234.2598851, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144505500793', 'lastCheck': 1384899232.1332829, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899237.2878251, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000345945358276', 'lastCheck': 1384899232.5510621, 'valid': True, 'code': 0}} >Thread-15911974::DEBUG::2013-11-19 22:14:01,885::task::495::TaskManager.Task::(_debug) Task dd52bec4-13ff-492a-bcf7-a715b40c6ab7: moving from state preparing -> state finished >Thread-15911974::DEBUG::2013-11-19 22:14:01,886::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911974::DEBUG::2013-11-19 22:14:01,886::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911974::DEBUG::2013-11-19 22:14:01,887::task::495::TaskManager.Task::(_debug) Task dd52bec4-13ff-492a-bcf7-a715b40c6ab7: ref 0 aborting False >Thread-15911974::INFO::2013-11-19 22:14:01,888::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000384092330933', 'lastCheck': 1384899234.2598851, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144505500793', 'lastCheck': 1384899232.1332829, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899237.2878251, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000345945358276', 'lastCheck': 1384899232.5510621, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:14:07,300::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911980::INFO::2013-11-19 22:14:12,019::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911980::DEBUG::2013-11-19 22:14:12,020::task::495::TaskManager.Task::(_debug) Task 4f5adafd-9d1c-4b7f-83ca-38a80f30535b: moving from state init -> state preparing >Thread-15911980::DEBUG::2013-11-19 22:14:12,020::task::495::TaskManager.Task::(_debug) Task 4f5adafd-9d1c-4b7f-83ca-38a80f30535b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000388860702515', 'lastCheck': 1384899244.2729459, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00193905830383', 'lastCheck': 1384899242.1497071, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899247.301384, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00049901008606', 'lastCheck': 1384899242.5630801, 'valid': True, 'code': 0}} >Thread-15911980::DEBUG::2013-11-19 22:14:12,024::task::495::TaskManager.Task::(_debug) Task 4f5adafd-9d1c-4b7f-83ca-38a80f30535b: moving from state preparing -> state finished >Thread-15911980::DEBUG::2013-11-19 22:14:12,025::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911980::DEBUG::2013-11-19 22:14:12,025::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911980::DEBUG::2013-11-19 22:14:12,026::task::495::TaskManager.Task::(_debug) Task 4f5adafd-9d1c-4b7f-83ca-38a80f30535b: ref 0 aborting False >Thread-15911980::INFO::2013-11-19 22:14:12,032::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000388860702515', 'lastCheck': 1384899244.2729459, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00193905830383', 'lastCheck': 1384899242.1497071, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899247.301384, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00049901008606', 'lastCheck': 1384899242.5630801, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:14:17,314::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911986::INFO::2013-11-19 22:14:22,160::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911986::DEBUG::2013-11-19 22:14:22,161::task::495::TaskManager.Task::(_debug) Task 52429d11-bba9-4703-a63c-28e3ddd345f6: moving from state init -> state preparing >Thread-15911986::DEBUG::2013-11-19 22:14:22,161::task::495::TaskManager.Task::(_debug) Task 52429d11-bba9-4703-a63c-28e3ddd345f6: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000406980514526', 'lastCheck': 1384899254.2859001, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00138401985168', 'lastCheck': 1384899252.163806, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899257.3148201, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000422954559326', 'lastCheck': 1384899252.5746009, 'valid': True, 'code': 0}} >Thread-15911986::DEBUG::2013-11-19 22:14:22,162::task::495::TaskManager.Task::(_debug) Task 52429d11-bba9-4703-a63c-28e3ddd345f6: moving from state preparing -> state finished >Thread-15911986::DEBUG::2013-11-19 22:14:22,163::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911986::DEBUG::2013-11-19 22:14:22,163::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911986::DEBUG::2013-11-19 22:14:22,164::task::495::TaskManager.Task::(_debug) Task 52429d11-bba9-4703-a63c-28e3ddd345f6: ref 0 aborting False >Thread-15911986::INFO::2013-11-19 22:14:22,169::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000406980514526', 'lastCheck': 1384899254.2859001, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00138401985168', 'lastCheck': 1384899252.163806, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899257.3148201, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000422954559326', 'lastCheck': 1384899252.5746009, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:14:27,327::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911992::INFO::2013-11-19 22:14:32,348::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911992::DEBUG::2013-11-19 22:14:32,349::task::495::TaskManager.Task::(_debug) Task 41afc542-8218-4160-afef-dd4bf49be72a: moving from state init -> state preparing >Thread-15911992::DEBUG::2013-11-19 22:14:32,349::task::495::TaskManager.Task::(_debug) Task 41afc542-8218-4160-afef-dd4bf49be72a: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00439286231995', 'lastCheck': 1384899264.3027229, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00135207176208', 'lastCheck': 1384899272.1942649, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899267.3283451, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000445127487183', 'lastCheck': 1384899262.586148, 'valid': True, 'code': 0}} >Thread-15911992::DEBUG::2013-11-19 22:14:32,360::task::495::TaskManager.Task::(_debug) Task 41afc542-8218-4160-afef-dd4bf49be72a: moving from state preparing -> state finished >Thread-15911992::DEBUG::2013-11-19 22:14:32,361::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911992::DEBUG::2013-11-19 22:14:32,361::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911992::DEBUG::2013-11-19 22:14:32,362::task::495::TaskManager.Task::(_debug) Task 41afc542-8218-4160-afef-dd4bf49be72a: ref 0 aborting False >Thread-15911992::INFO::2013-11-19 22:14:32,363::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00439286231995', 'lastCheck': 1384899264.3027229, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00135207176208', 'lastCheck': 1384899272.1942649, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899267.3283451, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000445127487183', 'lastCheck': 1384899262.586148, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:14:37,341::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15911998::INFO::2013-11-19 22:14:42,514::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15911998::DEBUG::2013-11-19 22:14:42,514::task::495::TaskManager.Task::(_debug) Task f7c0777e-8564-4295-a768-2be142847d84: moving from state init -> state preparing >Thread-15911998::DEBUG::2013-11-19 22:14:42,515::task::495::TaskManager.Task::(_debug) Task f7c0777e-8564-4295-a768-2be142847d84: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000399112701416', 'lastCheck': 1384899274.3156011, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152206420898', 'lastCheck': 1384899282.2101851, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899277.3423209, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000415086746216', 'lastCheck': 1384899272.597693, 'valid': True, 'code': 0}} >Thread-15911998::DEBUG::2013-11-19 22:14:42,517::task::495::TaskManager.Task::(_debug) Task f7c0777e-8564-4295-a768-2be142847d84: moving from state preparing -> state finished >Thread-15911998::DEBUG::2013-11-19 22:14:42,517::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15911998::DEBUG::2013-11-19 22:14:42,518::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15911998::DEBUG::2013-11-19 22:14:42,518::task::495::TaskManager.Task::(_debug) Task f7c0777e-8564-4295-a768-2be142847d84: ref 0 aborting False >Thread-15911998::INFO::2013-11-19 22:14:42,519::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000399112701416', 'lastCheck': 1384899274.3156011, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152206420898', 'lastCheck': 1384899282.2101851, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899277.3423209, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000415086746216', 'lastCheck': 1384899272.597693, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:14:47,355::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912004::INFO::2013-11-19 22:14:52,711::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912004::DEBUG::2013-11-19 22:14:52,711::task::495::TaskManager.Task::(_debug) Task 6ca27f82-b103-465f-8337-313a006141ed: moving from state init -> state preparing >Thread-15912004::DEBUG::2013-11-19 22:14:52,712::task::495::TaskManager.Task::(_debug) Task 6ca27f82-b103-465f-8337-313a006141ed: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000327110290527', 'lastCheck': 1384899284.328531, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147795677185', 'lastCheck': 1384899292.2256589, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899287.3562629, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000423908233643', 'lastCheck': 1384899292.621212, 'valid': True, 'code': 0}} >Thread-15912004::DEBUG::2013-11-19 22:14:52,714::task::495::TaskManager.Task::(_debug) Task 6ca27f82-b103-465f-8337-313a006141ed: moving from state preparing -> state finished >Thread-15912004::DEBUG::2013-11-19 22:14:52,714::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912004::DEBUG::2013-11-19 22:14:52,715::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912004::DEBUG::2013-11-19 22:14:52,715::task::495::TaskManager.Task::(_debug) Task 6ca27f82-b103-465f-8337-313a006141ed: ref 0 aborting False >Thread-15912004::INFO::2013-11-19 22:14:52,723::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000327110290527', 'lastCheck': 1384899284.328531, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147795677185', 'lastCheck': 1384899292.2256589, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899287.3562629, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000423908233643', 'lastCheck': 1384899292.621212, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:14:57,368::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912010::INFO::2013-11-19 22:15:02,895::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912010::DEBUG::2013-11-19 22:15:02,896::task::495::TaskManager.Task::(_debug) Task 6eb13251-e051-4bb2-9c6f-5c836f80c99b: moving from state init -> state preparing >Thread-15912010::DEBUG::2013-11-19 22:15:02,903::task::495::TaskManager.Task::(_debug) Task 6eb13251-e051-4bb2-9c6f-5c836f80c99b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000336170196533', 'lastCheck': 1384899294.3413451, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00226807594299', 'lastCheck': 1384899302.242744, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899297.3695071, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000341892242432', 'lastCheck': 1384899302.632704, 'valid': True, 'code': 0}} >Thread-15912010::DEBUG::2013-11-19 22:15:02,904::task::495::TaskManager.Task::(_debug) Task 6eb13251-e051-4bb2-9c6f-5c836f80c99b: moving from state preparing -> state finished >Thread-15912010::DEBUG::2013-11-19 22:15:02,905::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912010::DEBUG::2013-11-19 22:15:02,906::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912010::DEBUG::2013-11-19 22:15:02,906::task::495::TaskManager.Task::(_debug) Task 6eb13251-e051-4bb2-9c6f-5c836f80c99b: ref 0 aborting False >Thread-15912010::INFO::2013-11-19 22:15:02,907::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000336170196533', 'lastCheck': 1384899294.3413451, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00226807594299', 'lastCheck': 1384899302.242744, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899297.3695071, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000341892242432', 'lastCheck': 1384899302.632704, 'valid': True, 'code': 0}} >Thread-42838::DEBUG::2013-11-19 22:15:04,352::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 22:15:04,998::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:15:07,560::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912016::INFO::2013-11-19 22:15:13,071::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912016::DEBUG::2013-11-19 22:15:13,072::task::495::TaskManager.Task::(_debug) Task 5a1ed58f-8631-410b-84db-9d71ed23b7b0: moving from state init -> state preparing >Thread-15912016::DEBUG::2013-11-19 22:15:13,073::task::495::TaskManager.Task::(_debug) Task 5a1ed58f-8631-410b-84db-9d71ed23b7b0: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000308990478516', 'lastCheck': 1384899305.002492, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146317481995', 'lastCheck': 1384899312.258251, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899307.561516, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000353097915649', 'lastCheck': 1384899312.6443, 'valid': True, 'code': 0}} >Thread-15912016::DEBUG::2013-11-19 22:15:13,074::task::495::TaskManager.Task::(_debug) Task 5a1ed58f-8631-410b-84db-9d71ed23b7b0: moving from state preparing -> state finished >Thread-15912016::DEBUG::2013-11-19 22:15:13,074::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912016::DEBUG::2013-11-19 22:15:13,075::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912016::DEBUG::2013-11-19 22:15:13,076::task::495::TaskManager.Task::(_debug) Task 5a1ed58f-8631-410b-84db-9d71ed23b7b0: ref 0 aborting False >Thread-15912016::INFO::2013-11-19 22:15:13,081::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000308990478516', 'lastCheck': 1384899305.002492, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146317481995', 'lastCheck': 1384899312.258251, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899307.561516, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000353097915649', 'lastCheck': 1384899312.6443, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:15:17,573::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912022::INFO::2013-11-19 22:15:23,248::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912022::DEBUG::2013-11-19 22:15:23,249::task::495::TaskManager.Task::(_debug) Task d219e4f9-d6db-497b-bc6f-7d14f1319d44: moving from state init -> state preparing >Thread-15912022::DEBUG::2013-11-19 22:15:23,249::task::495::TaskManager.Task::(_debug) Task d219e4f9-d6db-497b-bc6f-7d14f1319d44: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000308990478516', 'lastCheck': 1384899315.015337, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0014591217041', 'lastCheck': 1384899322.273685, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899317.5743849, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000325918197632', 'lastCheck': 1384899322.6557441, 'valid': True, 'code': 0}} >Thread-15912022::DEBUG::2013-11-19 22:15:23,261::task::495::TaskManager.Task::(_debug) Task d219e4f9-d6db-497b-bc6f-7d14f1319d44: moving from state preparing -> state finished >Thread-15912022::DEBUG::2013-11-19 22:15:23,262::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912022::DEBUG::2013-11-19 22:15:23,262::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912022::DEBUG::2013-11-19 22:15:23,263::task::495::TaskManager.Task::(_debug) Task d219e4f9-d6db-497b-bc6f-7d14f1319d44: ref 0 aborting False >Thread-15912022::INFO::2013-11-19 22:15:23,263::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000308990478516', 'lastCheck': 1384899315.015337, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0014591217041', 'lastCheck': 1384899322.273685, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899317.5743849, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000325918197632', 'lastCheck': 1384899322.6557441, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:15:27,587::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912028::INFO::2013-11-19 22:15:33,404::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912028::DEBUG::2013-11-19 22:15:33,405::task::495::TaskManager.Task::(_debug) Task 82067f4e-fe11-4d7b-aa09-5283afd6580a: moving from state init -> state preparing >Thread-15912028::DEBUG::2013-11-19 22:15:33,405::task::495::TaskManager.Task::(_debug) Task 82067f4e-fe11-4d7b-aa09-5283afd6580a: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000300168991089', 'lastCheck': 1384899325.027838, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00135898590088', 'lastCheck': 1384899332.2889011, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899327.5881629, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000427007675171', 'lastCheck': 1384899332.6672339, 'valid': True, 'code': 0}} >Thread-15912028::DEBUG::2013-11-19 22:15:33,419::task::495::TaskManager.Task::(_debug) Task 82067f4e-fe11-4d7b-aa09-5283afd6580a: moving from state preparing -> state finished >Thread-15912028::DEBUG::2013-11-19 22:15:33,420::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912028::DEBUG::2013-11-19 22:15:33,420::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912028::DEBUG::2013-11-19 22:15:33,421::task::495::TaskManager.Task::(_debug) Task 82067f4e-fe11-4d7b-aa09-5283afd6580a: ref 0 aborting False >Thread-15912028::INFO::2013-11-19 22:15:33,421::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000300168991089', 'lastCheck': 1384899325.027838, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00135898590088', 'lastCheck': 1384899332.2889011, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899327.5881629, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000427007675171', 'lastCheck': 1384899332.6672339, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:15:37,601::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912034::INFO::2013-11-19 22:15:43,573::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912034::DEBUG::2013-11-19 22:15:43,574::task::495::TaskManager.Task::(_debug) Task 76010c56-a674-4529-82cb-c37dd1ac1c5c: moving from state init -> state preparing >Thread-15912034::DEBUG::2013-11-19 22:15:43,575::task::495::TaskManager.Task::(_debug) Task 76010c56-a674-4529-82cb-c37dd1ac1c5c: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000317096710205', 'lastCheck': 1384899335.04055, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00141906738281', 'lastCheck': 1384899342.3046041, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899337.601974, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000339984893799', 'lastCheck': 1384899342.6787241, 'valid': True, 'code': 0}} >Thread-15912034::DEBUG::2013-11-19 22:15:43,587::task::495::TaskManager.Task::(_debug) Task 76010c56-a674-4529-82cb-c37dd1ac1c5c: moving from state preparing -> state finished >Thread-15912034::DEBUG::2013-11-19 22:15:43,593::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912034::DEBUG::2013-11-19 22:15:43,594::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912034::DEBUG::2013-11-19 22:15:43,594::task::495::TaskManager.Task::(_debug) Task 76010c56-a674-4529-82cb-c37dd1ac1c5c: ref 0 aborting False >Thread-15912034::INFO::2013-11-19 22:15:43,595::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000317096710205', 'lastCheck': 1384899335.04055, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00141906738281', 'lastCheck': 1384899342.3046041, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899337.601974, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000339984893799', 'lastCheck': 1384899342.6787241, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:15:47,614::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912040::INFO::2013-11-19 22:15:53,883::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912040::DEBUG::2013-11-19 22:15:53,884::task::495::TaskManager.Task::(_debug) Task adcd5630-0a0d-4724-9d21-8035f0dccadc: moving from state init -> state preparing >Thread-15912040::DEBUG::2013-11-19 22:15:53,884::task::495::TaskManager.Task::(_debug) Task adcd5630-0a0d-4724-9d21-8035f0dccadc: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000272035598755', 'lastCheck': 1384899345.0530601, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00141286849976', 'lastCheck': 1384899352.3201671, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899347.615418, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000418901443481', 'lastCheck': 1384899352.6902349, 'valid': True, 'code': 0}} >Thread-15912040::DEBUG::2013-11-19 22:15:53,896::task::495::TaskManager.Task::(_debug) Task adcd5630-0a0d-4724-9d21-8035f0dccadc: moving from state preparing -> state finished >Thread-15912040::DEBUG::2013-11-19 22:15:53,897::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912040::DEBUG::2013-11-19 22:15:53,897::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912040::DEBUG::2013-11-19 22:15:53,898::task::495::TaskManager.Task::(_debug) Task adcd5630-0a0d-4724-9d21-8035f0dccadc: ref 0 aborting False >Thread-15912040::INFO::2013-11-19 22:15:53,898::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000272035598755', 'lastCheck': 1384899345.0530601, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00141286849976', 'lastCheck': 1384899352.3201671, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899347.615418, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000418901443481', 'lastCheck': 1384899352.6902349, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:15:57,628::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912046::INFO::2013-11-19 22:16:04,421::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912046::DEBUG::2013-11-19 22:16:04,422::task::495::TaskManager.Task::(_debug) Task 8dd6fe8d-010b-4f35-a5fb-8001b82949a8: moving from state init -> state preparing >Thread-15912046::DEBUG::2013-11-19 22:16:04,423::task::495::TaskManager.Task::(_debug) Task 8dd6fe8d-010b-4f35-a5fb-8001b82949a8: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000263929367065', 'lastCheck': 1384899355.0655861, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00213098526001', 'lastCheck': 1384899362.3366809, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899357.628926, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000370025634766', 'lastCheck': 1384899362.70175, 'valid': True, 'code': 0}} >Thread-15912046::DEBUG::2013-11-19 22:16:04,424::task::495::TaskManager.Task::(_debug) Task 8dd6fe8d-010b-4f35-a5fb-8001b82949a8: moving from state preparing -> state finished >Thread-15912046::DEBUG::2013-11-19 22:16:04,424::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912046::DEBUG::2013-11-19 22:16:04,425::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912046::DEBUG::2013-11-19 22:16:04,425::task::495::TaskManager.Task::(_debug) Task 8dd6fe8d-010b-4f35-a5fb-8001b82949a8: ref 0 aborting False >Thread-15912046::INFO::2013-11-19 22:16:04,426::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000263929367065', 'lastCheck': 1384899355.0655861, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00213098526001', 'lastCheck': 1384899362.3366809, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899357.628926, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000370025634766', 'lastCheck': 1384899362.70175, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:16:07,641::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912052::INFO::2013-11-19 22:16:15,280::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912052::DEBUG::2013-11-19 22:16:15,281::task::495::TaskManager.Task::(_debug) Task 77d006e6-eb45-461e-ab13-2847ba63a842: moving from state init -> state preparing >Thread-15912052::DEBUG::2013-11-19 22:16:15,282::task::495::TaskManager.Task::(_debug) Task 77d006e6-eb45-461e-ab13-2847ba63a842: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000263214111328', 'lastCheck': 1384899375.090596, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00138807296753', 'lastCheck': 1384899372.3520391, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899367.642592, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000328063964844', 'lastCheck': 1384899372.7132051, 'valid': True, 'code': 0}} >Thread-15912052::DEBUG::2013-11-19 22:16:15,288::task::495::TaskManager.Task::(_debug) Task 77d006e6-eb45-461e-ab13-2847ba63a842: moving from state preparing -> state finished >Thread-15912052::DEBUG::2013-11-19 22:16:15,288::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912052::DEBUG::2013-11-19 22:16:15,289::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912052::DEBUG::2013-11-19 22:16:15,290::task::495::TaskManager.Task::(_debug) Task 77d006e6-eb45-461e-ab13-2847ba63a842: ref 0 aborting False >Thread-15912052::INFO::2013-11-19 22:16:15,290::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000263214111328', 'lastCheck': 1384899375.090596, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00138807296753', 'lastCheck': 1384899372.3520391, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899367.642592, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000328063964844', 'lastCheck': 1384899372.7132051, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:16:17,655::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912058::INFO::2013-11-19 22:16:25,398::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912058::DEBUG::2013-11-19 22:16:25,399::task::495::TaskManager.Task::(_debug) Task df99c9b3-5eea-43bb-a7ad-7cc2bc65fb77: moving from state init -> state preparing >Thread-15912058::DEBUG::2013-11-19 22:16:25,399::task::495::TaskManager.Task::(_debug) Task df99c9b3-5eea-43bb-a7ad-7cc2bc65fb77: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000309944152832', 'lastCheck': 1384899385.1031539, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00135612487793', 'lastCheck': 1384899382.367857, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899377.656112, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000344038009644', 'lastCheck': 1384899382.7247281, 'valid': True, 'code': 0}} >Thread-15912058::DEBUG::2013-11-19 22:16:25,407::task::495::TaskManager.Task::(_debug) Task df99c9b3-5eea-43bb-a7ad-7cc2bc65fb77: moving from state preparing -> state finished >Thread-15912058::DEBUG::2013-11-19 22:16:25,408::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912058::DEBUG::2013-11-19 22:16:25,408::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912058::DEBUG::2013-11-19 22:16:25,409::task::495::TaskManager.Task::(_debug) Task df99c9b3-5eea-43bb-a7ad-7cc2bc65fb77: ref 0 aborting False >Thread-15912058::INFO::2013-11-19 22:16:25,410::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000309944152832', 'lastCheck': 1384899385.1031539, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00135612487793', 'lastCheck': 1384899382.367857, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899377.656112, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000344038009644', 'lastCheck': 1384899382.7247281, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:16:27,669::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912064::INFO::2013-11-19 22:16:35,583::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912064::DEBUG::2013-11-19 22:16:35,584::task::495::TaskManager.Task::(_debug) Task a80a3137-9e6d-40ac-b71d-06f5e8c7ba77: moving from state init -> state preparing >Thread-15912064::DEBUG::2013-11-19 22:16:35,585::task::495::TaskManager.Task::(_debug) Task a80a3137-9e6d-40ac-b71d-06f5e8c7ba77: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000319004058838', 'lastCheck': 1384899395.11589, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00136184692383', 'lastCheck': 1384899392.383126, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899387.6697431, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000412940979004', 'lastCheck': 1384899392.7363341, 'valid': True, 'code': 0}} >Thread-15912064::DEBUG::2013-11-19 22:16:35,592::task::495::TaskManager.Task::(_debug) Task a80a3137-9e6d-40ac-b71d-06f5e8c7ba77: moving from state preparing -> state finished >Thread-15912064::DEBUG::2013-11-19 22:16:35,592::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912064::DEBUG::2013-11-19 22:16:35,593::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912064::DEBUG::2013-11-19 22:16:35,593::task::495::TaskManager.Task::(_debug) Task a80a3137-9e6d-40ac-b71d-06f5e8c7ba77: ref 0 aborting False >Thread-15912064::INFO::2013-11-19 22:16:35,596::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000319004058838', 'lastCheck': 1384899395.11589, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00136184692383', 'lastCheck': 1384899392.383126, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899387.6697431, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000412940979004', 'lastCheck': 1384899392.7363341, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:16:37,683::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912070::INFO::2013-11-19 22:16:45,741::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912070::DEBUG::2013-11-19 22:16:45,742::task::495::TaskManager.Task::(_debug) Task d616bc14-e563-40e6-b281-edc31c2d3ca6: moving from state init -> state preparing >Thread-15912070::DEBUG::2013-11-19 22:16:45,742::task::495::TaskManager.Task::(_debug) Task d616bc14-e563-40e6-b281-edc31c2d3ca6: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000339031219482', 'lastCheck': 1384899405.1285341, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0014009475708', 'lastCheck': 1384899402.3989811, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899397.683785, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000348806381226', 'lastCheck': 1384899402.7478061, 'valid': True, 'code': 0}} >Thread-15912070::DEBUG::2013-11-19 22:16:45,753::task::495::TaskManager.Task::(_debug) Task d616bc14-e563-40e6-b281-edc31c2d3ca6: moving from state preparing -> state finished >Thread-15912070::DEBUG::2013-11-19 22:16:45,754::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912070::DEBUG::2013-11-19 22:16:45,754::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912070::DEBUG::2013-11-19 22:16:45,755::task::495::TaskManager.Task::(_debug) Task d616bc14-e563-40e6-b281-edc31c2d3ca6: ref 0 aborting False >Thread-15912070::INFO::2013-11-19 22:16:45,756::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000339031219482', 'lastCheck': 1384899405.1285341, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0014009475708', 'lastCheck': 1384899402.3989811, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899397.683785, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000348806381226', 'lastCheck': 1384899402.7478061, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:16:47,696::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912076::INFO::2013-11-19 22:16:57,188::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912076::DEBUG::2013-11-19 22:16:57,189::task::495::TaskManager.Task::(_debug) Task c90c6426-4327-408e-87a8-fd503b15ec55: moving from state init -> state preparing >Thread-15912076::DEBUG::2013-11-19 22:16:57,190::task::495::TaskManager.Task::(_debug) Task c90c6426-4327-408e-87a8-fd503b15ec55: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000268220901489', 'lastCheck': 1384899415.141047, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00141191482544', 'lastCheck': 1384899412.414417, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899407.6973529, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00032901763916', 'lastCheck': 1384899412.7615421, 'valid': True, 'code': 0}} >Thread-15912076::DEBUG::2013-11-19 22:16:57,197::task::495::TaskManager.Task::(_debug) Task c90c6426-4327-408e-87a8-fd503b15ec55: moving from state preparing -> state finished >Thread-15912076::DEBUG::2013-11-19 22:16:57,198::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912076::DEBUG::2013-11-19 22:16:57,198::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912076::DEBUG::2013-11-19 22:16:57,199::task::495::TaskManager.Task::(_debug) Task c90c6426-4327-408e-87a8-fd503b15ec55: ref 0 aborting False >Thread-15912076::INFO::2013-11-19 22:16:57,200::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000268220901489', 'lastCheck': 1384899415.141047, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00141191482544', 'lastCheck': 1384899412.414417, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899407.6973529, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00032901763916', 'lastCheck': 1384899412.7615421, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:16:57,710::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912082::INFO::2013-11-19 22:17:07,335::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912082::DEBUG::2013-11-19 22:17:07,335::task::495::TaskManager.Task::(_debug) Task 2f6bf88f-f4d0-472f-a44f-ee821980a76d: moving from state init -> state preparing >Thread-15912082::DEBUG::2013-11-19 22:17:07,336::task::495::TaskManager.Task::(_debug) Task 2f6bf88f-f4d0-472f-a44f-ee821980a76d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000919818878174', 'lastCheck': 1384899425.1547389, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0023238658905', 'lastCheck': 1384899422.431107, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899417.711009, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000437021255493', 'lastCheck': 1384899422.773242, 'valid': True, 'code': 0}} >Thread-15912082::DEBUG::2013-11-19 22:17:07,356::task::495::TaskManager.Task::(_debug) Task 2f6bf88f-f4d0-472f-a44f-ee821980a76d: moving from state preparing -> state finished >Thread-15912082::DEBUG::2013-11-19 22:17:07,356::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912082::DEBUG::2013-11-19 22:17:07,357::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912082::DEBUG::2013-11-19 22:17:07,358::task::495::TaskManager.Task::(_debug) Task 2f6bf88f-f4d0-472f-a44f-ee821980a76d: ref 0 aborting False >Thread-15912082::INFO::2013-11-19 22:17:07,362::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000919818878174', 'lastCheck': 1384899425.1547389, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0023238658905', 'lastCheck': 1384899422.431107, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899417.711009, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000437021255493', 'lastCheck': 1384899422.773242, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:17:07,724::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912088::INFO::2013-11-19 22:17:17,492::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912088::DEBUG::2013-11-19 22:17:17,492::task::495::TaskManager.Task::(_debug) Task 52e4651a-9f84-4dd0-b538-51cee427a12d: moving from state init -> state preparing >Thread-15912088::DEBUG::2013-11-19 22:17:17,493::task::495::TaskManager.Task::(_debug) Task 52e4651a-9f84-4dd0-b538-51cee427a12d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000259876251221', 'lastCheck': 1384899435.167232, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00140595436096', 'lastCheck': 1384899432.446408, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899427.7247379, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000328063964844', 'lastCheck': 1384899432.7848229, 'valid': True, 'code': 0}} >Thread-15912088::DEBUG::2013-11-19 22:17:17,502::task::495::TaskManager.Task::(_debug) Task 52e4651a-9f84-4dd0-b538-51cee427a12d: moving from state preparing -> state finished >Thread-15912088::DEBUG::2013-11-19 22:17:17,503::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912088::DEBUG::2013-11-19 22:17:17,504::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912088::DEBUG::2013-11-19 22:17:17,504::task::495::TaskManager.Task::(_debug) Task 52e4651a-9f84-4dd0-b538-51cee427a12d: ref 0 aborting False >Thread-15912088::INFO::2013-11-19 22:17:17,505::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000259876251221', 'lastCheck': 1384899435.167232, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00140595436096', 'lastCheck': 1384899432.446408, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899427.7247379, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000328063964844', 'lastCheck': 1384899432.7848229, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:17:17,737::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912094::INFO::2013-11-19 22:17:27,694::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912094::DEBUG::2013-11-19 22:17:27,695::task::495::TaskManager.Task::(_debug) Task 0e84c412-00ed-4ff8-bdef-6113c8e555cb: moving from state init -> state preparing >Thread-15912094::DEBUG::2013-11-19 22:17:27,696::task::495::TaskManager.Task::(_debug) Task 0e84c412-00ed-4ff8-bdef-6113c8e555cb: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000349044799805', 'lastCheck': 1384899445.1791861, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147795677185', 'lastCheck': 1384899442.4622481, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899437.7386169, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000340938568115', 'lastCheck': 1384899442.79636, 'valid': True, 'code': 0}} >Thread-15912094::DEBUG::2013-11-19 22:17:27,704::task::495::TaskManager.Task::(_debug) Task 0e84c412-00ed-4ff8-bdef-6113c8e555cb: moving from state preparing -> state finished >Thread-15912094::DEBUG::2013-11-19 22:17:27,704::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912094::DEBUG::2013-11-19 22:17:27,705::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912094::DEBUG::2013-11-19 22:17:27,705::task::495::TaskManager.Task::(_debug) Task 0e84c412-00ed-4ff8-bdef-6113c8e555cb: ref 0 aborting False >Thread-15912094::INFO::2013-11-19 22:17:27,709::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000349044799805', 'lastCheck': 1384899445.1791861, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147795677185', 'lastCheck': 1384899442.4622481, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899437.7386169, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000340938568115', 'lastCheck': 1384899442.79636, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:17:27,751::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 22:17:32,807::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 22:17:33,429::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:17:37,766::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912100::INFO::2013-11-19 22:17:37,900::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912100::DEBUG::2013-11-19 22:17:37,901::task::495::TaskManager.Task::(_debug) Task 476e9905-e3a3-4147-ad7a-16f533f697e0: moving from state init -> state preparing >Thread-15912100::DEBUG::2013-11-19 22:17:37,902::task::495::TaskManager.Task::(_debug) Task 476e9905-e3a3-4147-ad7a-16f533f697e0: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000566005706787', 'lastCheck': 1384899455.1925609, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00143504142761', 'lastCheck': 1384899452.477886, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899457.7668591, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000300884246826', 'lastCheck': 1384899453.4319601, 'valid': True, 'code': 0}} >Thread-15912100::DEBUG::2013-11-19 22:17:37,903::task::495::TaskManager.Task::(_debug) Task 476e9905-e3a3-4147-ad7a-16f533f697e0: moving from state preparing -> state finished >Thread-15912100::DEBUG::2013-11-19 22:17:37,903::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912100::DEBUG::2013-11-19 22:17:37,904::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912100::DEBUG::2013-11-19 22:17:37,904::task::495::TaskManager.Task::(_debug) Task 476e9905-e3a3-4147-ad7a-16f533f697e0: ref 0 aborting False >Thread-15912100::INFO::2013-11-19 22:17:37,905::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000566005706787', 'lastCheck': 1384899455.1925609, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00143504142761', 'lastCheck': 1384899452.477886, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899457.7668591, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000300884246826', 'lastCheck': 1384899453.4319601, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:17:47,779::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912106::INFO::2013-11-19 22:17:48,044::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912106::DEBUG::2013-11-19 22:17:48,044::task::495::TaskManager.Task::(_debug) Task edc323eb-d6d6-4569-9aa4-6fbb7721876a: moving from state init -> state preparing >Thread-15912106::DEBUG::2013-11-19 22:17:48,045::task::495::TaskManager.Task::(_debug) Task edc323eb-d6d6-4569-9aa4-6fbb7721876a: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000302076339722', 'lastCheck': 1384899465.2050431, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00219297409058', 'lastCheck': 1384899462.49615, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899467.7805221, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000438928604126', 'lastCheck': 1384899463.4437339, 'valid': True, 'code': 0}} >Thread-15912106::DEBUG::2013-11-19 22:17:48,049::task::495::TaskManager.Task::(_debug) Task edc323eb-d6d6-4569-9aa4-6fbb7721876a: moving from state preparing -> state finished >Thread-15912106::DEBUG::2013-11-19 22:17:48,050::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912106::DEBUG::2013-11-19 22:17:48,050::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912106::DEBUG::2013-11-19 22:17:48,051::task::495::TaskManager.Task::(_debug) Task edc323eb-d6d6-4569-9aa4-6fbb7721876a: ref 0 aborting False >Thread-15912106::INFO::2013-11-19 22:17:48,056::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000302076339722', 'lastCheck': 1384899465.2050431, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00219297409058', 'lastCheck': 1384899462.49615, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899467.7805221, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000438928604126', 'lastCheck': 1384899463.4437339, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:17:57,793::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912112::INFO::2013-11-19 22:17:58,198::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912112::DEBUG::2013-11-19 22:17:58,198::task::495::TaskManager.Task::(_debug) Task 3a8cf4d9-9e77-47ec-b28f-7164ae3fd1b8: moving from state init -> state preparing >Thread-15912112::DEBUG::2013-11-19 22:17:58,199::task::495::TaskManager.Task::(_debug) Task 3a8cf4d9-9e77-47ec-b28f-7164ae3fd1b8: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00026798248291', 'lastCheck': 1384899475.21753, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0012218952179', 'lastCheck': 1384899472.51017, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899477.794246, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000416994094849', 'lastCheck': 1384899473.4554241, 'valid': True, 'code': 0}} >Thread-15912112::DEBUG::2013-11-19 22:17:58,213::task::495::TaskManager.Task::(_debug) Task 3a8cf4d9-9e77-47ec-b28f-7164ae3fd1b8: moving from state preparing -> state finished >Thread-15912112::DEBUG::2013-11-19 22:17:58,213::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912112::DEBUG::2013-11-19 22:17:58,214::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912112::DEBUG::2013-11-19 22:17:58,214::task::495::TaskManager.Task::(_debug) Task 3a8cf4d9-9e77-47ec-b28f-7164ae3fd1b8: ref 0 aborting False >Thread-15912112::INFO::2013-11-19 22:17:58,219::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00026798248291', 'lastCheck': 1384899475.21753, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0012218952179', 'lastCheck': 1384899472.51017, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899477.794246, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000416994094849', 'lastCheck': 1384899473.4554241, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:18:07,807::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912118::INFO::2013-11-19 22:18:08,356::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912118::DEBUG::2013-11-19 22:18:08,357::task::495::TaskManager.Task::(_debug) Task f5d12dbc-a265-4c34-b5c0-d465ddb30744: moving from state init -> state preparing >Thread-15912118::DEBUG::2013-11-19 22:18:08,357::task::495::TaskManager.Task::(_debug) Task f5d12dbc-a265-4c34-b5c0-d465ddb30744: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000292062759399', 'lastCheck': 1384899485.2325189, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00216388702393', 'lastCheck': 1384899482.526546, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899487.808006, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000427961349487', 'lastCheck': 1384899483.467, 'valid': True, 'code': 0}} >Thread-15912118::DEBUG::2013-11-19 22:18:08,358::task::495::TaskManager.Task::(_debug) Task f5d12dbc-a265-4c34-b5c0-d465ddb30744: moving from state preparing -> state finished >Thread-15912118::DEBUG::2013-11-19 22:18:08,359::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912118::DEBUG::2013-11-19 22:18:08,359::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912118::DEBUG::2013-11-19 22:18:08,360::task::495::TaskManager.Task::(_debug) Task f5d12dbc-a265-4c34-b5c0-d465ddb30744: ref 0 aborting False >Thread-15912118::INFO::2013-11-19 22:18:08,364::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000292062759399', 'lastCheck': 1384899485.2325189, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00216388702393', 'lastCheck': 1384899482.526546, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899487.808006, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000427961349487', 'lastCheck': 1384899483.467, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:18:17,820::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912124::INFO::2013-11-19 22:18:18,522::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912124::DEBUG::2013-11-19 22:18:18,523::task::495::TaskManager.Task::(_debug) Task d1288538-90cb-46ff-ab5f-f66387c99922: moving from state init -> state preparing >Thread-15912124::DEBUG::2013-11-19 22:18:18,524::task::495::TaskManager.Task::(_debug) Task d1288538-90cb-46ff-ab5f-f66387c99922: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000270128250122', 'lastCheck': 1384899495.2450039, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139284133911', 'lastCheck': 1384899492.541883, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899497.821532, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000416994094849', 'lastCheck': 1384899493.478533, 'valid': True, 'code': 0}} >Thread-15912124::DEBUG::2013-11-19 22:18:18,535::task::495::TaskManager.Task::(_debug) Task d1288538-90cb-46ff-ab5f-f66387c99922: moving from state preparing -> state finished >Thread-15912124::DEBUG::2013-11-19 22:18:18,536::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912124::DEBUG::2013-11-19 22:18:18,536::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912124::DEBUG::2013-11-19 22:18:18,537::task::495::TaskManager.Task::(_debug) Task d1288538-90cb-46ff-ab5f-f66387c99922: ref 0 aborting False >Thread-15912124::INFO::2013-11-19 22:18:18,537::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000270128250122', 'lastCheck': 1384899495.2450039, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139284133911', 'lastCheck': 1384899492.541883, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899497.821532, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000416994094849', 'lastCheck': 1384899493.478533, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:18:27,834::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912130::INFO::2013-11-19 22:18:29,433::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912130::DEBUG::2013-11-19 22:18:29,434::task::495::TaskManager.Task::(_debug) Task 5b55d6bb-1199-4997-8925-251cc901e01d: moving from state init -> state preparing >Thread-15912130::DEBUG::2013-11-19 22:18:29,435::task::495::TaskManager.Task::(_debug) Task 5b55d6bb-1199-4997-8925-251cc901e01d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000297069549561', 'lastCheck': 1384899505.2576079, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00141501426697', 'lastCheck': 1384899502.5570641, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899507.835078, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000444889068604', 'lastCheck': 1384899503.4905341, 'valid': True, 'code': 0}} >Thread-15912130::DEBUG::2013-11-19 22:18:29,435::task::495::TaskManager.Task::(_debug) Task 5b55d6bb-1199-4997-8925-251cc901e01d: moving from state preparing -> state finished >Thread-15912130::DEBUG::2013-11-19 22:18:29,436::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912130::DEBUG::2013-11-19 22:18:29,437::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912130::DEBUG::2013-11-19 22:18:29,437::task::495::TaskManager.Task::(_debug) Task 5b55d6bb-1199-4997-8925-251cc901e01d: ref 0 aborting False >Thread-15912130::INFO::2013-11-19 22:18:29,438::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000297069549561', 'lastCheck': 1384899505.2576079, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00141501426697', 'lastCheck': 1384899502.5570641, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899507.835078, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000444889068604', 'lastCheck': 1384899503.4905341, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:18:37,848::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912136::INFO::2013-11-19 22:18:39,559::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912136::DEBUG::2013-11-19 22:18:39,560::task::495::TaskManager.Task::(_debug) Task 1fd652de-e184-4468-af59-8354af2964a9: moving from state init -> state preparing >Thread-15912136::DEBUG::2013-11-19 22:18:39,560::task::495::TaskManager.Task::(_debug) Task 1fd652de-e184-4468-af59-8354af2964a9: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000331878662109', 'lastCheck': 1384899515.2703471, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00134015083313', 'lastCheck': 1384899512.5721719, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899517.84884, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000470876693726', 'lastCheck': 1384899513.5021441, 'valid': True, 'code': 0}} >Thread-15912136::DEBUG::2013-11-19 22:18:39,561::task::495::TaskManager.Task::(_debug) Task 1fd652de-e184-4468-af59-8354af2964a9: moving from state preparing -> state finished >Thread-15912136::DEBUG::2013-11-19 22:18:39,562::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912136::DEBUG::2013-11-19 22:18:39,563::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912136::DEBUG::2013-11-19 22:18:39,563::task::495::TaskManager.Task::(_debug) Task 1fd652de-e184-4468-af59-8354af2964a9: ref 0 aborting False >Thread-15912136::INFO::2013-11-19 22:18:39,569::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000331878662109', 'lastCheck': 1384899515.2703471, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00134015083313', 'lastCheck': 1384899512.5721719, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899517.84884, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000470876693726', 'lastCheck': 1384899513.5021441, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:18:47,861::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912142::INFO::2013-11-19 22:18:50,340::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912142::DEBUG::2013-11-19 22:18:50,340::task::495::TaskManager.Task::(_debug) Task 1c3a0894-e91f-4ec2-9b02-0e6da2c02a8a: moving from state init -> state preparing >Thread-15912142::DEBUG::2013-11-19 22:18:50,341::task::495::TaskManager.Task::(_debug) Task 1c3a0894-e91f-4ec2-9b02-0e6da2c02a8a: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000277042388916', 'lastCheck': 1384899525.2828851, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00138711929321', 'lastCheck': 1384899522.5883319, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899527.862237, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000442028045654', 'lastCheck': 1384899523.5135751, 'valid': True, 'code': 0}} >Thread-15912142::DEBUG::2013-11-19 22:18:50,346::task::495::TaskManager.Task::(_debug) Task 1c3a0894-e91f-4ec2-9b02-0e6da2c02a8a: moving from state preparing -> state finished >Thread-15912142::DEBUG::2013-11-19 22:18:50,347::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912142::DEBUG::2013-11-19 22:18:50,347::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912142::DEBUG::2013-11-19 22:18:50,348::task::495::TaskManager.Task::(_debug) Task 1c3a0894-e91f-4ec2-9b02-0e6da2c02a8a: ref 0 aborting False >Thread-15912142::INFO::2013-11-19 22:18:50,349::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000277042388916', 'lastCheck': 1384899525.2828851, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00138711929321', 'lastCheck': 1384899522.5883319, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899527.862237, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000442028045654', 'lastCheck': 1384899523.5135751, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:18:57,874::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912148::INFO::2013-11-19 22:19:00,492::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912148::DEBUG::2013-11-19 22:19:00,493::task::495::TaskManager.Task::(_debug) Task a466f55e-c069-457a-8fa5-9db3b1e618f4: moving from state init -> state preparing >Thread-15912148::DEBUG::2013-11-19 22:19:00,503::task::495::TaskManager.Task::(_debug) Task a466f55e-c069-457a-8fa5-9db3b1e618f4: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000269174575806', 'lastCheck': 1384899535.295486, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00187802314758', 'lastCheck': 1384899532.6040151, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899537.875514, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000416994094849', 'lastCheck': 1384899533.524935, 'valid': True, 'code': 0}} >Thread-15912148::DEBUG::2013-11-19 22:19:00,505::task::495::TaskManager.Task::(_debug) Task a466f55e-c069-457a-8fa5-9db3b1e618f4: moving from state preparing -> state finished >Thread-15912148::DEBUG::2013-11-19 22:19:00,507::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912148::DEBUG::2013-11-19 22:19:00,509::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912148::DEBUG::2013-11-19 22:19:00,510::task::495::TaskManager.Task::(_debug) Task a466f55e-c069-457a-8fa5-9db3b1e618f4: ref 0 aborting False >Thread-15912148::INFO::2013-11-19 22:19:00,518::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000269174575806', 'lastCheck': 1384899535.295486, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00187802314758', 'lastCheck': 1384899532.6040151, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899537.875514, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000416994094849', 'lastCheck': 1384899533.524935, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:19:07,888::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912154::INFO::2013-11-19 22:19:10,678::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912154::DEBUG::2013-11-19 22:19:10,679::task::495::TaskManager.Task::(_debug) Task 0687b899-9b9e-4f4c-ae90-cdcb51b152f8: moving from state init -> state preparing >Thread-15912154::DEBUG::2013-11-19 22:19:10,680::task::495::TaskManager.Task::(_debug) Task 0687b899-9b9e-4f4c-ae90-cdcb51b152f8: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.0002601146698', 'lastCheck': 1384899545.3079841, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00216698646545', 'lastCheck': 1384899542.6189549, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899547.8887539, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000435829162598', 'lastCheck': 1384899543.5361271, 'valid': True, 'code': 0}} >Thread-15912154::DEBUG::2013-11-19 22:19:10,687::task::495::TaskManager.Task::(_debug) Task 0687b899-9b9e-4f4c-ae90-cdcb51b152f8: moving from state preparing -> state finished >Thread-15912154::DEBUG::2013-11-19 22:19:10,688::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912154::DEBUG::2013-11-19 22:19:10,688::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912154::DEBUG::2013-11-19 22:19:10,689::task::495::TaskManager.Task::(_debug) Task 0687b899-9b9e-4f4c-ae90-cdcb51b152f8: ref 0 aborting False >Thread-15912154::INFO::2013-11-19 22:19:10,692::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.0002601146698', 'lastCheck': 1384899545.3079841, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00216698646545', 'lastCheck': 1384899542.6189549, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899547.8887539, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000435829162598', 'lastCheck': 1384899543.5361271, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:19:17,901::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912160::INFO::2013-11-19 22:19:20,857::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912160::DEBUG::2013-11-19 22:19:20,858::task::495::TaskManager.Task::(_debug) Task 931fb449-27e3-4428-beb4-292adcd1b745: moving from state init -> state preparing >Thread-15912160::DEBUG::2013-11-19 22:19:20,858::task::495::TaskManager.Task::(_debug) Task 931fb449-27e3-4428-beb4-292adcd1b745: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000334024429321', 'lastCheck': 1384899555.320451, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00131106376648', 'lastCheck': 1384899552.634002, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899557.90201, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000376224517822', 'lastCheck': 1384899553.547652, 'valid': True, 'code': 0}} >Thread-15912160::DEBUG::2013-11-19 22:19:20,868::task::495::TaskManager.Task::(_debug) Task 931fb449-27e3-4428-beb4-292adcd1b745: moving from state preparing -> state finished >Thread-15912160::DEBUG::2013-11-19 22:19:20,869::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912160::DEBUG::2013-11-19 22:19:20,869::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912160::DEBUG::2013-11-19 22:19:20,870::task::495::TaskManager.Task::(_debug) Task 931fb449-27e3-4428-beb4-292adcd1b745: ref 0 aborting False >Thread-15912160::INFO::2013-11-19 22:19:20,875::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000334024429321', 'lastCheck': 1384899555.320451, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00131106376648', 'lastCheck': 1384899552.634002, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899557.90201, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000376224517822', 'lastCheck': 1384899553.547652, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:19:27,914::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912166::INFO::2013-11-19 22:19:31,087::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912166::DEBUG::2013-11-19 22:19:31,088::task::495::TaskManager.Task::(_debug) Task e120a92d-ea55-4304-9ee9-469396fa9037: moving from state init -> state preparing >Thread-15912166::DEBUG::2013-11-19 22:19:31,099::task::495::TaskManager.Task::(_debug) Task e120a92d-ea55-4304-9ee9-469396fa9037: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000263929367065', 'lastCheck': 1384899565.332953, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00140619277954', 'lastCheck': 1384899562.649209, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899567.9150159, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000338077545166', 'lastCheck': 1384899563.55913, 'valid': True, 'code': 0}} >Thread-15912166::DEBUG::2013-11-19 22:19:31,109::task::495::TaskManager.Task::(_debug) Task e120a92d-ea55-4304-9ee9-469396fa9037: moving from state preparing -> state finished >Thread-15912166::DEBUG::2013-11-19 22:19:31,109::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912166::DEBUG::2013-11-19 22:19:31,110::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912166::DEBUG::2013-11-19 22:19:31,110::task::495::TaskManager.Task::(_debug) Task e120a92d-ea55-4304-9ee9-469396fa9037: ref 0 aborting False >Thread-15912166::INFO::2013-11-19 22:19:31,111::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000263929367065', 'lastCheck': 1384899565.332953, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00140619277954', 'lastCheck': 1384899562.649209, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899567.9150159, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000338077545166', 'lastCheck': 1384899563.55913, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:19:37,928::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912172::INFO::2013-11-19 22:19:41,245::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912172::DEBUG::2013-11-19 22:19:41,246::task::495::TaskManager.Task::(_debug) Task 11ea9bf7-8821-4ffe-bf78-4cc2c615fffc: moving from state init -> state preparing >Thread-15912172::DEBUG::2013-11-19 22:19:41,246::task::495::TaskManager.Task::(_debug) Task 11ea9bf7-8821-4ffe-bf78-4cc2c615fffc: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000371932983398', 'lastCheck': 1384899575.3456371, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00134205818176', 'lastCheck': 1384899572.664288, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899577.9289351, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000322103500366', 'lastCheck': 1384899573.57059, 'valid': True, 'code': 0}} >Thread-15912172::DEBUG::2013-11-19 22:19:41,247::task::495::TaskManager.Task::(_debug) Task 11ea9bf7-8821-4ffe-bf78-4cc2c615fffc: moving from state preparing -> state finished >Thread-15912172::DEBUG::2013-11-19 22:19:41,248::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912172::DEBUG::2013-11-19 22:19:41,248::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912172::DEBUG::2013-11-19 22:19:41,249::task::495::TaskManager.Task::(_debug) Task 11ea9bf7-8821-4ffe-bf78-4cc2c615fffc: ref 0 aborting False >Thread-15912172::INFO::2013-11-19 22:19:41,258::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000371932983398', 'lastCheck': 1384899575.3456371, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00134205818176', 'lastCheck': 1384899572.664288, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899577.9289351, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000322103500366', 'lastCheck': 1384899573.57059, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:19:47,941::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912178::INFO::2013-11-19 22:19:51,382::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912178::DEBUG::2013-11-19 22:19:51,383::task::495::TaskManager.Task::(_debug) Task 7220fbeb-9a3c-468e-8a09-7da0f9d0599f: moving from state init -> state preparing >Thread-15912178::DEBUG::2013-11-19 22:19:51,383::task::495::TaskManager.Task::(_debug) Task 7220fbeb-9a3c-468e-8a09-7da0f9d0599f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000255823135376', 'lastCheck': 1384899585.358366, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00151896476746', 'lastCheck': 1384899582.6801791, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899587.942589, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000344038009644', 'lastCheck': 1384899583.5820351, 'valid': True, 'code': 0}} >Thread-15912178::DEBUG::2013-11-19 22:19:51,392::task::495::TaskManager.Task::(_debug) Task 7220fbeb-9a3c-468e-8a09-7da0f9d0599f: moving from state preparing -> state finished >Thread-15912178::DEBUG::2013-11-19 22:19:51,392::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912178::DEBUG::2013-11-19 22:19:51,393::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912178::DEBUG::2013-11-19 22:19:51,393::task::495::TaskManager.Task::(_debug) Task 7220fbeb-9a3c-468e-8a09-7da0f9d0599f: ref 0 aborting False >Thread-15912178::INFO::2013-11-19 22:19:51,394::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000255823135376', 'lastCheck': 1384899585.358366, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00151896476746', 'lastCheck': 1384899582.6801791, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384899587.942589, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000344038009644', 'lastCheck': 1384899583.5820351, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-19 22:19:57,955::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 22:20:05,382::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 22:20:06,037::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:20:07,969::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:20:17,983::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:20:27,997::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:20:38,011::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:20:48,025::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:20:58,038::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:21:08,052::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:21:18,066::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:21:28,080::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:21:38,094::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:21:48,107::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:21:58,121::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:22:08,134::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:22:18,148::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:22:28,161::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 22:22:33,778::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 22:22:34,450::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:22:38,179::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:22:48,193::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:22:58,206::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:23:08,220::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:23:18,233::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:23:28,247::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:23:38,260::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:23:48,274::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:23:58,287::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:24:08,301::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:24:18,314::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:24:28,328::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:24:38,341::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:24:48,355::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:24:58,369::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 22:25:06,422::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 22:25:07,142::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:25:08,383::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:25:18,397::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:25:28,410::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:25:38,424::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:25:48,438::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:25:58,452::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:26:08,465::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:26:18,479::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:26:28,493::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:26:38,507::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:26:48,521::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:26:58,534::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:27:08,548::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:27:18,562::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:27:28,575::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 22:27:34,802::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 22:27:35,463::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:27:38,590::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:27:48,604::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:27:58,618::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:28:08,632::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:28:18,645::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:28:28,658::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:28:38,672::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:28:48,685::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:28:58,699::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:29:08,712::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:29:18,726::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:29:28,739::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:29:38,753::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:29:48,766::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:29:58,779::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 22:30:07,524::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 22:30:08,226::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:30:08,792::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:30:18,806::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:30:28,819::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:30:38,832::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:30:48,846::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:30:58,859::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:31:08,872::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:31:18,885::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:31:28,898::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:31:38,911::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:31:48,924::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:31:58,937::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:32:08,950::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:32:18,963::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:32:28,976::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 22:32:35,914::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 22:32:36,569::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:32:39,128::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:32:49,143::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:32:59,157::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:33:09,169::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:33:19,182::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:33:29,195::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:33:39,209::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:33:49,223::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:33:59,239::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:34:09,252::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:34:19,265::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:34:29,279::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:34:39,292::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:34:49,306::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:34:59,320::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 22:35:08,612::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 22:35:09,273::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:35:09,334::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:35:19,348::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:35:29,361::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:35:39,375::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:35:49,389::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:35:59,403::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:36:09,416::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:36:19,430::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:36:29,444::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:36:39,457::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:36:49,471::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:36:59,485::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:37:09,499::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:37:19,512::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:37:29,526::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 22:37:37,002::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 22:37:37,682::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:37:39,540::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:37:49,554::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:37:59,568::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:38:09,582::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:38:19,596::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:38:29,610::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:38:39,624::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:38:49,638::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:38:59,652::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:39:09,666::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:39:19,680::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:39:29,694::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:39:39,708::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:39:49,721::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:39:59,735::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 22:40:09,664::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-1024767::ERROR::2013-11-19 22:40:09,920::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 22:40:10,301::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:40:19,934::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:40:29,948::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:40:39,962::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:40:49,975::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:40:59,989::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:41:10,003::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:41:20,016::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:41:30,030::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:41:40,044::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:41:50,058::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:42:00,072::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:42:10,086::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:42:20,100::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:42:30,114::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 22:42:38,054::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 22:42:38,708::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:42:40,274::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:42:50,289::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:43:00,303::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:43:10,316::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:43:20,330::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:43:30,344::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:43:40,358::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:43:50,371::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:44:00,385::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:44:10,398::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:44:20,412::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:44:30,426::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:44:40,439::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:44:50,453::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:45:00,467::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:45:10,481::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 22:45:10,689::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 22:45:11,333::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:45:20,495::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:45:30,508::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:45:40,522::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:45:50,536::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:46:00,549::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:46:10,563::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:46:20,576::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:46:30,590::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:46:40,603::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:46:50,617::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:47:00,630::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:47:10,644::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:47:20,657::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:47:30,670::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 22:47:39,072::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 22:47:39,713::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:47:40,684::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:47:50,697::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:48:00,710::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:48:10,724::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:48:20,737::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:48:30,751::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:48:40,764::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:48:50,778::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:49:00,791::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:49:10,805::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:49:20,818::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:49:30,832::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:49:40,846::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:49:50,859::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:50:00,873::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:50:10,886::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 22:50:11,729::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 22:50:12,331::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:50:20,957::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:50:30,970::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:50:40,984::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:50:50,997::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:51:01,011::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:51:11,024::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:51:21,038::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:51:31,051::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:51:41,064::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:51:51,078::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:52:01,092::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:52:11,105::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:52:21,119::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:52:31,132::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 22:52:40,130::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 22:52:40,795::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:52:41,359::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:52:51,373::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:53:01,386::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:53:11,399::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:53:21,411::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:53:31,424::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:53:41,439::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:53:51,452::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:54:01,466::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:54:11,479::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:54:21,493::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:54:31,507::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:54:41,520::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:54:51,534::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:55:01,548::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:55:11,561::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 22:55:12,716::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 22:55:13,387::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:55:21,575::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:55:31,589::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:55:41,603::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:55:51,617::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:56:01,631::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:56:11,645::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:56:21,658::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:56:31,672::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:56:41,686::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:56:51,700::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:57:01,714::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:57:11,727::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:57:21,741::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:57:31,755::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 22:57:41,139::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-1024767::ERROR::2013-11-19 22:57:41,770::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 22:57:41,861::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 22:57:51,785::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:58:01,799::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:58:11,813::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:58:21,826::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:58:31,841::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:58:41,855::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:58:51,869::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:59:01,883::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:59:11,897::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:59:21,911::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:59:31,925::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:59:41,939::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 22:59:51,953::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:00:01,967::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:00:11,980::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 23:00:13,772::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 23:00:14,454::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:00:22,002::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:00:32,020::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:00:42,034::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:00:52,047::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:01:02,060::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:01:12,074::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:01:22,087::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:01:32,100::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:01:42,114::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:01:52,127::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:02:02,141::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:02:12,155::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:02:22,168::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:02:32,181::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:02:42,194::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 23:02:42,207::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 23:02:42,839::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:02:52,208::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:03:02,222::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:03:12,235::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:03:22,248::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:03:32,262::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:03:42,275::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:03:52,289::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:04:02,302::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:04:12,316::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:04:22,329::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:04:32,343::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:04:42,356::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:04:52,369::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:05:02,383::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:05:12,397::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 23:05:14,842::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 23:05:15,453::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:05:22,410::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:05:32,425::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:05:42,439::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:05:52,452::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:06:02,466::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:06:12,480::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:06:22,493::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:06:32,507::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:06:42,520::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:06:52,534::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:07:02,547::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:07:12,561::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:07:22,574::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:07:32,588::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:07:42,601::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 23:07:43,234::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 23:07:43,956::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:07:52,615::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:08:02,629::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:08:12,643::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:08:22,656::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:08:32,670::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:08:42,683::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:08:52,697::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:09:02,710::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:09:12,724::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:09:22,738::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:09:32,751::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:09:42,765::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:09:52,779::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:10:02,792::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:10:12,805::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 23:10:15,826::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 23:10:16,472::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:10:22,819::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:10:32,833::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:10:42,847::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:10:52,860::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:11:02,873::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:11:12,887::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:11:22,900::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:11:32,913::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:11:42,927::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:11:52,940::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:12:02,954::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:12:12,967::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:12:22,981::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:12:32,995::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:12:43,009::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 23:12:44,302::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 23:12:45,025::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:12:53,023::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:13:03,037::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:13:13,054::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:13:23,068::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:13:33,083::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:13:43,096::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:13:53,110::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:14:03,124::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:14:13,137::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:14:23,150::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:14:33,163::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:14:43,176::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:14:53,189::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:15:03,203::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:15:13,216::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 23:15:16,862::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 23:15:17,569::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:15:23,229::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:15:33,243::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:15:43,257::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:15:53,270::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:16:03,283::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:16:13,297::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:16:23,310::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:16:33,323::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:16:43,336::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:16:53,350::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:17:03,363::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:17:13,376::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:17:23,389::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:17:33,403::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:17:43,417::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 23:17:45,372::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 23:17:46,047::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:17:53,606::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:18:03,619::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:18:13,633::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:18:23,646::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:18:33,660::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:18:43,673::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:18:53,687::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:19:03,700::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:19:13,714::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:19:23,727::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:19:33,741::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:19:43,755::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:19:53,768::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:20:03,782::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:20:13,796::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 23:20:17,953::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 23:20:18,562::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:20:23,810::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:20:33,824::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:20:43,838::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:20:53,852::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:21:03,865::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:21:13,879::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:21:23,892::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:21:33,906::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:21:43,918::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:21:53,931::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:22:03,945::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:22:13,958::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:22:23,971::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:22:33,984::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:22:43,998::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 23:22:46,390::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 23:22:47,048::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:22:54,012::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:23:04,025::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:23:14,039::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:23:24,052::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:23:34,066::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:23:44,079::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:23:54,093::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:24:04,108::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:24:14,119::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:24:24,133::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:24:34,145::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:24:44,159::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:24:54,172::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:25:04,185::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:25:14,199::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 23:25:18,949::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 23:25:19,592::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:25:24,212::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:25:34,226::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:25:44,239::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:25:54,253::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:26:04,266::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:26:14,280::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:26:24,293::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:26:34,307::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:26:44,320::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:26:54,333::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:27:04,346::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:27:14,359::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:27:24,372::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:27:34,386::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:27:44,398::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 23:27:47,390::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 23:27:48,050::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:27:54,618::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:28:04,632::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:28:14,650::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:28:24,666::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:28:34,680::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:28:44,693::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:28:54,707::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:29:04,720::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:29:14,734::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:29:24,747::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:29:34,761::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:29:44,775::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:29:54,788::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:30:04,802::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:30:14,816::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 23:30:19,986::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 23:30:20,671::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:30:24,830::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:30:34,844::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:30:44,858::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:30:54,872::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:31:04,886::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:31:14,899::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:31:24,913::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:31:34,927::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:31:44,940::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:31:54,954::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:32:04,968::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:32:14,982::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:32:24,996::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:32:35,010::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:32:45,024::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 23:32:48,399::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 23:32:49,051::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:32:55,038::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:33:05,052::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:33:15,066::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:33:25,080::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:33:35,094::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:33:45,108::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:33:55,123::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:34:05,136::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:34:15,150::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:34:25,163::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:34:35,176::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:34:45,189::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:34:55,202::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:35:05,215::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:35:15,228::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 23:35:21,054::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 23:35:21,695::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:35:25,259::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:35:35,273::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:35:45,286::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:35:55,299::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:36:05,312::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:36:15,325::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:36:25,338::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:36:35,352::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:36:45,365::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:36:55,378::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:37:05,391::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:37:15,404::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:37:25,417::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:37:35,431::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:37:45,445::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 23:37:49,427::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 23:37:50,137::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:37:55,634::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:38:05,648::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:38:15,661::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:38:25,674::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:38:35,687::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:38:45,700::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:38:55,713::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:39:05,726::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:39:15,740::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:39:25,753::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:39:35,766::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:39:45,780::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:39:55,793::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:40:05,806::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:40:15,819::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 23:40:22,081::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 23:40:22,732::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:40:25,833::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:40:35,847::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:40:45,860::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:40:55,873::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:41:05,886::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:41:15,899::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:41:25,912::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:41:35,926::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:41:45,939::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:41:55,953::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:42:05,966::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:42:15,979::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:42:25,993::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:42:36,006::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:42:46,019::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 23:42:50,494::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 23:42:51,203::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:42:56,033::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:43:06,047::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:43:16,060::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:43:26,073::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:43:36,087::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:43:46,100::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:43:56,114::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:44:06,127::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:44:16,141::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:44:26,154::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:44:36,167::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:44:46,180::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:44:56,194::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:45:06,207::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:45:16,220::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 23:45:23,116::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 23:45:23,811::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:45:26,340::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:45:36,354::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:45:46,369::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:45:56,384::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:46:06,397::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:46:16,410::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:46:26,424::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:46:36,437::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:46:46,451::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:46:56,464::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:47:06,477::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:47:16,491::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:47:26,504::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:47:36,518::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:47:46,531::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 23:47:51,548::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 23:47:52,184::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:47:56,545::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:48:06,559::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:48:16,573::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:48:26,586::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:48:36,600::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:48:46,614::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:48:56,627::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:49:06,641::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:49:16,654::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:49:26,668::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:49:36,681::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:49:46,695::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:49:56,709::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:50:06,723::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:50:16,736::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 23:50:24,197::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 23:50:24,844::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:50:26,751::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:50:36,765::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:50:46,778::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:50:56,793::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:51:06,806::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:51:16,820::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:51:26,833::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:51:36,846::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:51:46,860::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:51:56,874::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:52:06,887::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:52:16,901::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:52:26,914::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:52:36,928::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:52:46,942::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 23:52:52,591::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 23:52:53,330::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:52:56,956::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:53:06,970::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:53:16,984::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:53:26,997::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:53:37,011::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:53:47,024::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:53:57,038::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:54:07,052::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:54:17,066::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:54:27,080::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:54:37,093::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:54:47,107::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:54:57,121::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:55:07,134::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:55:17,147::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-19 23:55:25,222::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-19 23:55:25,954::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:55:27,161::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:55:37,175::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:55:47,189::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:55:57,202::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:56:07,216::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:56:17,229::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:56:27,242::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:56:37,256::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:56:47,270::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:56:57,283::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:57:07,296::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:57:17,309::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:57:27,322::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:57:37,336::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:57:47,349::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-19 23:57:53,675::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-19 23:57:54,415::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-19 23:57:57,363::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:58:07,377::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:58:17,390::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:58:27,404::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:58:37,416::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:58:47,431::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:58:57,445::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:59:07,459::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:59:17,472::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:59:27,485::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:59:37,498::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:59:47,511::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-19 23:59:57,524::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:00:07,538::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:00:17,551::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 00:00:26,340::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 00:00:27,075::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:00:27,579::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:00:37,591::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:00:47,605::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:00:57,619::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:01:07,632::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:01:17,646::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:01:27,659::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:01:37,673::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:01:47,686::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:01:57,700::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:02:07,713::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:02:17,726::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:02:27,740::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:02:37,753::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:02:47,766::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 00:02:54,761::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 00:02:55,412::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:02:57,979::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:03:07,993::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:03:18,007::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:03:28,021::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:03:38,035::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:03:48,048::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:03:58,062::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:04:08,076::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:04:18,089::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:04:28,103::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:04:38,117::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:04:48,130::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:04:58,144::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:05:08,158::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:05:18,171::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 00:05:27,459::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-1024767::ERROR::2013-11-20 00:05:28,187::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 00:05:28,193::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:05:38,201::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:05:48,215::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:05:58,228::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:06:08,242::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:06:18,256::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:06:28,269::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:06:38,283::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:06:48,298::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:06:58,313::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:07:08,327::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:07:18,341::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:07:28,355::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:07:38,369::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:07:48,382::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 00:07:55,848::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 00:07:56,488::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:07:58,396::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:08:08,411::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:08:18,424::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:08:28,438::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:08:38,452::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:08:48,465::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:08:58,478::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:09:08,490::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:09:18,502::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:09:28,515::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:09:38,527::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:09:48,540::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:09:58,553::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:10:08,566::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:10:18,579::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 00:10:28,583::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-1024767::ERROR::2013-11-20 00:10:28,805::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 00:10:29,236::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:10:38,819::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:10:48,832::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:10:58,846::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:11:08,858::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:11:18,872::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:11:28,885::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:11:38,898::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:11:48,911::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:11:58,924::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:12:08,938::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:12:18,951::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:12:28,964::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:12:38,977::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:12:48,991::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 00:12:56,971::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 00:12:57,695::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:12:59,198::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:13:09,213::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:13:19,226::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:13:29,240::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:13:39,253::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:13:49,267::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:13:59,280::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:14:09,294::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:14:19,307::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:14:29,321::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:14:39,334::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:14:49,348::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:14:59,361::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:15:09,375::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:15:19,388::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:15:29,402::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 00:15:29,617::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 00:15:30,285::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:15:39,415::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:15:49,429::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:15:59,442::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:16:09,456::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:16:19,469::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:16:29,482::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:16:39,496::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:16:49,509::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:16:59,522::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:17:09,536::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:17:19,549::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:17:29,563::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:17:39,576::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:17:49,590::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 00:17:58,040::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 00:17:58,773::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:17:59,604::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:18:09,618::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:18:19,631::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:18:29,644::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:18:39,658::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:18:49,671::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:18:59,685::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:19:09,698::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:19:19,712::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:19:29,725::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:19:39,738::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:19:49,752::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:19:59,765::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:20:09,778::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:20:19,792::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:20:29,805::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 00:20:30,671::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 00:20:31,307::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:20:39,878::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:20:49,891::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:20:59,904::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:21:09,918::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:21:19,931::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:21:29,945::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:21:39,958::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:21:49,971::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:21:59,985::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:22:09,998::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:22:20,012::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:22:30,025::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:22:40,039::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:22:50,053::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 00:22:59,113::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 00:22:59,822::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:23:00,067::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:23:10,081::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:23:20,095::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:23:30,108::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:23:40,122::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:23:50,136::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:24:00,149::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:24:10,162::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:24:20,175::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:24:30,189::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:24:40,202::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:24:50,216::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:25:00,229::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:25:10,242::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:25:20,256::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:25:30,272::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 00:25:31,688::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 00:25:32,406::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:25:40,286::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:25:50,300::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:26:00,313::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:26:10,327::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:26:20,340::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:26:30,354::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:26:40,367::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:26:50,380::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:27:00,394::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:27:10,407::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:27:20,420::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:27:30,433::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:27:40,447::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:27:50,459::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 00:28:00,177::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-1024767::ERROR::2013-11-20 00:28:00,475::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 00:28:00,899::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:28:10,489::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:28:20,503::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:28:30,516::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:28:40,530::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:28:50,543::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:29:00,557::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:29:10,570::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:29:20,584::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:29:30,597::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:29:40,610::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:29:50,624::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:30:00,637::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:30:10,650::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:30:20,663::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:30:30,677::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 00:30:32,785::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 00:30:33,511::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:30:40,690::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:30:50,704::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:31:00,717::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:31:10,731::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:31:20,744::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:31:30,757::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:31:40,771::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:31:50,783::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:32:00,797::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:32:10,810::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:32:20,823::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:32:30,837::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:32:40,850::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:32:50,863::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:33:00,877::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 00:33:01,245::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 00:33:01,957::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:33:10,890::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:33:20,904::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:33:30,917::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:33:40,931::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:33:50,945::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:34:00,958::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:34:10,972::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:34:20,985::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:34:30,999::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:34:41,012::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:34:51,026::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:35:01,039::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:35:11,053::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:35:21,066::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:35:31,080::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 00:35:33,890::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 00:35:34,545::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:35:41,107::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:35:51,121::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:36:01,135::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:36:11,147::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:36:21,161::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:36:31,173::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:36:41,186::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:36:51,200::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:37:01,213::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:37:11,226::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:37:21,240::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:37:31,253::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:37:41,266::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:37:51,280::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:38:01,293::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 00:38:02,302::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 00:38:02,959::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:38:11,307::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:38:21,321::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:38:31,334::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:38:41,348::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:38:51,361::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:39:01,375::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:39:11,388::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:39:21,402::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:39:31,415::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:39:41,429::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:39:51,443::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:40:01,457::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:40:11,470::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:40:21,483::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:40:31,496::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 00:40:34,930::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 00:40:35,666::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:40:41,510::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:40:51,523::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:41:01,537::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:41:11,550::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:41:21,563::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:41:31,577::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:41:41,590::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:41:51,603::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:42:01,617::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:42:11,630::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:42:21,644::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:42:31,656::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:42:41,668::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:42:51,681::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:43:01,694::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 00:43:03,315::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 00:43:04,052::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:43:11,708::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:43:21,721::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:43:31,735::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:43:41,748::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:43:51,761::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:44:01,775::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:44:11,788::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:44:21,802::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:44:31,815::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:44:41,829::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:44:51,843::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:45:01,856::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:45:11,869::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:45:21,883::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:45:31,896::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 00:45:36,050::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 00:45:36,779::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:45:41,910::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:45:51,923::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:46:01,937::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:46:11,950::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:46:21,964::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:46:31,977::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:46:41,990::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:46:52,004::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:47:02,017::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:47:12,030::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:47:22,044::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:47:32,057::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:47:42,070::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:47:52,083::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:48:02,097::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 00:48:04,468::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 00:48:05,158::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:48:12,110::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:48:22,124::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:48:32,137::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:48:42,150::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:48:52,164::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:49:02,177::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:49:12,190::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:49:22,203::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:49:32,216::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:49:42,229::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:49:52,242::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:50:02,256::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:50:12,270::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:50:22,283::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:50:32,297::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 00:50:37,165::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 00:50:37,793::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:50:42,370::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:50:52,384::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:51:02,397::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:51:12,411::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:51:22,424::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:51:32,438::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:51:42,451::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:51:52,465::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:52:02,479::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:52:12,492::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:52:22,505::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:52:32,519::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:52:42,532::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:52:52,546::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:53:02,560::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 00:53:05,536::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 00:53:06,211::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:53:12,745::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:53:22,758::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:53:32,771::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:53:42,785::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:53:52,799::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:54:02,813::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:54:12,826::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:54:22,840::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:54:32,854::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:54:42,867::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:54:52,881::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:55:02,894::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:55:12,908::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:55:22,922::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:55:32,935::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 00:55:38,183::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 00:55:38,826::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:55:42,949::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:55:52,962::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:56:02,976::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:56:12,989::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:56:23,002::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:56:33,016::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:56:43,029::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:56:53,042::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:57:03,056::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:57:13,069::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:57:23,082::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:57:33,096::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:57:43,109::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:57:53,122::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:58:03,136::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 00:58:06,570::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 00:58:07,256::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 00:58:13,150::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:58:23,163::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:58:33,175::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:58:43,188::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:58:53,201::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:59:03,214::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:59:13,227::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:59:23,240::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:59:33,253::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:59:43,266::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 00:59:53,280::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:00:03,293::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:00:13,306::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:00:23,320::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:00:33,333::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 01:00:39,215::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 01:00:39,935::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:00:43,440::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:00:53,453::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:01:03,470::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:01:13,486::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:01:23,498::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:01:33,511::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:01:43,524::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:01:53,537::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:02:03,551::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:02:13,564::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:02:23,577::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:02:33,590::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:02:43,604::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:02:53,618::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:03:03,631::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 01:03:07,632::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 01:03:08,369::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:03:13,858::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:03:23,872::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:03:33,885::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:03:43,898::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:03:53,912::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:04:03,925::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:04:13,938::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:04:23,952::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:04:33,965::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:04:43,979::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:04:53,993::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:05:04,006::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:05:14,019::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:05:24,033::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:05:34,046::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 01:05:40,318::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 01:05:40,965::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:05:44,060::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:05:54,073::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:06:04,087::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:06:14,100::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:06:24,114::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:06:34,127::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:06:44,141::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:06:54,155::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:07:04,168::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:07:14,181::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:07:24,195::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:07:34,208::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:07:44,221::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:07:54,235::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:08:04,248::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 01:08:08,720::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 01:08:09,377::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:08:14,262::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:08:24,275::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:08:34,289::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:08:44,302::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:08:54,316::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:09:04,329::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:09:14,343::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:09:24,356::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:09:34,369::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:09:44,383::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:09:54,396::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:10:04,410::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:10:14,424::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:10:24,437::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:10:34,451::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 01:10:41,348::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 01:10:41,979::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:10:44,565::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:10:54,579::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:11:04,593::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:11:14,606::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:11:24,620::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:11:34,633::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:11:44,647::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:11:54,661::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:12:04,674::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:12:14,688::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:12:24,702::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:12:34,716::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:12:44,729::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:12:54,743::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:13:04,756::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 01:13:09,729::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 01:13:10,435::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:13:14,955::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:13:24,972::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:13:34,987::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:13:45,001::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:13:55,014::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:14:05,028::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:14:15,041::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:14:25,055::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:14:35,068::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:14:45,081::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:14:55,095::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:15:05,109::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:15:15,122::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:15:25,135::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:15:35,148::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 01:15:42,363::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 01:15:43,077::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:15:45,162::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:15:55,176::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:16:05,189::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:16:15,202::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:16:25,215::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:16:35,228::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:16:45,242::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:16:55,255::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:17:05,282::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:17:15,295::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:17:25,309::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:17:35,322::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:17:45,335::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:17:55,349::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:18:05,362::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 01:18:10,783::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 01:18:11,509::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:18:15,376::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:18:25,390::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:18:35,403::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:18:45,416::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:18:55,430::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:19:05,444::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:19:15,457::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:19:25,470::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:19:35,484::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:19:45,497::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:19:55,510::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:20:05,524::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:20:15,537::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:20:25,550::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:20:35,564::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 01:20:43,460::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 01:20:44,155::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:20:45,682::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:20:55,696::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:21:05,709::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:21:15,722::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:21:25,736::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:21:35,750::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:21:45,763::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:21:55,776::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:22:05,790::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:22:15,803::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:22:25,816::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:22:35,830::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:22:45,844::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:22:55,857::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:23:05,871::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 01:23:11,853::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 01:23:12,484::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:23:16,070::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:23:26,084::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:23:36,097::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:23:46,111::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:23:56,124::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:24:06,138::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:24:16,152::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:24:26,165::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:24:36,178::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:24:46,191::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:24:56,204::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:25:06,218::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:25:16,231::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:25:26,244::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:25:36,258::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 01:25:44,537::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 01:25:45,262::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:25:46,271::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:25:56,284::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:26:06,297::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:26:16,310::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:26:26,323::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:26:36,336::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:26:46,349::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:26:56,362::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:27:06,375::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:27:16,388::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:27:26,401::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:27:36,414::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:27:46,427::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:27:56,440::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:28:06,453::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 01:28:12,930::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 01:28:13,660::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:28:16,467::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:28:26,480::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:28:36,493::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:28:46,507::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:28:56,520::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:29:06,533::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:29:16,546::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:29:26,559::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:29:36,573::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:29:46,586::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:29:56,599::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:30:06,612::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:30:16,625::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:30:26,639::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:30:36,652::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 01:30:45,644::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 01:30:46,368::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:30:46,855::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:30:56,867::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:31:06,883::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:31:16,897::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:31:26,910::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:31:36,923::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:31:46,937::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:31:56,950::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:32:06,964::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:32:16,977::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:32:26,991::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:32:37,004::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:32:47,018::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:32:57,031::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:33:07,044::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 01:33:14,030::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 01:33:14,754::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:33:17,256::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:33:27,270::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:33:37,283::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:33:47,295::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:33:57,308::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:34:07,322::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:34:17,335::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:34:27,349::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:34:37,362::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:34:47,376::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:34:57,389::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:35:07,403::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:35:17,416::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:35:27,430::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:35:37,443::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 01:35:46,753::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 01:35:47,398::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:35:47,458::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:35:57,471::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:36:07,485::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:36:17,499::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:36:27,512::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:36:37,526::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:36:47,540::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:36:57,553::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:37:07,567::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:37:17,580::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:37:27,594::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:37:37,608::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:37:47,621::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:37:57,635::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:38:07,649::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 01:38:15,127::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 01:38:15,767::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:38:17,663::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:38:27,677::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:38:37,691::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:38:47,705::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:38:57,718::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:39:07,732::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:39:17,745::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:39:27,759::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:39:37,773::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:39:47,787::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:39:57,800::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:40:07,814::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:40:17,828::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:40:27,842::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:40:37,855::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 01:40:47,783::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-1024767::ERROR::2013-11-20 01:40:48,009::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 01:40:48,478::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:40:58,023::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:41:08,037::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:41:18,050::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:41:28,064::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:41:38,078::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:41:48,092::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:41:58,105::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:42:08,119::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:42:18,133::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:42:28,146::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:42:38,160::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:42:48,174::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:42:58,187::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:43:08,201::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 01:43:16,177::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 01:43:16,920::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:43:18,405::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:43:28,419::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:43:38,434::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:43:48,447::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:43:58,461::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:44:08,474::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:44:18,488::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:44:28,502::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:44:38,515::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:44:48,529::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:44:58,543::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:45:08,556::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:45:18,570::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:45:28,583::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:45:38,597::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:45:48,610::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 01:45:48,863::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 01:45:49,611::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:45:58,624::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:46:08,638::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:46:18,652::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:46:28,665::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:46:38,678::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:46:48,692::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:46:58,705::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:47:08,718::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:47:18,731::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:47:28,745::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:47:38,758::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:47:48,771::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:47:58,785::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:48:08,798::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 01:48:17,271::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 01:48:17,924::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:48:18,812::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:48:28,826::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:48:38,839::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:48:48,853::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:48:58,867::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:49:08,880::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:49:18,894::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:49:28,908::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:49:38,921::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:49:48,935::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:49:58,949::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:50:08,962::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:50:18,976::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:50:28,990::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:50:39,004::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:50:49,017::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 01:50:49,994::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 01:50:50,599::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:50:59,202::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:51:09,215::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:51:19,229::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:51:29,242::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:51:39,255::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:51:49,268::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:51:59,282::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:52:09,295::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:52:19,308::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:52:29,321::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:52:39,334::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:52:49,348::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:52:59,361::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:53:09,374::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 01:53:18,365::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 01:53:19,061::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:53:19,572::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:53:29,586::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:53:39,599::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:53:49,613::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:53:59,627::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:54:09,641::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:54:19,654::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:54:29,667::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:54:39,681::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:54:49,695::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:54:59,708::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:55:09,722::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:55:19,735::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:55:29,749::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:55:39,763::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:55:49,776::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 01:55:50,984::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 01:55:51,703::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:55:59,790::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:56:09,803::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:56:19,816::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:56:29,829::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:56:39,842::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:56:49,856::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:56:59,869::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:57:09,882::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:57:19,896::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:57:29,908::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:57:39,922::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:57:49,936::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:57:59,949::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:58:09,963::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 01:58:19,416::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-1024767::ERROR::2013-11-20 01:58:19,980::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 01:58:20,132::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 01:58:29,994::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:58:40,007::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:58:50,020::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:59:00,034::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:59:10,048::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:59:20,061::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:59:30,075::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:59:40,089::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 01:59:50,103::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:00:00,116::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:00:10,130::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:00:20,144::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:00:30,158::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:00:40,171::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:00:50,183::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 02:00:52,087::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 02:00:52,812::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 02:01:00,329::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:01:10,343::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:01:20,356::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:01:30,369::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:01:40,383::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:01:50,396::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:02:00,409::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:02:10,423::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:02:20,436::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:02:30,449::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:02:40,463::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:02:50,476::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:03:00,489::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:03:10,502::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 02:03:20,485::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-1024767::ERROR::2013-11-20 02:03:20,738::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 02:03:21,130::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 02:03:30,752::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:03:40,766::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:03:50,779::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:04:00,792::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:04:10,805::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:04:20,818::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:04:30,831::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:04:40,844::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:04:50,857::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:05:00,870::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:05:10,883::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:05:20,896::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:05:30,909::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:05:40,923::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:05:50,936::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 02:05:53,201::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 02:05:53,841::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 02:06:00,949::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:06:10,963::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:06:20,976::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:06:30,989::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:06:41,003::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:06:51,017::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:07:01,030::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:07:11,043::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:07:21,057::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:07:31,070::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:07:41,084::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:07:51,097::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:08:01,111::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:08:11,124::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:08:21,138::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 02:08:21,579::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 02:08:22,223::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 02:08:31,152::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:08:41,165::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:08:51,178::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:09:01,192::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:09:11,205::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:09:21,218::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:09:31,231::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:09:41,245::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:09:51,258::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:10:01,271::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:10:11,284::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:10:21,297::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:10:31,311::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:10:41,324::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:10:51,337::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 02:10:54,225::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 02:10:54,926::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 02:11:01,434::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:11:11,447::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:11:21,461::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:11:31,474::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:11:41,488::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:11:51,501::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:12:01,515::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:12:11,528::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:12:21,542::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:12:31,555::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:12:41,569::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:12:51,582::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:13:01,595::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:13:11,609::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:13:21,623::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 02:13:22,599::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 02:13:23,241::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 02:13:31,847::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:13:41,861::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:13:51,875::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:14:01,888::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:14:11,902::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:14:21,916::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:14:31,929::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:14:41,943::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:14:51,956::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:15:01,970::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:15:11,984::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:15:21,998::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:15:32,011::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:15:42,025::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:15:52,039::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 02:15:55,312::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 02:15:55,972::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 02:16:02,052::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:16:12,066::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:16:22,080::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:16:32,094::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:16:42,108::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:16:52,122::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:17:02,135::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:17:12,148::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:17:22,162::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:17:32,175::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:17:42,188::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:17:52,201::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:18:02,215::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:18:12,228::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:18:22,241::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 02:18:23,707::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 02:18:24,430::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 02:18:32,255::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:18:42,268::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:18:52,282::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:19:02,295::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:19:12,309::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:19:22,322::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:19:32,335::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:19:42,349::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:19:52,362::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:20:02,376::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:20:12,390::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:20:22,403::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:20:32,417::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:20:42,430::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:20:52,444::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42838::DEBUG::2013-11-20 02:20:56,353::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 02:20:56,969::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 02:21:02,558::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:21:12,572::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:21:22,588::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:21:32,604::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:21:42,617::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:21:52,631::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:22:02,645::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:22:12,659::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:22:22,672::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:22:32,686::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:22:42,700::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:22:52,713::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:23:02,727::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:23:12,741::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-1024767::ERROR::2013-11-20 02:23:22,754::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-42840::DEBUG::2013-11-20 02:23:24,787::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 02:23:25,516::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912180::INFO::2013-11-20 02:23:30,364::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912180::DEBUG::2013-11-20 02:23:30,365::task::495::TaskManager.Task::(_debug) Task 2b684da7-ca08-42df-a66c-91527cfb2c77: moving from state init -> state preparing >Thread-15912180::DEBUG::2013-11-20 02:23:30,374::task::495::TaskManager.Task::(_debug) Task 2b684da7-ca08-42df-a66c-91527cfb2c77: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000353097915649', 'lastCheck': 1384914207.1672649, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137400627136', 'lastCheck': 1384914201.457849, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384914202.7555621, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000349998474121', 'lastCheck': 1384914205.5183051, 'valid': True, 'code': 0}} >Thread-15912180::DEBUG::2013-11-20 02:23:30,380::task::495::TaskManager.Task::(_debug) Task 2b684da7-ca08-42df-a66c-91527cfb2c77: moving from state preparing -> state finished >Thread-15912180::DEBUG::2013-11-20 02:23:30,381::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912180::DEBUG::2013-11-20 02:23:30,381::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912180::DEBUG::2013-11-20 02:23:30,382::task::495::TaskManager.Task::(_debug) Task 2b684da7-ca08-42df-a66c-91527cfb2c77: ref 0 aborting False >Thread-15912180::INFO::2013-11-20 02:23:30,388::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000353097915649', 'lastCheck': 1384914207.1672649, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137400627136', 'lastCheck': 1384914201.457849, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384914202.7555621, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000349998474121', 'lastCheck': 1384914205.5183051, 'valid': True, 'code': 0}} >Thread-15912181::DEBUG::2013-11-20 02:23:30,623::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVdsCapabilities with () {} >Thread-15912181::DEBUG::2013-11-20 02:23:31,078::utils::579::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-img-rhev' (cwd None) >Thread-15912181::DEBUG::2013-11-20 02:23:31,507::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912181::DEBUG::2013-11-20 02:23:31,509::utils::579::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-kvm-rhev' (cwd None) >Thread-15912181::DEBUG::2013-11-20 02:23:31,932::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912181::DEBUG::2013-11-20 02:23:31,934::utils::579::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" vdsm' (cwd None) >Thread-15912181::DEBUG::2013-11-20 02:23:32,367::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912181::DEBUG::2013-11-20 02:23:32,369::utils::579::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" spice-server' (cwd None) >Thread-15912181::DEBUG::2013-11-20 02:23:32,781::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912181::DEBUG::2013-11-20 02:23:32,783::utils::579::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" libvirt' (cwd None) >Thread-1024767::ERROR::2013-11-20 02:23:33,049::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912181::DEBUG::2013-11-20 02:23:33,232::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912181::DEBUG::2013-11-20 02:23:33,235::clientIF::59::vds::(wrapper) return getVdsCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:c936ebb147a8'}], 'FC': []}, 'packages2': {'kernel': {'release': '279.11.1.el6.x86_64', 'buildtime': 1348297826.0, 'version': '2.6.32'}, 'spice-server': {'release': '10.el6', 'buildtime': '1337611492', 'version': '0.10.1'}, 'vdsm': {'release': '113.4.el6_3', 'buildtime': '1347534072', 'version': '4.9'}, 'qemu-kvm': {'release': '2.295.el6_3.2', 'buildtime': '1346673796', 'version': '0.12.1.2'}, 'libvirt': {'release': '21.el6_3.5', 'buildtime': '1348687577', 'version': '0.9.10'}, 'qemu-img': {'release': '2.295.el6_3.2', 'buildtime': '1346673796', 'version': '0.12.1.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz', 'hooks': {'before_vm_start': {'50_vhostmd': {'md5': '1635e45584995d9ad2629939cb0bcdbf'}}, 'after_vm_destroy': {'50_vhostmd': {'md5': '7e7509430304099f38e4a9de966af090'}}, 'before_vm_dehibernate': {'50_vhostmd': {'md5': '1635e45584995d9ad2629939cb0bcdbf'}}, 'before_vm_migrate_destination': {'50_vhostmd': {'md5': '1635e45584995d9ad2629939cb0bcdbf'}}}, 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'addr': '192.168.100.133', 'cfg': {'IPADDR': '192.168.100.133', 'GATEWAY': '192.168.100.1', 'DELAY': '0', 'NETMASK': '255.255.128.0', 'STP': 'no', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ports': ['bond0'], 'netmask': '255.255.128.0', 'stp': 'off', 'gateway': '192.168.100.1'}, 'DMZ': {'addr': '172.16.1.133', 'cfg': {'IPADDR': '172.16.1.133', 'DELAY': '0', 'NETMASK': '255.255.255.0', 'STP': 'no', 'DEVICE': 'DMZ', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ports': ['eth0'], 'netmask': '255.255.255.0', 'stp': 'off', 'gateway': '0.0.0.0'}}, 'uuid': '30343536-3138-5A43-4A32-333630373442_AC:16:2D:76:EB:F0', 'lastClientIface': 'rhevm', 'nics': {'eth3': {'hwaddr': 'AC:16:2D:76:EB:F2', 'netmask': '', 'speed': 1000, 'addr': '', 'permhwaddr': 'AC:16:2D:76:EB:F3'}, 'eth2': {'hwaddr': 'AC:16:2D:76:EB:F2', 'netmask': '', 'speed': 1000, 'addr': '', 'permhwaddr': 'AC:16:2D:76:EB:F2'}, 'eth1': {'hwaddr': 'AC:16:2D:76:EB:F1', 'netmask': '', 'speed': 0, 'addr': ''}, 'eth0': {'hwaddr': 'AC:16:2D:76:EB:F0', 'netmask': '', 'speed': 1000, 'addr': ''}}, 'software_revision': '113.4', 'management_ip': '192.168.100.133', 'clusterLevels': ['3.0'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,dca,sse4_1,sse4_2,x2apic,popcnt,aes,xsave,avx,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_486,model_pentium,model_pentium2,model_pentium3,model_pentiumpro,model_qemu32,model_cpu64-rhel5,model_cpu64-rhel6,model_kvm32,model_coreduo,model_kvm64,model_core2duo,model_n270,model_Conroe,model_Penryn,model_Nehalem,model_Westmere,model_SandyBridge,model_Opteron_G1', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:c936ebb147a8', 'memSize': '64392', 'reservedMem': '256', 'bondings': {'bond4': {'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'netmask': '', 'addr': '', 'slaves': []}, 'bond0': {'hwaddr': 'AC:16:2D:76:EB:F2', 'cfg': {'DEVICE': 'bond0', 'BRIDGE': 'rhevm', 'BOOTPROTO': 'none', 'ONBOOT': 'yes', 'BONDING_OPTS': 'mode=1 miimon=100'}, 'netmask': '', 'addr': '', 'slaves': ['eth2', 'eth3']}, 'bond1': {'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'netmask': '', 'addr': '', 'slaves': []}, 'bond2': {'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'netmask': '', 'addr': '', 'slaves': []}, 'bond3': {'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'netmask': '', 'addr': '', 'slaves': []}}, 'software_version': '4.9', 'cpuSpeed': '1995.126', 'cpuSockets': '2', 'vlans': {}, 'cpuCores': '12', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'version_name': 'Snow Man', 'emulatedMachines': [u'rhel6.3.0', u'pc', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '20121012.0.el6_3', 'version': '6.3', 'name': 'RHEV Hypervisor'}, 'lastClient': '192.168.100.135'}} >Thread-15912184::DEBUG::2013-11-20 02:23:37,366::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912184::INFO::2013-11-20 02:23:37,368::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912184::DEBUG::2013-11-20 02:23:37,369::task::495::TaskManager.Task::(_debug) Task 7c3c0c07-af15-47ed-92cb-859532c0d677: moving from state init -> state preparing >Thread-15912184::DEBUG::2013-11-20 02:23:37,369::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=Free spmLver=285 spmId=-1 >Thread-15912184::DEBUG::2013-11-20 02:23:37,370::task::495::TaskManager.Task::(_debug) Task 7c3c0c07-af15-47ed-92cb-859532c0d677: finished: {'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 285}} >Thread-15912184::DEBUG::2013-11-20 02:23:37,371::task::495::TaskManager.Task::(_debug) Task 7c3c0c07-af15-47ed-92cb-859532c0d677: moving from state preparing -> state finished >Thread-15912184::DEBUG::2013-11-20 02:23:37,371::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912184::DEBUG::2013-11-20 02:23:37,372::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912184::DEBUG::2013-11-20 02:23:37,372::task::495::TaskManager.Task::(_debug) Task 7c3c0c07-af15-47ed-92cb-859532c0d677: ref 0 aborting False >Thread-15912184::INFO::2013-11-20 02:23:37,373::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': -1, 'spmStatus': 'Free', 'spmLver': 285}} >Thread-15912185::DEBUG::2013-11-20 02:23:37,424::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912185::INFO::2013-11-20 02:23:37,425::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: spmStart, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 prevID=-1 prevLVER=285 recoveryMode=0 scsiFencing=false maxHostID=250 domVersion=2) >Thread-15912185::DEBUG::2013-11-20 02:23:37,425::task::495::TaskManager.Task::(_debug) Task 378b55ad-db2a-4978-8944-3175e4685c11: moving from state init -> state preparing >Thread-15912185::DEBUG::2013-11-20 02:23:37,427::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3eb220e1-a94e-4fe7-bbe4-68dc9b877d0b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '330' at 'public_spmStart' >Thread-15912185::DEBUG::2013-11-20 02:23:37,427::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >Thread-15912185::DEBUG::2013-11-20 02:23:37,428::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >Thread-15912185::DEBUG::2013-11-20 02:23:37,429::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3eb220e1-a94e-4fe7-bbe4-68dc9b877d0b`::Granted request >Thread-15912185::DEBUG::2013-11-20 02:23:37,430::task::495::TaskManager.Task::(_debug) Task 378b55ad-db2a-4978-8944-3175e4685c11: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (exclusive) >Thread-15912185::DEBUG::2013-11-20 02:23:37,430::task::495::TaskManager.Task::(_debug) Task 378b55ad-db2a-4978-8944-3175e4685c11: ref 1 aborting False >Thread-15912185::DEBUG::2013-11-20 02:23:37,431::task::495::TaskManager.Task::(_debug) Task 378b55ad-db2a-4978-8944-3175e4685c11: Prepare: 1 jobs exist, move to acquiring >Thread-15912185::DEBUG::2013-11-20 02:23:37,432::task::495::TaskManager.Task::(_debug) Task 378b55ad-db2a-4978-8944-3175e4685c11: moving from state preparing -> state acquiring >Thread-15912185::DEBUG::2013-11-20 02:23:37,432::task::495::TaskManager.Task::(_debug) Task 378b55ad-db2a-4978-8944-3175e4685c11: moving from state acquiring -> state queued >Thread-15912185::DEBUG::2013-11-20 02:23:37,433::taskManager::37::TaskManager::(_queueTask) queueing task: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912185::DEBUG::2013-11-20 02:23:37,433::taskManager::43::TaskManager::(_queueTask) task queued: 378b55ad-db2a-4978-8944-3175e4685c11 >baa736ad-461c-4f0e-89d8-6875408fdc80::DEBUG::2013-11-20 02:23:37,434::threadPool::47::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-15912185::DEBUG::2013-11-20 02:23:37,434::task::1183::TaskManager.Task::(prepare) returning >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:37,435::threadPool::192::Misc.ThreadPool.WorkerThread::(run) Task: 378b55ad-db2a-4978-8944-3175e4685c11 running: <bound method Task.commit of <storage.task.Task instance at 0x7f487c044cf8>> with: None >Thread-15912185::DEBUG::2013-11-20 02:23:37,436::task::495::TaskManager.Task::(_debug) Task 378b55ad-db2a-4978-8944-3175e4685c11: ref 0 aborting False >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:37,436::task::1195::TaskManager.Task::(commit) Commiting task: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912185::INFO::2013-11-20 02:23:37,437::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: spmStart, Return response: {'status': {'message': 'OK', 'code': 0}, 'uuid': '378b55ad-db2a-4978-8944-3175e4685c11'} >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:37,438::task::495::TaskManager.Task::(_debug) Task 378b55ad-db2a-4978-8944-3175e4685c11: moving from state queued -> state running >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:37,439::task::907::TaskManager.Task::(_runJobs) Task.run: running job 0: spmStart: <bound method SPM.start of <storage.spm.SPM instance at 0x1bf9f38>> (args: ('c4e5889a-348d-11e2-83fe-9f72369eab30', -1, '285', 0, 'false', 250, 2) kwargs: {}) >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:37,440::task::300::TaskManager.Task::(run) Job.run: running spmStart: <bound method SPM.start of <storage.spm.SPM instance at 0x1bf9f38>> (args: ('c4e5889a-348d-11e2-83fe-9f72369eab30', -1, '285', 0, 'false', 250, 2) kwargs: {}) callback None >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:37,441::spm::433::Storage.SPM::(start) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:37,441::misc::1010::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:37,442::misc::1012::SamplingMethod::(__call__) Got in to sampling method >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:37,442::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:37,443::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:37,444::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:37,444::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:37,445::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:37,446::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:38,162::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:38,165::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:38,166::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_ID=1', u'POOL_SPM_LVER=286', u'_SHA_CKSUM=09323c06acf399057c7d77c795a76a028bc8ee27'] >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:38,168::misc::1020::SamplingMethod::(__call__) Returning last result >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:38,169::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:38,169::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:38,170::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:38,170::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:38,171::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:38,172::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15912186::DEBUG::2013-11-20 02:23:38,462::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912186::INFO::2013-11-20 02:23:38,465::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912186::DEBUG::2013-11-20 02:23:38,465::task::495::TaskManager.Task::(_debug) Task b528e0ad-1118-47da-ba4b-bb7a60108173: moving from state init -> state preparing >Thread-15912186::DEBUG::2013-11-20 02:23:38,466::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912186::DEBUG::2013-11-20 02:23:38,467::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912186::DEBUG::2013-11-20 02:23:38,467::task::495::TaskManager.Task::(_debug) Task b528e0ad-1118-47da-ba4b-bb7a60108173: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912186::DEBUG::2013-11-20 02:23:38,468::task::495::TaskManager.Task::(_debug) Task b528e0ad-1118-47da-ba4b-bb7a60108173: moving from state preparing -> state finished >Thread-15912186::DEBUG::2013-11-20 02:23:38,469::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912186::DEBUG::2013-11-20 02:23:38,469::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912186::DEBUG::2013-11-20 02:23:38,470::task::495::TaskManager.Task::(_debug) Task b528e0ad-1118-47da-ba4b-bb7a60108173: ref 0 aborting False >Thread-15912186::INFO::2013-11-20 02:23:38,471::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:38,774::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:38,777::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:38,778::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_ID=1', u'POOL_SPM_LVER=286', u'_SHA_CKSUM=09323c06acf399057c7d77c795a76a028bc8ee27'] >378b55ad-db2a-4978-8944-3175e4685c11::INFO::2013-11-20 02:23:38,780::spm::457::Storage.SPM::(start) expected previd:1 lver:286 got request for previd:-1 lver:285 >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:38,781::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:38,782::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:38,782::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:38,783::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:38,784::persistentDict::163::Storage.PersistentDict::(__init__) Created a persistant dict with LvMetadataRW backend >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:38,784::blockSD::164::Storage.Misc.excCmd::(readlines) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:39,203::blockSD::164::Storage.Misc.excCmd::(readlines) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000377853 s, 5.4 MB/s\n'; <rc> = 0 >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:39,205::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000377853 s, 5.4 MB/s'], size: 2048 >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:39,205::persistentDict::204::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >378b55ad-db2a-4978-8944-3175e4685c11::WARNING::2013-11-20 02:23:39,206::persistentDict::226::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:39,206::persistentDict::163::Storage.PersistentDict::(__init__) Created a persistant dict with VGTagMetadataRW backend >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:39,207::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:39,208::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:39,208::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:39,209::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:39,210::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:39,211::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15912187::DEBUG::2013-11-20 02:23:39,495::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912187::INFO::2013-11-20 02:23:39,497::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912187::DEBUG::2013-11-20 02:23:39,498::task::495::TaskManager.Task::(_debug) Task 496de84f-15ea-456f-b27a-82a25d45f34b: moving from state init -> state preparing >Thread-15912187::DEBUG::2013-11-20 02:23:39,499::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912187::DEBUG::2013-11-20 02:23:39,499::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912187::DEBUG::2013-11-20 02:23:39,500::task::495::TaskManager.Task::(_debug) Task 496de84f-15ea-456f-b27a-82a25d45f34b: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912187::DEBUG::2013-11-20 02:23:39,501::task::495::TaskManager.Task::(_debug) Task 496de84f-15ea-456f-b27a-82a25d45f34b: moving from state preparing -> state finished >Thread-15912187::DEBUG::2013-11-20 02:23:39,501::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912187::DEBUG::2013-11-20 02:23:39,502::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912187::DEBUG::2013-11-20 02:23:39,503::task::495::TaskManager.Task::(_debug) Task 496de84f-15ea-456f-b27a-82a25d45f34b: ref 0 aborting False >Thread-15912187::INFO::2013-11-20 02:23:39,503::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:39,796::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:39,799::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:39,800::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_ID=1', u'POOL_SPM_LVER=286', u'_SHA_CKSUM=09323c06acf399057c7d77c795a76a028bc8ee27'] >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:39,801::safelease::52::ClusterLock::(acquire) Acquiring cluster lock for domain 6b892067-ac7e-4039-a7be-5b7387525b16 >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:23:39,802::safelease::61::Storage.Misc.excCmd::(acquire) '/usr/bin/sudo -n /usr/bin/setsid /usr/bin/ionice -c1 -n0 /bin/su vdsm -s /bin/sh -c "/usr/libexec/vdsm/spmprotect.sh start 6b892067-ac7e-4039-a7be-5b7387525b16 2 5 /dev/6b892067-ac7e-4039-a7be-5b7387525b16/leases 60000 10000 3"' (cwd /usr/libexec/vdsm/) >Thread-15912189::DEBUG::2013-11-20 02:23:40,523::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912189::INFO::2013-11-20 02:23:40,525::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912189::DEBUG::2013-11-20 02:23:40,525::task::495::TaskManager.Task::(_debug) Task 4119e7e8-0761-47b0-8eff-0c6f7a74af1c: moving from state init -> state preparing >Thread-15912189::DEBUG::2013-11-20 02:23:40,526::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912189::DEBUG::2013-11-20 02:23:40,526::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912189::DEBUG::2013-11-20 02:23:40,527::task::495::TaskManager.Task::(_debug) Task 4119e7e8-0761-47b0-8eff-0c6f7a74af1c: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912189::DEBUG::2013-11-20 02:23:40,528::task::495::TaskManager.Task::(_debug) Task 4119e7e8-0761-47b0-8eff-0c6f7a74af1c: moving from state preparing -> state finished >Thread-15912189::DEBUG::2013-11-20 02:23:40,528::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912189::DEBUG::2013-11-20 02:23:40,529::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912189::DEBUG::2013-11-20 02:23:40,529::task::495::TaskManager.Task::(_debug) Task 4119e7e8-0761-47b0-8eff-0c6f7a74af1c: ref 0 aborting False >Thread-15912189::INFO::2013-11-20 02:23:40,530::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912190::DEBUG::2013-11-20 02:23:41,551::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912190::INFO::2013-11-20 02:23:41,552::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912190::DEBUG::2013-11-20 02:23:41,552::task::495::TaskManager.Task::(_debug) Task 64ce2bc1-7ab6-4ebd-9f8e-3a73b0687057: moving from state init -> state preparing >Thread-15912190::DEBUG::2013-11-20 02:23:41,553::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912190::DEBUG::2013-11-20 02:23:41,553::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912190::DEBUG::2013-11-20 02:23:41,554::task::495::TaskManager.Task::(_debug) Task 64ce2bc1-7ab6-4ebd-9f8e-3a73b0687057: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912190::DEBUG::2013-11-20 02:23:41,554::task::495::TaskManager.Task::(_debug) Task 64ce2bc1-7ab6-4ebd-9f8e-3a73b0687057: moving from state preparing -> state finished >Thread-15912190::DEBUG::2013-11-20 02:23:41,555::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912190::DEBUG::2013-11-20 02:23:41,556::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912190::DEBUG::2013-11-20 02:23:41,556::task::495::TaskManager.Task::(_debug) Task 64ce2bc1-7ab6-4ebd-9f8e-3a73b0687057: ref 0 aborting False >Thread-15912190::INFO::2013-11-20 02:23:41,557::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912191::DEBUG::2013-11-20 02:23:42,577::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912191::INFO::2013-11-20 02:23:42,578::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912191::DEBUG::2013-11-20 02:23:42,578::task::495::TaskManager.Task::(_debug) Task 73529b0e-3dd9-46d7-9969-dc1f4a0bae86: moving from state init -> state preparing >Thread-15912191::DEBUG::2013-11-20 02:23:42,579::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912191::DEBUG::2013-11-20 02:23:42,579::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912191::DEBUG::2013-11-20 02:23:42,580::task::495::TaskManager.Task::(_debug) Task 73529b0e-3dd9-46d7-9969-dc1f4a0bae86: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912191::DEBUG::2013-11-20 02:23:42,581::task::495::TaskManager.Task::(_debug) Task 73529b0e-3dd9-46d7-9969-dc1f4a0bae86: moving from state preparing -> state finished >Thread-15912191::DEBUG::2013-11-20 02:23:42,581::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912191::DEBUG::2013-11-20 02:23:42,582::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912191::DEBUG::2013-11-20 02:23:42,582::task::495::TaskManager.Task::(_debug) Task 73529b0e-3dd9-46d7-9969-dc1f4a0bae86: ref 0 aborting False >Thread-15912191::INFO::2013-11-20 02:23:42,583::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-1024767::ERROR::2013-11-20 02:23:43,064::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912193::DEBUG::2013-11-20 02:23:43,601::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912193::INFO::2013-11-20 02:23:43,602::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912193::DEBUG::2013-11-20 02:23:43,603::task::495::TaskManager.Task::(_debug) Task f23bc60b-df39-47cf-8f5c-7c57f016bff0: moving from state init -> state preparing >Thread-15912193::DEBUG::2013-11-20 02:23:43,603::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912193::DEBUG::2013-11-20 02:23:43,604::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912193::DEBUG::2013-11-20 02:23:43,605::task::495::TaskManager.Task::(_debug) Task f23bc60b-df39-47cf-8f5c-7c57f016bff0: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912193::DEBUG::2013-11-20 02:23:43,605::task::495::TaskManager.Task::(_debug) Task f23bc60b-df39-47cf-8f5c-7c57f016bff0: moving from state preparing -> state finished >Thread-15912193::DEBUG::2013-11-20 02:23:43,606::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912193::DEBUG::2013-11-20 02:23:43,606::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912193::DEBUG::2013-11-20 02:23:43,607::task::495::TaskManager.Task::(_debug) Task f23bc60b-df39-47cf-8f5c-7c57f016bff0: ref 0 aborting False >Thread-15912193::INFO::2013-11-20 02:23:43,607::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912194::DEBUG::2013-11-20 02:23:44,626::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912194::INFO::2013-11-20 02:23:44,627::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912194::DEBUG::2013-11-20 02:23:44,627::task::495::TaskManager.Task::(_debug) Task 8edd6109-40c5-4482-81ea-db3e563f08f2: moving from state init -> state preparing >Thread-15912194::DEBUG::2013-11-20 02:23:44,628::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912194::DEBUG::2013-11-20 02:23:44,628::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912194::DEBUG::2013-11-20 02:23:44,629::task::495::TaskManager.Task::(_debug) Task 8edd6109-40c5-4482-81ea-db3e563f08f2: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912194::DEBUG::2013-11-20 02:23:44,630::task::495::TaskManager.Task::(_debug) Task 8edd6109-40c5-4482-81ea-db3e563f08f2: moving from state preparing -> state finished >Thread-15912194::DEBUG::2013-11-20 02:23:44,630::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912194::DEBUG::2013-11-20 02:23:44,631::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912194::DEBUG::2013-11-20 02:23:44,631::task::495::TaskManager.Task::(_debug) Task 8edd6109-40c5-4482-81ea-db3e563f08f2: ref 0 aborting False >Thread-15912194::INFO::2013-11-20 02:23:44,632::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912195::DEBUG::2013-11-20 02:23:45,656::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912195::INFO::2013-11-20 02:23:45,657::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912195::DEBUG::2013-11-20 02:23:45,657::task::495::TaskManager.Task::(_debug) Task 418a03b6-fc4f-43c5-9292-1fa607a62cd9: moving from state init -> state preparing >Thread-15912195::DEBUG::2013-11-20 02:23:45,658::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912195::DEBUG::2013-11-20 02:23:45,658::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912195::DEBUG::2013-11-20 02:23:45,659::task::495::TaskManager.Task::(_debug) Task 418a03b6-fc4f-43c5-9292-1fa607a62cd9: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912195::DEBUG::2013-11-20 02:23:45,659::task::495::TaskManager.Task::(_debug) Task 418a03b6-fc4f-43c5-9292-1fa607a62cd9: moving from state preparing -> state finished >Thread-15912195::DEBUG::2013-11-20 02:23:45,660::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912195::DEBUG::2013-11-20 02:23:45,661::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912195::DEBUG::2013-11-20 02:23:45,661::task::495::TaskManager.Task::(_debug) Task 418a03b6-fc4f-43c5-9292-1fa607a62cd9: ref 0 aborting False >Thread-15912195::INFO::2013-11-20 02:23:45,662::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912197::DEBUG::2013-11-20 02:23:46,680::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912197::INFO::2013-11-20 02:23:46,681::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912197::DEBUG::2013-11-20 02:23:46,682::task::495::TaskManager.Task::(_debug) Task afc0fd8f-339f-4a51-bee9-4c8fe6239eee: moving from state init -> state preparing >Thread-15912197::DEBUG::2013-11-20 02:23:46,682::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912197::DEBUG::2013-11-20 02:23:46,683::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912197::DEBUG::2013-11-20 02:23:46,683::task::495::TaskManager.Task::(_debug) Task afc0fd8f-339f-4a51-bee9-4c8fe6239eee: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912197::DEBUG::2013-11-20 02:23:46,684::task::495::TaskManager.Task::(_debug) Task afc0fd8f-339f-4a51-bee9-4c8fe6239eee: moving from state preparing -> state finished >Thread-15912197::DEBUG::2013-11-20 02:23:46,685::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912197::DEBUG::2013-11-20 02:23:46,685::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912197::DEBUG::2013-11-20 02:23:46,686::task::495::TaskManager.Task::(_debug) Task afc0fd8f-339f-4a51-bee9-4c8fe6239eee: ref 0 aborting False >Thread-15912197::INFO::2013-11-20 02:23:46,686::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912198::DEBUG::2013-11-20 02:23:47,705::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912198::INFO::2013-11-20 02:23:47,706::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912198::DEBUG::2013-11-20 02:23:47,706::task::495::TaskManager.Task::(_debug) Task 0471cc38-fa01-4cf5-99e5-50b8fcd668b9: moving from state init -> state preparing >Thread-15912198::DEBUG::2013-11-20 02:23:47,707::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912198::DEBUG::2013-11-20 02:23:47,707::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912198::DEBUG::2013-11-20 02:23:47,708::task::495::TaskManager.Task::(_debug) Task 0471cc38-fa01-4cf5-99e5-50b8fcd668b9: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912198::DEBUG::2013-11-20 02:23:47,709::task::495::TaskManager.Task::(_debug) Task 0471cc38-fa01-4cf5-99e5-50b8fcd668b9: moving from state preparing -> state finished >Thread-15912198::DEBUG::2013-11-20 02:23:47,709::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912198::DEBUG::2013-11-20 02:23:47,710::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912198::DEBUG::2013-11-20 02:23:47,710::task::495::TaskManager.Task::(_debug) Task 0471cc38-fa01-4cf5-99e5-50b8fcd668b9: ref 0 aborting False >Thread-15912198::INFO::2013-11-20 02:23:47,711::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912199::DEBUG::2013-11-20 02:23:48,729::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912199::INFO::2013-11-20 02:23:48,730::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912199::DEBUG::2013-11-20 02:23:48,730::task::495::TaskManager.Task::(_debug) Task 79e57ed0-fe38-4671-8ecc-25cb845d147a: moving from state init -> state preparing >Thread-15912199::DEBUG::2013-11-20 02:23:48,731::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912199::DEBUG::2013-11-20 02:23:48,731::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912199::DEBUG::2013-11-20 02:23:48,732::task::495::TaskManager.Task::(_debug) Task 79e57ed0-fe38-4671-8ecc-25cb845d147a: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912199::DEBUG::2013-11-20 02:23:48,733::task::495::TaskManager.Task::(_debug) Task 79e57ed0-fe38-4671-8ecc-25cb845d147a: moving from state preparing -> state finished >Thread-15912199::DEBUG::2013-11-20 02:23:48,733::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912199::DEBUG::2013-11-20 02:23:48,734::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912199::DEBUG::2013-11-20 02:23:48,734::task::495::TaskManager.Task::(_debug) Task 79e57ed0-fe38-4671-8ecc-25cb845d147a: ref 0 aborting False >Thread-15912199::INFO::2013-11-20 02:23:48,735::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912200::INFO::2013-11-20 02:23:49,273::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912200::DEBUG::2013-11-20 02:23:49,274::task::495::TaskManager.Task::(_debug) Task af9d74fb-5b58-4565-b6ec-53259db6f509: moving from state init -> state preparing >Thread-15912200::DEBUG::2013-11-20 02:23:49,274::task::495::TaskManager.Task::(_debug) Task af9d74fb-5b58-4565-b6ec-53259db6f509: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00028920173645', 'lastCheck': 1384914227.4294109, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00163197517395', 'lastCheck': 1384914221.662358, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384914223.065506, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00045108795166', 'lastCheck': 1384914225.7559669, 'valid': True, 'code': 0}} >Thread-15912200::DEBUG::2013-11-20 02:23:49,282::task::495::TaskManager.Task::(_debug) Task af9d74fb-5b58-4565-b6ec-53259db6f509: moving from state preparing -> state finished >Thread-15912200::DEBUG::2013-11-20 02:23:49,282::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912200::DEBUG::2013-11-20 02:23:49,283::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912200::DEBUG::2013-11-20 02:23:49,283::task::495::TaskManager.Task::(_debug) Task af9d74fb-5b58-4565-b6ec-53259db6f509: ref 0 aborting False >Thread-15912200::INFO::2013-11-20 02:23:49,286::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00028920173645', 'lastCheck': 1384914227.4294109, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00163197517395', 'lastCheck': 1384914221.662358, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384914223.065506, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00045108795166', 'lastCheck': 1384914225.7559669, 'valid': True, 'code': 0}} >Thread-15912202::DEBUG::2013-11-20 02:23:49,758::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912202::INFO::2013-11-20 02:23:49,764::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912202::DEBUG::2013-11-20 02:23:49,766::task::495::TaskManager.Task::(_debug) Task 5a416a22-d534-454c-b6c0-6fce64c077ff: moving from state init -> state preparing >Thread-15912202::DEBUG::2013-11-20 02:23:49,768::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912202::DEBUG::2013-11-20 02:23:49,769::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912202::DEBUG::2013-11-20 02:23:49,771::task::495::TaskManager.Task::(_debug) Task 5a416a22-d534-454c-b6c0-6fce64c077ff: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912202::DEBUG::2013-11-20 02:23:49,772::task::495::TaskManager.Task::(_debug) Task 5a416a22-d534-454c-b6c0-6fce64c077ff: moving from state preparing -> state finished >Thread-15912202::DEBUG::2013-11-20 02:23:49,774::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912202::DEBUG::2013-11-20 02:23:49,775::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912202::DEBUG::2013-11-20 02:23:49,776::task::495::TaskManager.Task::(_debug) Task 5a416a22-d534-454c-b6c0-6fce64c077ff: ref 0 aborting False >Thread-15912202::INFO::2013-11-20 02:23:49,776::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912203::DEBUG::2013-11-20 02:23:50,794::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912203::INFO::2013-11-20 02:23:50,795::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912203::DEBUG::2013-11-20 02:23:50,805::task::495::TaskManager.Task::(_debug) Task 0e5dbc65-b30b-435c-996d-90b50f0830f0: moving from state init -> state preparing >Thread-15912203::DEBUG::2013-11-20 02:23:50,806::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912203::DEBUG::2013-11-20 02:23:50,807::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912203::DEBUG::2013-11-20 02:23:50,807::task::495::TaskManager.Task::(_debug) Task 0e5dbc65-b30b-435c-996d-90b50f0830f0: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912203::DEBUG::2013-11-20 02:23:50,808::task::495::TaskManager.Task::(_debug) Task 0e5dbc65-b30b-435c-996d-90b50f0830f0: moving from state preparing -> state finished >Thread-15912203::DEBUG::2013-11-20 02:23:50,808::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912203::DEBUG::2013-11-20 02:23:50,809::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912203::DEBUG::2013-11-20 02:23:50,809::task::495::TaskManager.Task::(_debug) Task 0e5dbc65-b30b-435c-996d-90b50f0830f0: ref 0 aborting False >Thread-15912203::INFO::2013-11-20 02:23:50,810::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912204::DEBUG::2013-11-20 02:23:51,828::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912204::INFO::2013-11-20 02:23:51,829::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912204::DEBUG::2013-11-20 02:23:51,830::task::495::TaskManager.Task::(_debug) Task ca9dd415-6388-4d77-bda7-033942af1490: moving from state init -> state preparing >Thread-15912204::DEBUG::2013-11-20 02:23:51,830::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912204::DEBUG::2013-11-20 02:23:51,831::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912204::DEBUG::2013-11-20 02:23:51,831::task::495::TaskManager.Task::(_debug) Task ca9dd415-6388-4d77-bda7-033942af1490: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912204::DEBUG::2013-11-20 02:23:51,832::task::495::TaskManager.Task::(_debug) Task ca9dd415-6388-4d77-bda7-033942af1490: moving from state preparing -> state finished >Thread-15912204::DEBUG::2013-11-20 02:23:51,832::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912204::DEBUG::2013-11-20 02:23:51,833::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912204::DEBUG::2013-11-20 02:23:51,834::task::495::TaskManager.Task::(_debug) Task ca9dd415-6388-4d77-bda7-033942af1490: ref 0 aborting False >Thread-15912204::INFO::2013-11-20 02:23:51,834::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912206::DEBUG::2013-11-20 02:23:52,853::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912206::INFO::2013-11-20 02:23:52,854::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912206::DEBUG::2013-11-20 02:23:52,854::task::495::TaskManager.Task::(_debug) Task a1769181-bd6a-4adc-b241-6f63246139b3: moving from state init -> state preparing >Thread-15912206::DEBUG::2013-11-20 02:23:52,855::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912206::DEBUG::2013-11-20 02:23:52,856::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912206::DEBUG::2013-11-20 02:23:52,857::task::495::TaskManager.Task::(_debug) Task a1769181-bd6a-4adc-b241-6f63246139b3: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912206::DEBUG::2013-11-20 02:23:52,857::task::495::TaskManager.Task::(_debug) Task a1769181-bd6a-4adc-b241-6f63246139b3: moving from state preparing -> state finished >Thread-15912206::DEBUG::2013-11-20 02:23:52,858::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912206::DEBUG::2013-11-20 02:23:52,858::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912206::DEBUG::2013-11-20 02:23:52,859::task::495::TaskManager.Task::(_debug) Task a1769181-bd6a-4adc-b241-6f63246139b3: ref 0 aborting False >Thread-15912206::INFO::2013-11-20 02:23:52,860::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-1024767::ERROR::2013-11-20 02:23:53,078::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912207::DEBUG::2013-11-20 02:23:53,878::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912207::INFO::2013-11-20 02:23:53,879::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912207::DEBUG::2013-11-20 02:23:53,880::task::495::TaskManager.Task::(_debug) Task 068dd90f-f5d1-4c98-a5c4-1415ea869f87: moving from state init -> state preparing >Thread-15912207::DEBUG::2013-11-20 02:23:53,880::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912207::DEBUG::2013-11-20 02:23:53,881::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912207::DEBUG::2013-11-20 02:23:53,881::task::495::TaskManager.Task::(_debug) Task 068dd90f-f5d1-4c98-a5c4-1415ea869f87: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912207::DEBUG::2013-11-20 02:23:53,882::task::495::TaskManager.Task::(_debug) Task 068dd90f-f5d1-4c98-a5c4-1415ea869f87: moving from state preparing -> state finished >Thread-15912207::DEBUG::2013-11-20 02:23:53,883::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912207::DEBUG::2013-11-20 02:23:53,883::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912207::DEBUG::2013-11-20 02:23:53,884::task::495::TaskManager.Task::(_debug) Task 068dd90f-f5d1-4c98-a5c4-1415ea869f87: ref 0 aborting False >Thread-15912207::INFO::2013-11-20 02:23:53,884::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912208::DEBUG::2013-11-20 02:23:54,902::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912208::INFO::2013-11-20 02:23:54,903::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912208::DEBUG::2013-11-20 02:23:54,904::task::495::TaskManager.Task::(_debug) Task d0e3b0ac-47c5-490f-99f1-af04391ec6d4: moving from state init -> state preparing >Thread-15912208::DEBUG::2013-11-20 02:23:54,904::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912208::DEBUG::2013-11-20 02:23:54,905::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912208::DEBUG::2013-11-20 02:23:54,906::task::495::TaskManager.Task::(_debug) Task d0e3b0ac-47c5-490f-99f1-af04391ec6d4: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912208::DEBUG::2013-11-20 02:23:54,906::task::495::TaskManager.Task::(_debug) Task d0e3b0ac-47c5-490f-99f1-af04391ec6d4: moving from state preparing -> state finished >Thread-15912208::DEBUG::2013-11-20 02:23:54,907::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912208::DEBUG::2013-11-20 02:23:54,907::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912208::DEBUG::2013-11-20 02:23:54,908::task::495::TaskManager.Task::(_debug) Task d0e3b0ac-47c5-490f-99f1-af04391ec6d4: ref 0 aborting False >Thread-15912208::INFO::2013-11-20 02:23:54,908::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912210::DEBUG::2013-11-20 02:23:55,926::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912210::INFO::2013-11-20 02:23:55,927::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912210::DEBUG::2013-11-20 02:23:55,935::task::495::TaskManager.Task::(_debug) Task 39070c10-d9ec-4677-ac13-dbfa81947b3c: moving from state init -> state preparing >Thread-15912210::DEBUG::2013-11-20 02:23:55,936::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912210::DEBUG::2013-11-20 02:23:55,936::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912210::DEBUG::2013-11-20 02:23:55,937::task::495::TaskManager.Task::(_debug) Task 39070c10-d9ec-4677-ac13-dbfa81947b3c: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912210::DEBUG::2013-11-20 02:23:55,937::task::495::TaskManager.Task::(_debug) Task 39070c10-d9ec-4677-ac13-dbfa81947b3c: moving from state preparing -> state finished >Thread-15912210::DEBUG::2013-11-20 02:23:55,938::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912210::DEBUG::2013-11-20 02:23:55,939::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912210::DEBUG::2013-11-20 02:23:55,939::task::495::TaskManager.Task::(_debug) Task 39070c10-d9ec-4677-ac13-dbfa81947b3c: ref 0 aborting False >Thread-15912210::INFO::2013-11-20 02:23:55,940::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912211::DEBUG::2013-11-20 02:23:57,047::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912211::INFO::2013-11-20 02:23:57,050::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912211::DEBUG::2013-11-20 02:23:57,051::task::495::TaskManager.Task::(_debug) Task 33f6663f-b21c-4613-949b-2345537b14b1: moving from state init -> state preparing >Thread-15912211::DEBUG::2013-11-20 02:23:57,052::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912211::DEBUG::2013-11-20 02:23:57,053::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912211::DEBUG::2013-11-20 02:23:57,055::task::495::TaskManager.Task::(_debug) Task 33f6663f-b21c-4613-949b-2345537b14b1: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912211::DEBUG::2013-11-20 02:23:57,057::task::495::TaskManager.Task::(_debug) Task 33f6663f-b21c-4613-949b-2345537b14b1: moving from state preparing -> state finished >Thread-15912211::DEBUG::2013-11-20 02:23:57,058::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912211::DEBUG::2013-11-20 02:23:57,060::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912211::DEBUG::2013-11-20 02:23:57,061::task::495::TaskManager.Task::(_debug) Task 33f6663f-b21c-4613-949b-2345537b14b1: ref 0 aborting False >Thread-15912211::INFO::2013-11-20 02:23:57,062::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912212::DEBUG::2013-11-20 02:23:58,077::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912212::INFO::2013-11-20 02:23:58,077::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912212::DEBUG::2013-11-20 02:23:58,078::task::495::TaskManager.Task::(_debug) Task 50ada3f8-8403-440c-84de-ecb85b453a17: moving from state init -> state preparing >Thread-15912212::DEBUG::2013-11-20 02:23:58,079::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912212::DEBUG::2013-11-20 02:23:58,079::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912212::DEBUG::2013-11-20 02:23:58,080::task::495::TaskManager.Task::(_debug) Task 50ada3f8-8403-440c-84de-ecb85b453a17: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912212::DEBUG::2013-11-20 02:23:58,080::task::495::TaskManager.Task::(_debug) Task 50ada3f8-8403-440c-84de-ecb85b453a17: moving from state preparing -> state finished >Thread-15912212::DEBUG::2013-11-20 02:23:58,081::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912212::DEBUG::2013-11-20 02:23:58,081::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912212::DEBUG::2013-11-20 02:23:58,082::task::495::TaskManager.Task::(_debug) Task 50ada3f8-8403-440c-84de-ecb85b453a17: ref 0 aborting False >Thread-15912212::INFO::2013-11-20 02:23:58,083::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912214::DEBUG::2013-11-20 02:23:59,099::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912214::INFO::2013-11-20 02:23:59,100::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912214::DEBUG::2013-11-20 02:23:59,101::task::495::TaskManager.Task::(_debug) Task c6c45ce2-5979-455f-8a0a-334cc2426a90: moving from state init -> state preparing >Thread-15912214::DEBUG::2013-11-20 02:23:59,101::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912214::DEBUG::2013-11-20 02:23:59,102::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912214::DEBUG::2013-11-20 02:23:59,103::task::495::TaskManager.Task::(_debug) Task c6c45ce2-5979-455f-8a0a-334cc2426a90: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912214::DEBUG::2013-11-20 02:23:59,103::task::495::TaskManager.Task::(_debug) Task c6c45ce2-5979-455f-8a0a-334cc2426a90: moving from state preparing -> state finished >Thread-15912214::DEBUG::2013-11-20 02:23:59,104::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912214::DEBUG::2013-11-20 02:23:59,104::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912214::DEBUG::2013-11-20 02:23:59,105::task::495::TaskManager.Task::(_debug) Task c6c45ce2-5979-455f-8a0a-334cc2426a90: ref 0 aborting False >Thread-15912214::INFO::2013-11-20 02:23:59,105::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912215::DEBUG::2013-11-20 02:24:00,123::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912215::INFO::2013-11-20 02:24:00,123::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912215::DEBUG::2013-11-20 02:24:00,124::task::495::TaskManager.Task::(_debug) Task 91577527-f981-4e23-92c1-44f505cbdb53: moving from state init -> state preparing >Thread-15912215::DEBUG::2013-11-20 02:24:00,125::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912215::DEBUG::2013-11-20 02:24:00,125::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912215::DEBUG::2013-11-20 02:24:00,126::task::495::TaskManager.Task::(_debug) Task 91577527-f981-4e23-92c1-44f505cbdb53: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912215::DEBUG::2013-11-20 02:24:00,126::task::495::TaskManager.Task::(_debug) Task 91577527-f981-4e23-92c1-44f505cbdb53: moving from state preparing -> state finished >Thread-15912215::DEBUG::2013-11-20 02:24:00,127::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912215::DEBUG::2013-11-20 02:24:00,127::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912215::DEBUG::2013-11-20 02:24:00,128::task::495::TaskManager.Task::(_debug) Task 91577527-f981-4e23-92c1-44f505cbdb53: ref 0 aborting False >Thread-15912215::INFO::2013-11-20 02:24:00,129::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912216::DEBUG::2013-11-20 02:24:01,146::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912216::INFO::2013-11-20 02:24:01,156::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912216::DEBUG::2013-11-20 02:24:01,157::task::495::TaskManager.Task::(_debug) Task 3513f87b-a1aa-4ebb-bd35-621250a497d7: moving from state init -> state preparing >Thread-15912216::DEBUG::2013-11-20 02:24:01,157::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912216::DEBUG::2013-11-20 02:24:01,158::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912216::DEBUG::2013-11-20 02:24:01,158::task::495::TaskManager.Task::(_debug) Task 3513f87b-a1aa-4ebb-bd35-621250a497d7: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912216::DEBUG::2013-11-20 02:24:01,159::task::495::TaskManager.Task::(_debug) Task 3513f87b-a1aa-4ebb-bd35-621250a497d7: moving from state preparing -> state finished >Thread-15912216::DEBUG::2013-11-20 02:24:01,160::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912216::DEBUG::2013-11-20 02:24:01,160::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912216::DEBUG::2013-11-20 02:24:01,161::task::495::TaskManager.Task::(_debug) Task 3513f87b-a1aa-4ebb-bd35-621250a497d7: ref 0 aborting False >Thread-15912216::INFO::2013-11-20 02:24:01,161::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912218::DEBUG::2013-11-20 02:24:02,178::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912218::INFO::2013-11-20 02:24:02,179::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912218::DEBUG::2013-11-20 02:24:02,180::task::495::TaskManager.Task::(_debug) Task 1908030a-d4d6-411d-89ac-3c10a465011d: moving from state init -> state preparing >Thread-15912218::DEBUG::2013-11-20 02:24:02,180::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912218::DEBUG::2013-11-20 02:24:02,181::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912218::DEBUG::2013-11-20 02:24:02,182::task::495::TaskManager.Task::(_debug) Task 1908030a-d4d6-411d-89ac-3c10a465011d: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912218::DEBUG::2013-11-20 02:24:02,182::task::495::TaskManager.Task::(_debug) Task 1908030a-d4d6-411d-89ac-3c10a465011d: moving from state preparing -> state finished >Thread-15912218::DEBUG::2013-11-20 02:24:02,183::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912218::DEBUG::2013-11-20 02:24:02,183::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912218::DEBUG::2013-11-20 02:24:02,184::task::495::TaskManager.Task::(_debug) Task 1908030a-d4d6-411d-89ac-3c10a465011d: ref 0 aborting False >Thread-15912218::INFO::2013-11-20 02:24:02,184::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-1024767::ERROR::2013-11-20 02:24:03,091::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912219::DEBUG::2013-11-20 02:24:03,201::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912219::INFO::2013-11-20 02:24:03,202::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912219::DEBUG::2013-11-20 02:24:03,202::task::495::TaskManager.Task::(_debug) Task 81fe65b5-aadf-482f-859f-8f57ecfd6792: moving from state init -> state preparing >Thread-15912219::DEBUG::2013-11-20 02:24:03,203::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912219::DEBUG::2013-11-20 02:24:03,203::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912219::DEBUG::2013-11-20 02:24:03,204::task::495::TaskManager.Task::(_debug) Task 81fe65b5-aadf-482f-859f-8f57ecfd6792: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912219::DEBUG::2013-11-20 02:24:03,205::task::495::TaskManager.Task::(_debug) Task 81fe65b5-aadf-482f-859f-8f57ecfd6792: moving from state preparing -> state finished >Thread-15912219::DEBUG::2013-11-20 02:24:03,205::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912219::DEBUG::2013-11-20 02:24:03,206::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912219::DEBUG::2013-11-20 02:24:03,206::task::495::TaskManager.Task::(_debug) Task 81fe65b5-aadf-482f-859f-8f57ecfd6792: ref 0 aborting False >Thread-15912219::INFO::2013-11-20 02:24:03,207::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912220::DEBUG::2013-11-20 02:24:04,224::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912220::INFO::2013-11-20 02:24:04,224::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912220::DEBUG::2013-11-20 02:24:04,225::task::495::TaskManager.Task::(_debug) Task 59a840ba-3ca0-43e1-a37c-91846f476bca: moving from state init -> state preparing >Thread-15912220::DEBUG::2013-11-20 02:24:04,226::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912220::DEBUG::2013-11-20 02:24:04,226::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912220::DEBUG::2013-11-20 02:24:04,227::task::495::TaskManager.Task::(_debug) Task 59a840ba-3ca0-43e1-a37c-91846f476bca: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912220::DEBUG::2013-11-20 02:24:04,227::task::495::TaskManager.Task::(_debug) Task 59a840ba-3ca0-43e1-a37c-91846f476bca: moving from state preparing -> state finished >Thread-15912220::DEBUG::2013-11-20 02:24:04,228::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912220::DEBUG::2013-11-20 02:24:04,229::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912220::DEBUG::2013-11-20 02:24:04,229::task::495::TaskManager.Task::(_debug) Task 59a840ba-3ca0-43e1-a37c-91846f476bca: ref 0 aborting False >Thread-15912220::INFO::2013-11-20 02:24:04,230::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912221::INFO::2013-11-20 02:24:04,689::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912221::DEBUG::2013-11-20 02:24:04,689::task::495::TaskManager.Task::(_debug) Task 7755a3c3-14e2-4700-9ecd-43b64446c35c: moving from state init -> state preparing >Thread-15912221::DEBUG::2013-11-20 02:24:04,690::task::495::TaskManager.Task::(_debug) Task 7755a3c3-14e2-4700-9ecd-43b64446c35c: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000368118286133', 'lastCheck': 1384914237.44241, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00153207778931', 'lastCheck': 1384914241.6933701, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384914243.092361, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00133299827576', 'lastCheck': 1384914235.768889, 'valid': True, 'code': 0}} >Thread-15912221::DEBUG::2013-11-20 02:24:04,696::task::495::TaskManager.Task::(_debug) Task 7755a3c3-14e2-4700-9ecd-43b64446c35c: moving from state preparing -> state finished >Thread-15912221::DEBUG::2013-11-20 02:24:04,697::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912221::DEBUG::2013-11-20 02:24:04,698::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912221::DEBUG::2013-11-20 02:24:04,698::task::495::TaskManager.Task::(_debug) Task 7755a3c3-14e2-4700-9ecd-43b64446c35c: ref 0 aborting False >Thread-15912221::INFO::2013-11-20 02:24:04,699::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000368118286133', 'lastCheck': 1384914237.44241, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00153207778931', 'lastCheck': 1384914241.6933701, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384914243.092361, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00133299827576', 'lastCheck': 1384914235.768889, 'valid': True, 'code': 0}} >Thread-15912223::DEBUG::2013-11-20 02:24:05,247::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912223::INFO::2013-11-20 02:24:05,248::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912223::DEBUG::2013-11-20 02:24:05,248::task::495::TaskManager.Task::(_debug) Task 87bad5af-01e0-43a8-a2f9-596b2caeef03: moving from state init -> state preparing >Thread-15912223::DEBUG::2013-11-20 02:24:05,249::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912223::DEBUG::2013-11-20 02:24:05,249::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912223::DEBUG::2013-11-20 02:24:05,250::task::495::TaskManager.Task::(_debug) Task 87bad5af-01e0-43a8-a2f9-596b2caeef03: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912223::DEBUG::2013-11-20 02:24:05,250::task::495::TaskManager.Task::(_debug) Task 87bad5af-01e0-43a8-a2f9-596b2caeef03: moving from state preparing -> state finished >Thread-15912223::DEBUG::2013-11-20 02:24:05,251::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912223::DEBUG::2013-11-20 02:24:05,251::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912223::DEBUG::2013-11-20 02:24:05,252::task::495::TaskManager.Task::(_debug) Task 87bad5af-01e0-43a8-a2f9-596b2caeef03: ref 0 aborting False >Thread-15912223::INFO::2013-11-20 02:24:05,253::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912224::DEBUG::2013-11-20 02:24:06,269::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912224::INFO::2013-11-20 02:24:06,270::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912224::DEBUG::2013-11-20 02:24:06,270::task::495::TaskManager.Task::(_debug) Task a0e052e6-1fa9-48a2-b39d-3732fbef8a47: moving from state init -> state preparing >Thread-15912224::DEBUG::2013-11-20 02:24:06,271::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912224::DEBUG::2013-11-20 02:24:06,271::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912224::DEBUG::2013-11-20 02:24:06,272::task::495::TaskManager.Task::(_debug) Task a0e052e6-1fa9-48a2-b39d-3732fbef8a47: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912224::DEBUG::2013-11-20 02:24:06,272::task::495::TaskManager.Task::(_debug) Task a0e052e6-1fa9-48a2-b39d-3732fbef8a47: moving from state preparing -> state finished >Thread-15912224::DEBUG::2013-11-20 02:24:06,273::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912224::DEBUG::2013-11-20 02:24:06,274::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912224::DEBUG::2013-11-20 02:24:06,274::task::495::TaskManager.Task::(_debug) Task a0e052e6-1fa9-48a2-b39d-3732fbef8a47: ref 0 aborting False >Thread-15912224::INFO::2013-11-20 02:24:06,275::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912225::DEBUG::2013-11-20 02:24:07,291::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912225::INFO::2013-11-20 02:24:07,292::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912225::DEBUG::2013-11-20 02:24:07,293::task::495::TaskManager.Task::(_debug) Task b8ed8649-6aff-42f2-b4ea-61f7270352c3: moving from state init -> state preparing >Thread-15912225::DEBUG::2013-11-20 02:24:07,293::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912225::DEBUG::2013-11-20 02:24:07,294::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912225::DEBUG::2013-11-20 02:24:07,295::task::495::TaskManager.Task::(_debug) Task b8ed8649-6aff-42f2-b4ea-61f7270352c3: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912225::DEBUG::2013-11-20 02:24:07,295::task::495::TaskManager.Task::(_debug) Task b8ed8649-6aff-42f2-b4ea-61f7270352c3: moving from state preparing -> state finished >Thread-15912225::DEBUG::2013-11-20 02:24:07,296::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912225::DEBUG::2013-11-20 02:24:07,296::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912225::DEBUG::2013-11-20 02:24:07,297::task::495::TaskManager.Task::(_debug) Task b8ed8649-6aff-42f2-b4ea-61f7270352c3: ref 0 aborting False >Thread-15912225::INFO::2013-11-20 02:24:07,297::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912227::DEBUG::2013-11-20 02:24:08,370::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912227::INFO::2013-11-20 02:24:08,371::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912227::DEBUG::2013-11-20 02:24:08,371::task::495::TaskManager.Task::(_debug) Task 48fbbb8c-6324-49e6-b1c9-3e98e67127d9: moving from state init -> state preparing >Thread-15912227::DEBUG::2013-11-20 02:24:08,372::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912227::DEBUG::2013-11-20 02:24:08,373::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912227::DEBUG::2013-11-20 02:24:08,373::task::495::TaskManager.Task::(_debug) Task 48fbbb8c-6324-49e6-b1c9-3e98e67127d9: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912227::DEBUG::2013-11-20 02:24:08,374::task::495::TaskManager.Task::(_debug) Task 48fbbb8c-6324-49e6-b1c9-3e98e67127d9: moving from state preparing -> state finished >Thread-15912227::DEBUG::2013-11-20 02:24:08,374::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912227::DEBUG::2013-11-20 02:24:08,375::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912227::DEBUG::2013-11-20 02:24:08,375::task::495::TaskManager.Task::(_debug) Task 48fbbb8c-6324-49e6-b1c9-3e98e67127d9: ref 0 aborting False >Thread-15912227::INFO::2013-11-20 02:24:08,376::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912228::DEBUG::2013-11-20 02:24:09,392::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912228::INFO::2013-11-20 02:24:09,393::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912228::DEBUG::2013-11-20 02:24:09,394::task::495::TaskManager.Task::(_debug) Task 04aecaaa-e288-4a00-9539-77f459f57cca: moving from state init -> state preparing >Thread-15912228::DEBUG::2013-11-20 02:24:09,394::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912228::DEBUG::2013-11-20 02:24:09,395::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912228::DEBUG::2013-11-20 02:24:09,396::task::495::TaskManager.Task::(_debug) Task 04aecaaa-e288-4a00-9539-77f459f57cca: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912228::DEBUG::2013-11-20 02:24:09,396::task::495::TaskManager.Task::(_debug) Task 04aecaaa-e288-4a00-9539-77f459f57cca: moving from state preparing -> state finished >Thread-15912228::DEBUG::2013-11-20 02:24:09,397::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912228::DEBUG::2013-11-20 02:24:09,397::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912228::DEBUG::2013-11-20 02:24:09,398::task::495::TaskManager.Task::(_debug) Task 04aecaaa-e288-4a00-9539-77f459f57cca: ref 0 aborting False >Thread-15912228::INFO::2013-11-20 02:24:09,399::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912229::DEBUG::2013-11-20 02:24:10,426::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912229::INFO::2013-11-20 02:24:10,427::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912229::DEBUG::2013-11-20 02:24:10,427::task::495::TaskManager.Task::(_debug) Task 615bafca-d21d-4994-aa7f-514b30a33af2: moving from state init -> state preparing >Thread-15912229::DEBUG::2013-11-20 02:24:10,428::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912229::DEBUG::2013-11-20 02:24:10,429::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912229::DEBUG::2013-11-20 02:24:10,429::task::495::TaskManager.Task::(_debug) Task 615bafca-d21d-4994-aa7f-514b30a33af2: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912229::DEBUG::2013-11-20 02:24:10,430::task::495::TaskManager.Task::(_debug) Task 615bafca-d21d-4994-aa7f-514b30a33af2: moving from state preparing -> state finished >Thread-15912229::DEBUG::2013-11-20 02:24:10,430::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912229::DEBUG::2013-11-20 02:24:10,431::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912229::DEBUG::2013-11-20 02:24:10,431::task::495::TaskManager.Task::(_debug) Task 615bafca-d21d-4994-aa7f-514b30a33af2: ref 0 aborting False >Thread-15912229::INFO::2013-11-20 02:24:10,432::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912231::DEBUG::2013-11-20 02:24:11,448::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912231::INFO::2013-11-20 02:24:11,449::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912231::DEBUG::2013-11-20 02:24:11,462::task::495::TaskManager.Task::(_debug) Task 471c6f10-0654-4081-8f65-054754b14c6d: moving from state init -> state preparing >Thread-15912231::DEBUG::2013-11-20 02:24:11,463::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912231::DEBUG::2013-11-20 02:24:11,464::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912231::DEBUG::2013-11-20 02:24:11,464::task::495::TaskManager.Task::(_debug) Task 471c6f10-0654-4081-8f65-054754b14c6d: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912231::DEBUG::2013-11-20 02:24:11,465::task::495::TaskManager.Task::(_debug) Task 471c6f10-0654-4081-8f65-054754b14c6d: moving from state preparing -> state finished >Thread-15912231::DEBUG::2013-11-20 02:24:11,465::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912231::DEBUG::2013-11-20 02:24:11,466::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912231::DEBUG::2013-11-20 02:24:11,466::task::495::TaskManager.Task::(_debug) Task 471c6f10-0654-4081-8f65-054754b14c6d: ref 0 aborting False >Thread-15912231::INFO::2013-11-20 02:24:11,467::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912232::DEBUG::2013-11-20 02:24:12,483::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912232::INFO::2013-11-20 02:24:12,484::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912232::DEBUG::2013-11-20 02:24:12,485::task::495::TaskManager.Task::(_debug) Task 61e4d095-8d63-4c8a-9b0f-5a4b7d83048d: moving from state init -> state preparing >Thread-15912232::DEBUG::2013-11-20 02:24:12,485::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912232::DEBUG::2013-11-20 02:24:12,486::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912232::DEBUG::2013-11-20 02:24:12,486::task::495::TaskManager.Task::(_debug) Task 61e4d095-8d63-4c8a-9b0f-5a4b7d83048d: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912232::DEBUG::2013-11-20 02:24:12,487::task::495::TaskManager.Task::(_debug) Task 61e4d095-8d63-4c8a-9b0f-5a4b7d83048d: moving from state preparing -> state finished >Thread-15912232::DEBUG::2013-11-20 02:24:12,487::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912232::DEBUG::2013-11-20 02:24:12,488::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912232::DEBUG::2013-11-20 02:24:12,489::task::495::TaskManager.Task::(_debug) Task 61e4d095-8d63-4c8a-9b0f-5a4b7d83048d: ref 0 aborting False >Thread-15912232::INFO::2013-11-20 02:24:12,489::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-1024767::ERROR::2013-11-20 02:24:13,104::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912233::DEBUG::2013-11-20 02:24:13,506::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912233::INFO::2013-11-20 02:24:13,507::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912233::DEBUG::2013-11-20 02:24:13,507::task::495::TaskManager.Task::(_debug) Task 0360ec9e-a963-482a-8133-771bbb8897b4: moving from state init -> state preparing >Thread-15912233::DEBUG::2013-11-20 02:24:13,508::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912233::DEBUG::2013-11-20 02:24:13,508::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912233::DEBUG::2013-11-20 02:24:13,509::task::495::TaskManager.Task::(_debug) Task 0360ec9e-a963-482a-8133-771bbb8897b4: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912233::DEBUG::2013-11-20 02:24:13,510::task::495::TaskManager.Task::(_debug) Task 0360ec9e-a963-482a-8133-771bbb8897b4: moving from state preparing -> state finished >Thread-15912233::DEBUG::2013-11-20 02:24:13,510::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912233::DEBUG::2013-11-20 02:24:13,511::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912233::DEBUG::2013-11-20 02:24:13,511::task::495::TaskManager.Task::(_debug) Task 0360ec9e-a963-482a-8133-771bbb8897b4: ref 0 aborting False >Thread-15912233::INFO::2013-11-20 02:24:13,512::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912235::DEBUG::2013-11-20 02:24:14,528::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912235::INFO::2013-11-20 02:24:14,528::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912235::DEBUG::2013-11-20 02:24:14,529::task::495::TaskManager.Task::(_debug) Task 033d1e7f-1f8f-4382-93a3-200bdf508f3b: moving from state init -> state preparing >Thread-15912235::DEBUG::2013-11-20 02:24:14,530::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912235::DEBUG::2013-11-20 02:24:14,530::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912235::DEBUG::2013-11-20 02:24:14,531::task::495::TaskManager.Task::(_debug) Task 033d1e7f-1f8f-4382-93a3-200bdf508f3b: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912235::DEBUG::2013-11-20 02:24:14,531::task::495::TaskManager.Task::(_debug) Task 033d1e7f-1f8f-4382-93a3-200bdf508f3b: moving from state preparing -> state finished >Thread-15912235::DEBUG::2013-11-20 02:24:14,532::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912235::DEBUG::2013-11-20 02:24:14,532::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912235::DEBUG::2013-11-20 02:24:14,533::task::495::TaskManager.Task::(_debug) Task 033d1e7f-1f8f-4382-93a3-200bdf508f3b: ref 0 aborting False >Thread-15912235::INFO::2013-11-20 02:24:14,534::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912236::DEBUG::2013-11-20 02:24:15,550::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912236::INFO::2013-11-20 02:24:15,551::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912236::DEBUG::2013-11-20 02:24:15,552::task::495::TaskManager.Task::(_debug) Task 0ba1c62a-efb8-4c6b-851a-eccd49e770f6: moving from state init -> state preparing >Thread-15912236::DEBUG::2013-11-20 02:24:15,552::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912236::DEBUG::2013-11-20 02:24:15,553::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912236::DEBUG::2013-11-20 02:24:15,553::task::495::TaskManager.Task::(_debug) Task 0ba1c62a-efb8-4c6b-851a-eccd49e770f6: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912236::DEBUG::2013-11-20 02:24:15,554::task::495::TaskManager.Task::(_debug) Task 0ba1c62a-efb8-4c6b-851a-eccd49e770f6: moving from state preparing -> state finished >Thread-15912236::DEBUG::2013-11-20 02:24:15,554::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912236::DEBUG::2013-11-20 02:24:15,555::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912236::DEBUG::2013-11-20 02:24:15,556::task::495::TaskManager.Task::(_debug) Task 0ba1c62a-efb8-4c6b-851a-eccd49e770f6: ref 0 aborting False >Thread-15912236::INFO::2013-11-20 02:24:15,556::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912237::DEBUG::2013-11-20 02:24:16,572::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912237::INFO::2013-11-20 02:24:16,573::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912237::DEBUG::2013-11-20 02:24:16,583::task::495::TaskManager.Task::(_debug) Task b8c87626-2967-46dd-b7c0-130ac7b73b0d: moving from state init -> state preparing >Thread-15912237::DEBUG::2013-11-20 02:24:16,584::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912237::DEBUG::2013-11-20 02:24:16,584::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912237::DEBUG::2013-11-20 02:24:16,585::task::495::TaskManager.Task::(_debug) Task b8c87626-2967-46dd-b7c0-130ac7b73b0d: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912237::DEBUG::2013-11-20 02:24:16,586::task::495::TaskManager.Task::(_debug) Task b8c87626-2967-46dd-b7c0-130ac7b73b0d: moving from state preparing -> state finished >Thread-15912237::DEBUG::2013-11-20 02:24:16,586::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912237::DEBUG::2013-11-20 02:24:16,587::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912237::DEBUG::2013-11-20 02:24:16,587::task::495::TaskManager.Task::(_debug) Task b8c87626-2967-46dd-b7c0-130ac7b73b0d: ref 0 aborting False >Thread-15912237::INFO::2013-11-20 02:24:16,588::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912239::DEBUG::2013-11-20 02:24:17,604::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912239::INFO::2013-11-20 02:24:17,605::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912239::DEBUG::2013-11-20 02:24:17,605::task::495::TaskManager.Task::(_debug) Task 0e607d07-16fe-4918-81d4-d82bf9004ff5: moving from state init -> state preparing >Thread-15912239::DEBUG::2013-11-20 02:24:17,606::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912239::DEBUG::2013-11-20 02:24:17,606::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912239::DEBUG::2013-11-20 02:24:17,607::task::495::TaskManager.Task::(_debug) Task 0e607d07-16fe-4918-81d4-d82bf9004ff5: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912239::DEBUG::2013-11-20 02:24:17,608::task::495::TaskManager.Task::(_debug) Task 0e607d07-16fe-4918-81d4-d82bf9004ff5: moving from state preparing -> state finished >Thread-15912239::DEBUG::2013-11-20 02:24:17,608::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912239::DEBUG::2013-11-20 02:24:17,609::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912239::DEBUG::2013-11-20 02:24:17,609::task::495::TaskManager.Task::(_debug) Task 0e607d07-16fe-4918-81d4-d82bf9004ff5: ref 0 aborting False >Thread-15912239::INFO::2013-11-20 02:24:17,610::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912240::DEBUG::2013-11-20 02:24:18,626::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912240::INFO::2013-11-20 02:24:18,627::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912240::DEBUG::2013-11-20 02:24:18,627::task::495::TaskManager.Task::(_debug) Task 82255666-02c8-4863-a52b-19f8789a9ff2: moving from state init -> state preparing >Thread-15912240::DEBUG::2013-11-20 02:24:18,628::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912240::DEBUG::2013-11-20 02:24:18,629::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912240::DEBUG::2013-11-20 02:24:18,629::task::495::TaskManager.Task::(_debug) Task 82255666-02c8-4863-a52b-19f8789a9ff2: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912240::DEBUG::2013-11-20 02:24:18,630::task::495::TaskManager.Task::(_debug) Task 82255666-02c8-4863-a52b-19f8789a9ff2: moving from state preparing -> state finished >Thread-15912240::DEBUG::2013-11-20 02:24:18,630::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912240::DEBUG::2013-11-20 02:24:18,631::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912240::DEBUG::2013-11-20 02:24:18,631::task::495::TaskManager.Task::(_debug) Task 82255666-02c8-4863-a52b-19f8789a9ff2: ref 0 aborting False >Thread-15912240::INFO::2013-11-20 02:24:18,632::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912241::DEBUG::2013-11-20 02:24:19,648::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912241::INFO::2013-11-20 02:24:19,649::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912241::DEBUG::2013-11-20 02:24:19,650::task::495::TaskManager.Task::(_debug) Task 2e5c3fef-f318-486b-a5d6-9752ae9db30a: moving from state init -> state preparing >Thread-15912241::DEBUG::2013-11-20 02:24:19,650::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912241::DEBUG::2013-11-20 02:24:19,651::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912241::DEBUG::2013-11-20 02:24:19,651::task::495::TaskManager.Task::(_debug) Task 2e5c3fef-f318-486b-a5d6-9752ae9db30a: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912241::DEBUG::2013-11-20 02:24:19,652::task::495::TaskManager.Task::(_debug) Task 2e5c3fef-f318-486b-a5d6-9752ae9db30a: moving from state preparing -> state finished >Thread-15912241::DEBUG::2013-11-20 02:24:19,653::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912241::DEBUG::2013-11-20 02:24:19,653::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912241::DEBUG::2013-11-20 02:24:19,654::task::495::TaskManager.Task::(_debug) Task 2e5c3fef-f318-486b-a5d6-9752ae9db30a: ref 0 aborting False >Thread-15912241::INFO::2013-11-20 02:24:19,654::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912242::INFO::2013-11-20 02:24:19,974::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912242::DEBUG::2013-11-20 02:24:19,975::task::495::TaskManager.Task::(_debug) Task 89d49a38-d7dc-46a0-9765-24868f00ee0f: moving from state init -> state preparing >Thread-15912242::DEBUG::2013-11-20 02:24:19,976::task::495::TaskManager.Task::(_debug) Task 89d49a38-d7dc-46a0-9765-24868f00ee0f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000293970108032', 'lastCheck': 1384914257.467984, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00145292282104', 'lastCheck': 1384914251.7094791, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384914253.1056781, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000391006469727', 'lastCheck': 1384914255.791477, 'valid': True, 'code': 0}} >Thread-15912242::DEBUG::2013-11-20 02:24:19,992::task::495::TaskManager.Task::(_debug) Task 89d49a38-d7dc-46a0-9765-24868f00ee0f: moving from state preparing -> state finished >Thread-15912242::DEBUG::2013-11-20 02:24:19,993::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912242::DEBUG::2013-11-20 02:24:19,993::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912242::DEBUG::2013-11-20 02:24:19,994::task::495::TaskManager.Task::(_debug) Task 89d49a38-d7dc-46a0-9765-24868f00ee0f: ref 0 aborting False >Thread-15912242::INFO::2013-11-20 02:24:19,995::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000293970108032', 'lastCheck': 1384914257.467984, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00145292282104', 'lastCheck': 1384914251.7094791, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384914253.1056781, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000391006469727', 'lastCheck': 1384914255.791477, 'valid': True, 'code': 0}} >Thread-15912244::DEBUG::2013-11-20 02:24:20,671::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912244::INFO::2013-11-20 02:24:20,671::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912244::DEBUG::2013-11-20 02:24:20,672::task::495::TaskManager.Task::(_debug) Task fe2e264c-5938-4ca3-a40e-34fe3209d985: moving from state init -> state preparing >Thread-15912244::DEBUG::2013-11-20 02:24:20,673::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912244::DEBUG::2013-11-20 02:24:20,673::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912244::DEBUG::2013-11-20 02:24:20,674::task::495::TaskManager.Task::(_debug) Task fe2e264c-5938-4ca3-a40e-34fe3209d985: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912244::DEBUG::2013-11-20 02:24:20,674::task::495::TaskManager.Task::(_debug) Task fe2e264c-5938-4ca3-a40e-34fe3209d985: moving from state preparing -> state finished >Thread-15912244::DEBUG::2013-11-20 02:24:20,675::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912244::DEBUG::2013-11-20 02:24:20,675::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912244::DEBUG::2013-11-20 02:24:20,676::task::495::TaskManager.Task::(_debug) Task fe2e264c-5938-4ca3-a40e-34fe3209d985: ref 0 aborting False >Thread-15912244::INFO::2013-11-20 02:24:20,676::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912245::DEBUG::2013-11-20 02:24:21,692::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912245::INFO::2013-11-20 02:24:21,693::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912245::DEBUG::2013-11-20 02:24:21,704::task::495::TaskManager.Task::(_debug) Task dddc1e7b-e597-46b6-bb77-512fbe36fa10: moving from state init -> state preparing >Thread-15912245::DEBUG::2013-11-20 02:24:21,704::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912245::DEBUG::2013-11-20 02:24:21,705::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912245::DEBUG::2013-11-20 02:24:21,705::task::495::TaskManager.Task::(_debug) Task dddc1e7b-e597-46b6-bb77-512fbe36fa10: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912245::DEBUG::2013-11-20 02:24:21,706::task::495::TaskManager.Task::(_debug) Task dddc1e7b-e597-46b6-bb77-512fbe36fa10: moving from state preparing -> state finished >Thread-15912245::DEBUG::2013-11-20 02:24:21,707::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912245::DEBUG::2013-11-20 02:24:21,707::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912245::DEBUG::2013-11-20 02:24:21,708::task::495::TaskManager.Task::(_debug) Task dddc1e7b-e597-46b6-bb77-512fbe36fa10: ref 0 aborting False >Thread-15912245::INFO::2013-11-20 02:24:21,708::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912246::DEBUG::2013-11-20 02:24:22,724::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912246::INFO::2013-11-20 02:24:22,725::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912246::DEBUG::2013-11-20 02:24:22,725::task::495::TaskManager.Task::(_debug) Task 0a03f74d-9698-4f19-871b-9ea76ff3db07: moving from state init -> state preparing >Thread-15912246::DEBUG::2013-11-20 02:24:22,726::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912246::DEBUG::2013-11-20 02:24:22,727::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912246::DEBUG::2013-11-20 02:24:22,727::task::495::TaskManager.Task::(_debug) Task 0a03f74d-9698-4f19-871b-9ea76ff3db07: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912246::DEBUG::2013-11-20 02:24:22,728::task::495::TaskManager.Task::(_debug) Task 0a03f74d-9698-4f19-871b-9ea76ff3db07: moving from state preparing -> state finished >Thread-15912246::DEBUG::2013-11-20 02:24:22,729::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912246::DEBUG::2013-11-20 02:24:22,729::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912246::DEBUG::2013-11-20 02:24:22,730::task::495::TaskManager.Task::(_debug) Task 0a03f74d-9698-4f19-871b-9ea76ff3db07: ref 0 aborting False >Thread-15912246::INFO::2013-11-20 02:24:22,730::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-1024767::ERROR::2013-11-20 02:24:23,118::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912248::DEBUG::2013-11-20 02:24:23,747::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912248::INFO::2013-11-20 02:24:23,748::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912248::DEBUG::2013-11-20 02:24:23,749::task::495::TaskManager.Task::(_debug) Task 12ce0d53-c02f-402a-88e1-5d82b6d3f605: moving from state init -> state preparing >Thread-15912248::DEBUG::2013-11-20 02:24:23,749::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912248::DEBUG::2013-11-20 02:24:23,750::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912248::DEBUG::2013-11-20 02:24:23,750::task::495::TaskManager.Task::(_debug) Task 12ce0d53-c02f-402a-88e1-5d82b6d3f605: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912248::DEBUG::2013-11-20 02:24:23,751::task::495::TaskManager.Task::(_debug) Task 12ce0d53-c02f-402a-88e1-5d82b6d3f605: moving from state preparing -> state finished >Thread-15912248::DEBUG::2013-11-20 02:24:23,752::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912248::DEBUG::2013-11-20 02:24:23,752::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912248::DEBUG::2013-11-20 02:24:23,753::task::495::TaskManager.Task::(_debug) Task 12ce0d53-c02f-402a-88e1-5d82b6d3f605: ref 0 aborting False >Thread-15912248::INFO::2013-11-20 02:24:23,753::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912249::DEBUG::2013-11-20 02:24:24,770::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912249::INFO::2013-11-20 02:24:24,771::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912249::DEBUG::2013-11-20 02:24:24,771::task::495::TaskManager.Task::(_debug) Task 48028d05-bb67-4182-99f3-750d218cd0b2: moving from state init -> state preparing >Thread-15912249::DEBUG::2013-11-20 02:24:24,772::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912249::DEBUG::2013-11-20 02:24:24,772::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912249::DEBUG::2013-11-20 02:24:24,773::task::495::TaskManager.Task::(_debug) Task 48028d05-bb67-4182-99f3-750d218cd0b2: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912249::DEBUG::2013-11-20 02:24:24,774::task::495::TaskManager.Task::(_debug) Task 48028d05-bb67-4182-99f3-750d218cd0b2: moving from state preparing -> state finished >Thread-15912249::DEBUG::2013-11-20 02:24:24,774::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912249::DEBUG::2013-11-20 02:24:24,775::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912249::DEBUG::2013-11-20 02:24:24,775::task::495::TaskManager.Task::(_debug) Task 48028d05-bb67-4182-99f3-750d218cd0b2: ref 0 aborting False >Thread-15912249::INFO::2013-11-20 02:24:24,776::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912250::DEBUG::2013-11-20 02:24:25,792::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912250::INFO::2013-11-20 02:24:25,793::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912250::DEBUG::2013-11-20 02:24:25,793::task::495::TaskManager.Task::(_debug) Task c4f9e3b8-ed14-48d8-8bc1-fb41bb114084: moving from state init -> state preparing >Thread-15912250::DEBUG::2013-11-20 02:24:25,794::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912250::DEBUG::2013-11-20 02:24:25,794::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912250::DEBUG::2013-11-20 02:24:25,795::task::495::TaskManager.Task::(_debug) Task c4f9e3b8-ed14-48d8-8bc1-fb41bb114084: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912250::DEBUG::2013-11-20 02:24:25,796::task::495::TaskManager.Task::(_debug) Task c4f9e3b8-ed14-48d8-8bc1-fb41bb114084: moving from state preparing -> state finished >Thread-15912250::DEBUG::2013-11-20 02:24:25,796::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912250::DEBUG::2013-11-20 02:24:25,797::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912250::DEBUG::2013-11-20 02:24:25,797::task::495::TaskManager.Task::(_debug) Task c4f9e3b8-ed14-48d8-8bc1-fb41bb114084: ref 0 aborting False >Thread-15912250::INFO::2013-11-20 02:24:25,798::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912252::DEBUG::2013-11-20 02:24:26,814::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912252::INFO::2013-11-20 02:24:26,815::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912252::DEBUG::2013-11-20 02:24:26,826::task::495::TaskManager.Task::(_debug) Task d09f706e-0dd7-40e5-8905-4e653cd32c5b: moving from state init -> state preparing >Thread-15912252::DEBUG::2013-11-20 02:24:26,826::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912252::DEBUG::2013-11-20 02:24:26,827::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912252::DEBUG::2013-11-20 02:24:26,828::task::495::TaskManager.Task::(_debug) Task d09f706e-0dd7-40e5-8905-4e653cd32c5b: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912252::DEBUG::2013-11-20 02:24:26,828::task::495::TaskManager.Task::(_debug) Task d09f706e-0dd7-40e5-8905-4e653cd32c5b: moving from state preparing -> state finished >Thread-15912252::DEBUG::2013-11-20 02:24:26,829::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912252::DEBUG::2013-11-20 02:24:26,829::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912252::DEBUG::2013-11-20 02:24:26,830::task::495::TaskManager.Task::(_debug) Task d09f706e-0dd7-40e5-8905-4e653cd32c5b: ref 0 aborting False >Thread-15912252::INFO::2013-11-20 02:24:26,830::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912253::DEBUG::2013-11-20 02:24:27,846::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912253::INFO::2013-11-20 02:24:27,847::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912253::DEBUG::2013-11-20 02:24:27,848::task::495::TaskManager.Task::(_debug) Task 894a6175-4f1a-45b8-b2f9-5dddb3442783: moving from state init -> state preparing >Thread-15912253::DEBUG::2013-11-20 02:24:27,848::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912253::DEBUG::2013-11-20 02:24:27,849::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912253::DEBUG::2013-11-20 02:24:27,850::task::495::TaskManager.Task::(_debug) Task 894a6175-4f1a-45b8-b2f9-5dddb3442783: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912253::DEBUG::2013-11-20 02:24:27,850::task::495::TaskManager.Task::(_debug) Task 894a6175-4f1a-45b8-b2f9-5dddb3442783: moving from state preparing -> state finished >Thread-15912253::DEBUG::2013-11-20 02:24:27,851::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912253::DEBUG::2013-11-20 02:24:27,851::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912253::DEBUG::2013-11-20 02:24:27,852::task::495::TaskManager.Task::(_debug) Task 894a6175-4f1a-45b8-b2f9-5dddb3442783: ref 0 aborting False >Thread-15912253::INFO::2013-11-20 02:24:27,852::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912254::DEBUG::2013-11-20 02:24:28,870::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912254::INFO::2013-11-20 02:24:28,871::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912254::DEBUG::2013-11-20 02:24:28,872::task::495::TaskManager.Task::(_debug) Task a66c3c2b-6890-48c3-917d-d6c0e80a9222: moving from state init -> state preparing >Thread-15912254::DEBUG::2013-11-20 02:24:28,872::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912254::DEBUG::2013-11-20 02:24:28,873::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912254::DEBUG::2013-11-20 02:24:28,874::task::495::TaskManager.Task::(_debug) Task a66c3c2b-6890-48c3-917d-d6c0e80a9222: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912254::DEBUG::2013-11-20 02:24:28,874::task::495::TaskManager.Task::(_debug) Task a66c3c2b-6890-48c3-917d-d6c0e80a9222: moving from state preparing -> state finished >Thread-15912254::DEBUG::2013-11-20 02:24:28,875::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912254::DEBUG::2013-11-20 02:24:28,875::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912254::DEBUG::2013-11-20 02:24:28,878::task::495::TaskManager.Task::(_debug) Task a66c3c2b-6890-48c3-917d-d6c0e80a9222: ref 0 aborting False >Thread-15912254::INFO::2013-11-20 02:24:28,880::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912256::DEBUG::2013-11-20 02:24:29,898::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912256::INFO::2013-11-20 02:24:29,899::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912256::DEBUG::2013-11-20 02:24:29,900::task::495::TaskManager.Task::(_debug) Task 49549051-9435-44b8-b3a7-ec774e6978a6: moving from state init -> state preparing >Thread-15912256::DEBUG::2013-11-20 02:24:29,900::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912256::DEBUG::2013-11-20 02:24:29,901::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912256::DEBUG::2013-11-20 02:24:29,902::task::495::TaskManager.Task::(_debug) Task 49549051-9435-44b8-b3a7-ec774e6978a6: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912256::DEBUG::2013-11-20 02:24:29,902::task::495::TaskManager.Task::(_debug) Task 49549051-9435-44b8-b3a7-ec774e6978a6: moving from state preparing -> state finished >Thread-15912256::DEBUG::2013-11-20 02:24:29,903::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912256::DEBUG::2013-11-20 02:24:29,903::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912256::DEBUG::2013-11-20 02:24:29,904::task::495::TaskManager.Task::(_debug) Task 49549051-9435-44b8-b3a7-ec774e6978a6: ref 0 aborting False >Thread-15912256::INFO::2013-11-20 02:24:29,904::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912257::DEBUG::2013-11-20 02:24:30,921::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912257::INFO::2013-11-20 02:24:30,922::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912257::DEBUG::2013-11-20 02:24:30,922::task::495::TaskManager.Task::(_debug) Task 2b1921fc-c669-41d4-9411-be82d1e09264: moving from state init -> state preparing >Thread-15912257::DEBUG::2013-11-20 02:24:30,923::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912257::DEBUG::2013-11-20 02:24:30,923::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912257::DEBUG::2013-11-20 02:24:30,924::task::495::TaskManager.Task::(_debug) Task 2b1921fc-c669-41d4-9411-be82d1e09264: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912257::DEBUG::2013-11-20 02:24:30,925::task::495::TaskManager.Task::(_debug) Task 2b1921fc-c669-41d4-9411-be82d1e09264: moving from state preparing -> state finished >Thread-15912257::DEBUG::2013-11-20 02:24:30,925::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912257::DEBUG::2013-11-20 02:24:30,926::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912257::DEBUG::2013-11-20 02:24:30,926::task::495::TaskManager.Task::(_debug) Task 2b1921fc-c669-41d4-9411-be82d1e09264: ref 0 aborting False >Thread-15912257::INFO::2013-11-20 02:24:30,927::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912258::DEBUG::2013-11-20 02:24:31,942::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912258::INFO::2013-11-20 02:24:31,943::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912258::DEBUG::2013-11-20 02:24:31,955::task::495::TaskManager.Task::(_debug) Task 6c067dae-f0c0-4ffd-bf43-7f540bcc99e2: moving from state init -> state preparing >Thread-15912258::DEBUG::2013-11-20 02:24:31,956::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912258::DEBUG::2013-11-20 02:24:31,956::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912258::DEBUG::2013-11-20 02:24:31,957::task::495::TaskManager.Task::(_debug) Task 6c067dae-f0c0-4ffd-bf43-7f540bcc99e2: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912258::DEBUG::2013-11-20 02:24:31,957::task::495::TaskManager.Task::(_debug) Task 6c067dae-f0c0-4ffd-bf43-7f540bcc99e2: moving from state preparing -> state finished >Thread-15912258::DEBUG::2013-11-20 02:24:31,958::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912258::DEBUG::2013-11-20 02:24:31,958::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912258::DEBUG::2013-11-20 02:24:31,959::task::495::TaskManager.Task::(_debug) Task 6c067dae-f0c0-4ffd-bf43-7f540bcc99e2: ref 0 aborting False >Thread-15912258::INFO::2013-11-20 02:24:31,959::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912260::DEBUG::2013-11-20 02:24:32,975::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912260::INFO::2013-11-20 02:24:32,976::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912260::DEBUG::2013-11-20 02:24:32,976::task::495::TaskManager.Task::(_debug) Task 6840ef92-ae04-4583-81b6-7a9dcc8792cd: moving from state init -> state preparing >Thread-15912260::DEBUG::2013-11-20 02:24:32,977::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912260::DEBUG::2013-11-20 02:24:32,977::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912260::DEBUG::2013-11-20 02:24:32,978::task::495::TaskManager.Task::(_debug) Task 6840ef92-ae04-4583-81b6-7a9dcc8792cd: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912260::DEBUG::2013-11-20 02:24:32,979::task::495::TaskManager.Task::(_debug) Task 6840ef92-ae04-4583-81b6-7a9dcc8792cd: moving from state preparing -> state finished >Thread-15912260::DEBUG::2013-11-20 02:24:32,979::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912260::DEBUG::2013-11-20 02:24:32,980::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912260::DEBUG::2013-11-20 02:24:32,980::task::495::TaskManager.Task::(_debug) Task 6840ef92-ae04-4583-81b6-7a9dcc8792cd: ref 0 aborting False >Thread-15912260::INFO::2013-11-20 02:24:32,981::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-1024767::ERROR::2013-11-20 02:24:33,131::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912261::DEBUG::2013-11-20 02:24:34,000::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912261::INFO::2013-11-20 02:24:34,001::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912261::DEBUG::2013-11-20 02:24:34,002::task::495::TaskManager.Task::(_debug) Task 6bcdc9fd-29e7-4ba8-987a-5ee632d1ecf7: moving from state init -> state preparing >Thread-15912261::DEBUG::2013-11-20 02:24:34,002::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912261::DEBUG::2013-11-20 02:24:34,003::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912261::DEBUG::2013-11-20 02:24:34,004::task::495::TaskManager.Task::(_debug) Task 6bcdc9fd-29e7-4ba8-987a-5ee632d1ecf7: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912261::DEBUG::2013-11-20 02:24:34,004::task::495::TaskManager.Task::(_debug) Task 6bcdc9fd-29e7-4ba8-987a-5ee632d1ecf7: moving from state preparing -> state finished >Thread-15912261::DEBUG::2013-11-20 02:24:34,005::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912261::DEBUG::2013-11-20 02:24:34,005::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912261::DEBUG::2013-11-20 02:24:34,006::task::495::TaskManager.Task::(_debug) Task 6bcdc9fd-29e7-4ba8-987a-5ee632d1ecf7: ref 0 aborting False >Thread-15912261::INFO::2013-11-20 02:24:34,006::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912262::DEBUG::2013-11-20 02:24:35,022::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912262::INFO::2013-11-20 02:24:35,023::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912262::DEBUG::2013-11-20 02:24:35,024::task::495::TaskManager.Task::(_debug) Task 309ac11c-8b36-4a27-b512-b5b09bb83254: moving from state init -> state preparing >Thread-15912262::DEBUG::2013-11-20 02:24:35,024::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912262::DEBUG::2013-11-20 02:24:35,025::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912262::DEBUG::2013-11-20 02:24:35,025::task::495::TaskManager.Task::(_debug) Task 309ac11c-8b36-4a27-b512-b5b09bb83254: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912262::DEBUG::2013-11-20 02:24:35,026::task::495::TaskManager.Task::(_debug) Task 309ac11c-8b36-4a27-b512-b5b09bb83254: moving from state preparing -> state finished >Thread-15912262::DEBUG::2013-11-20 02:24:35,027::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912262::DEBUG::2013-11-20 02:24:35,027::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912262::DEBUG::2013-11-20 02:24:35,028::task::495::TaskManager.Task::(_debug) Task 309ac11c-8b36-4a27-b512-b5b09bb83254: ref 0 aborting False >Thread-15912262::INFO::2013-11-20 02:24:35,028::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912263::INFO::2013-11-20 02:24:35,214::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912263::DEBUG::2013-11-20 02:24:35,214::task::495::TaskManager.Task::(_debug) Task 65e28958-3e92-4ea9-b539-3264a396bb93: moving from state init -> state preparing >Thread-15912263::DEBUG::2013-11-20 02:24:35,215::task::495::TaskManager.Task::(_debug) Task 65e28958-3e92-4ea9-b539-3264a396bb93: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000334024429321', 'lastCheck': 1384914267.480865, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00227499008179', 'lastCheck': 1384914271.7418101, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384914273.132628, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000362157821655', 'lastCheck': 1384914265.8027561, 'valid': True, 'code': 0}} >Thread-15912263::DEBUG::2013-11-20 02:24:35,219::task::495::TaskManager.Task::(_debug) Task 65e28958-3e92-4ea9-b539-3264a396bb93: moving from state preparing -> state finished >Thread-15912263::DEBUG::2013-11-20 02:24:35,219::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912263::DEBUG::2013-11-20 02:24:35,220::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912263::DEBUG::2013-11-20 02:24:35,221::task::495::TaskManager.Task::(_debug) Task 65e28958-3e92-4ea9-b539-3264a396bb93: ref 0 aborting False >Thread-15912263::INFO::2013-11-20 02:24:35,221::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000334024429321', 'lastCheck': 1384914267.480865, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00227499008179', 'lastCheck': 1384914271.7418101, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384914273.132628, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000362157821655', 'lastCheck': 1384914265.8027561, 'valid': True, 'code': 0}} >Thread-15912265::DEBUG::2013-11-20 02:24:36,046::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912265::INFO::2013-11-20 02:24:36,049::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912265::DEBUG::2013-11-20 02:24:36,049::task::495::TaskManager.Task::(_debug) Task bd8b8521-c9ba-4a81-8f0f-875ce2dfc3fe: moving from state init -> state preparing >Thread-15912265::DEBUG::2013-11-20 02:24:36,050::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912265::DEBUG::2013-11-20 02:24:36,051::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912265::DEBUG::2013-11-20 02:24:36,052::task::495::TaskManager.Task::(_debug) Task bd8b8521-c9ba-4a81-8f0f-875ce2dfc3fe: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912265::DEBUG::2013-11-20 02:24:36,053::task::495::TaskManager.Task::(_debug) Task bd8b8521-c9ba-4a81-8f0f-875ce2dfc3fe: moving from state preparing -> state finished >Thread-15912265::DEBUG::2013-11-20 02:24:36,054::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912265::DEBUG::2013-11-20 02:24:36,055::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912265::DEBUG::2013-11-20 02:24:36,056::task::495::TaskManager.Task::(_debug) Task bd8b8521-c9ba-4a81-8f0f-875ce2dfc3fe: ref 0 aborting False >Thread-15912265::INFO::2013-11-20 02:24:36,057::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912266::DEBUG::2013-11-20 02:24:37,091::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912266::INFO::2013-11-20 02:24:37,092::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912266::DEBUG::2013-11-20 02:24:37,104::task::495::TaskManager.Task::(_debug) Task e6c42770-bd6c-4434-97e6-85981917707a: moving from state init -> state preparing >Thread-15912266::DEBUG::2013-11-20 02:24:37,105::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912266::DEBUG::2013-11-20 02:24:37,105::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912266::DEBUG::2013-11-20 02:24:37,106::task::495::TaskManager.Task::(_debug) Task e6c42770-bd6c-4434-97e6-85981917707a: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912266::DEBUG::2013-11-20 02:24:37,107::task::495::TaskManager.Task::(_debug) Task e6c42770-bd6c-4434-97e6-85981917707a: moving from state preparing -> state finished >Thread-15912266::DEBUG::2013-11-20 02:24:37,107::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912266::DEBUG::2013-11-20 02:24:37,108::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912266::DEBUG::2013-11-20 02:24:37,108::task::495::TaskManager.Task::(_debug) Task e6c42770-bd6c-4434-97e6-85981917707a: ref 0 aborting False >Thread-15912266::INFO::2013-11-20 02:24:37,109::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912267::DEBUG::2013-11-20 02:24:38,125::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912267::INFO::2013-11-20 02:24:38,125::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912267::DEBUG::2013-11-20 02:24:38,126::task::495::TaskManager.Task::(_debug) Task 935199fb-db37-43c9-aed6-4e61f6cec6b2: moving from state init -> state preparing >Thread-15912267::DEBUG::2013-11-20 02:24:38,127::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912267::DEBUG::2013-11-20 02:24:38,127::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912267::DEBUG::2013-11-20 02:24:38,128::task::495::TaskManager.Task::(_debug) Task 935199fb-db37-43c9-aed6-4e61f6cec6b2: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912267::DEBUG::2013-11-20 02:24:38,128::task::495::TaskManager.Task::(_debug) Task 935199fb-db37-43c9-aed6-4e61f6cec6b2: moving from state preparing -> state finished >Thread-15912267::DEBUG::2013-11-20 02:24:38,129::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912267::DEBUG::2013-11-20 02:24:38,129::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912267::DEBUG::2013-11-20 02:24:38,130::task::495::TaskManager.Task::(_debug) Task 935199fb-db37-43c9-aed6-4e61f6cec6b2: ref 0 aborting False >Thread-15912267::INFO::2013-11-20 02:24:38,130::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912269::DEBUG::2013-11-20 02:24:39,146::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912269::INFO::2013-11-20 02:24:39,147::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912269::DEBUG::2013-11-20 02:24:39,148::task::495::TaskManager.Task::(_debug) Task c88ce247-5d55-44e4-905f-100e1e70b18d: moving from state init -> state preparing >Thread-15912269::DEBUG::2013-11-20 02:24:39,148::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912269::DEBUG::2013-11-20 02:24:39,149::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912269::DEBUG::2013-11-20 02:24:39,150::task::495::TaskManager.Task::(_debug) Task c88ce247-5d55-44e4-905f-100e1e70b18d: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912269::DEBUG::2013-11-20 02:24:39,150::task::495::TaskManager.Task::(_debug) Task c88ce247-5d55-44e4-905f-100e1e70b18d: moving from state preparing -> state finished >Thread-15912269::DEBUG::2013-11-20 02:24:39,151::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912269::DEBUG::2013-11-20 02:24:39,151::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912269::DEBUG::2013-11-20 02:24:39,152::task::495::TaskManager.Task::(_debug) Task c88ce247-5d55-44e4-905f-100e1e70b18d: ref 0 aborting False >Thread-15912269::INFO::2013-11-20 02:24:39,152::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912270::DEBUG::2013-11-20 02:24:40,168::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912270::INFO::2013-11-20 02:24:40,169::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912270::DEBUG::2013-11-20 02:24:40,170::task::495::TaskManager.Task::(_debug) Task 2ea8fc1c-dec6-45d0-91b6-52e3688a1b01: moving from state init -> state preparing >Thread-15912270::DEBUG::2013-11-20 02:24:40,170::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912270::DEBUG::2013-11-20 02:24:40,171::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912270::DEBUG::2013-11-20 02:24:40,171::task::495::TaskManager.Task::(_debug) Task 2ea8fc1c-dec6-45d0-91b6-52e3688a1b01: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912270::DEBUG::2013-11-20 02:24:40,172::task::495::TaskManager.Task::(_debug) Task 2ea8fc1c-dec6-45d0-91b6-52e3688a1b01: moving from state preparing -> state finished >Thread-15912270::DEBUG::2013-11-20 02:24:40,172::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912270::DEBUG::2013-11-20 02:24:40,173::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912270::DEBUG::2013-11-20 02:24:40,174::task::495::TaskManager.Task::(_debug) Task 2ea8fc1c-dec6-45d0-91b6-52e3688a1b01: ref 0 aborting False >Thread-15912270::INFO::2013-11-20 02:24:40,174::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912271::DEBUG::2013-11-20 02:24:41,194::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912271::INFO::2013-11-20 02:24:41,195::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912271::DEBUG::2013-11-20 02:24:41,195::task::495::TaskManager.Task::(_debug) Task 48151d9f-c225-4d09-b8a0-341975f07787: moving from state init -> state preparing >Thread-15912271::DEBUG::2013-11-20 02:24:41,196::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912271::DEBUG::2013-11-20 02:24:41,196::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912271::DEBUG::2013-11-20 02:24:41,197::task::495::TaskManager.Task::(_debug) Task 48151d9f-c225-4d09-b8a0-341975f07787: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912271::DEBUG::2013-11-20 02:24:41,198::task::495::TaskManager.Task::(_debug) Task 48151d9f-c225-4d09-b8a0-341975f07787: moving from state preparing -> state finished >Thread-15912271::DEBUG::2013-11-20 02:24:41,198::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912271::DEBUG::2013-11-20 02:24:41,199::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912271::DEBUG::2013-11-20 02:24:41,199::task::495::TaskManager.Task::(_debug) Task 48151d9f-c225-4d09-b8a0-341975f07787: ref 0 aborting False >Thread-15912271::INFO::2013-11-20 02:24:41,200::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912273::DEBUG::2013-11-20 02:24:42,215::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912273::INFO::2013-11-20 02:24:42,216::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912273::DEBUG::2013-11-20 02:24:42,217::task::495::TaskManager.Task::(_debug) Task 280c5ea7-0281-4afb-ae5a-ddfba21eb14b: moving from state init -> state preparing >Thread-15912273::DEBUG::2013-11-20 02:24:42,217::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912273::DEBUG::2013-11-20 02:24:42,218::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912273::DEBUG::2013-11-20 02:24:42,219::task::495::TaskManager.Task::(_debug) Task 280c5ea7-0281-4afb-ae5a-ddfba21eb14b: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912273::DEBUG::2013-11-20 02:24:42,219::task::495::TaskManager.Task::(_debug) Task 280c5ea7-0281-4afb-ae5a-ddfba21eb14b: moving from state preparing -> state finished >Thread-15912273::DEBUG::2013-11-20 02:24:42,220::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912273::DEBUG::2013-11-20 02:24:42,220::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912273::DEBUG::2013-11-20 02:24:42,221::task::495::TaskManager.Task::(_debug) Task 280c5ea7-0281-4afb-ae5a-ddfba21eb14b: ref 0 aborting False >Thread-15912273::INFO::2013-11-20 02:24:42,221::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-1024767::ERROR::2013-11-20 02:24:43,145::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912274::DEBUG::2013-11-20 02:24:43,239::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912274::INFO::2013-11-20 02:24:43,240::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912274::DEBUG::2013-11-20 02:24:43,241::task::495::TaskManager.Task::(_debug) Task 70615572-118e-4228-ad0d-7e329b526fea: moving from state init -> state preparing >Thread-15912274::DEBUG::2013-11-20 02:24:43,241::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912274::DEBUG::2013-11-20 02:24:43,242::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912274::DEBUG::2013-11-20 02:24:43,243::task::495::TaskManager.Task::(_debug) Task 70615572-118e-4228-ad0d-7e329b526fea: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912274::DEBUG::2013-11-20 02:24:43,243::task::495::TaskManager.Task::(_debug) Task 70615572-118e-4228-ad0d-7e329b526fea: moving from state preparing -> state finished >Thread-15912274::DEBUG::2013-11-20 02:24:43,244::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912274::DEBUG::2013-11-20 02:24:43,244::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912274::DEBUG::2013-11-20 02:24:43,245::task::495::TaskManager.Task::(_debug) Task 70615572-118e-4228-ad0d-7e329b526fea: ref 0 aborting False >Thread-15912274::INFO::2013-11-20 02:24:43,245::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912275::DEBUG::2013-11-20 02:24:44,261::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912275::INFO::2013-11-20 02:24:44,262::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912275::DEBUG::2013-11-20 02:24:44,263::task::495::TaskManager.Task::(_debug) Task 849c03ee-6c79-4e3a-9f02-57f6a697bfda: moving from state init -> state preparing >Thread-15912275::DEBUG::2013-11-20 02:24:44,263::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912275::DEBUG::2013-11-20 02:24:44,264::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912275::DEBUG::2013-11-20 02:24:44,264::task::495::TaskManager.Task::(_debug) Task 849c03ee-6c79-4e3a-9f02-57f6a697bfda: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912275::DEBUG::2013-11-20 02:24:44,265::task::495::TaskManager.Task::(_debug) Task 849c03ee-6c79-4e3a-9f02-57f6a697bfda: moving from state preparing -> state finished >Thread-15912275::DEBUG::2013-11-20 02:24:44,266::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912275::DEBUG::2013-11-20 02:24:44,266::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912275::DEBUG::2013-11-20 02:24:44,267::task::495::TaskManager.Task::(_debug) Task 849c03ee-6c79-4e3a-9f02-57f6a697bfda: ref 0 aborting False >Thread-15912275::INFO::2013-11-20 02:24:44,267::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912277::DEBUG::2013-11-20 02:24:45,283::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912277::INFO::2013-11-20 02:24:45,284::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912277::DEBUG::2013-11-20 02:24:45,285::task::495::TaskManager.Task::(_debug) Task 47295853-bdb2-4d50-b02c-1ff290b06197: moving from state init -> state preparing >Thread-15912277::DEBUG::2013-11-20 02:24:45,285::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912277::DEBUG::2013-11-20 02:24:45,286::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912277::DEBUG::2013-11-20 02:24:45,286::task::495::TaskManager.Task::(_debug) Task 47295853-bdb2-4d50-b02c-1ff290b06197: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912277::DEBUG::2013-11-20 02:24:45,287::task::495::TaskManager.Task::(_debug) Task 47295853-bdb2-4d50-b02c-1ff290b06197: moving from state preparing -> state finished >Thread-15912277::DEBUG::2013-11-20 02:24:45,288::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912277::DEBUG::2013-11-20 02:24:45,288::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912277::DEBUG::2013-11-20 02:24:45,289::task::495::TaskManager.Task::(_debug) Task 47295853-bdb2-4d50-b02c-1ff290b06197: ref 0 aborting False >Thread-15912277::INFO::2013-11-20 02:24:45,289::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912278::DEBUG::2013-11-20 02:24:46,305::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912278::INFO::2013-11-20 02:24:46,306::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912278::DEBUG::2013-11-20 02:24:46,307::task::495::TaskManager.Task::(_debug) Task ba6296fa-8eb2-451c-869b-7c742722fe38: moving from state init -> state preparing >Thread-15912278::DEBUG::2013-11-20 02:24:46,307::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912278::DEBUG::2013-11-20 02:24:46,308::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912278::DEBUG::2013-11-20 02:24:46,308::task::495::TaskManager.Task::(_debug) Task ba6296fa-8eb2-451c-869b-7c742722fe38: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912278::DEBUG::2013-11-20 02:24:46,309::task::495::TaskManager.Task::(_debug) Task ba6296fa-8eb2-451c-869b-7c742722fe38: moving from state preparing -> state finished >Thread-15912278::DEBUG::2013-11-20 02:24:46,309::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912278::DEBUG::2013-11-20 02:24:46,310::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912278::DEBUG::2013-11-20 02:24:46,310::task::495::TaskManager.Task::(_debug) Task ba6296fa-8eb2-451c-869b-7c742722fe38: ref 0 aborting False >Thread-15912278::INFO::2013-11-20 02:24:46,311::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912279::DEBUG::2013-11-20 02:24:47,327::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912279::INFO::2013-11-20 02:24:47,327::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912279::DEBUG::2013-11-20 02:24:47,338::task::495::TaskManager.Task::(_debug) Task ec92d3c8-33bf-425d-989c-6aea6ed1e9a5: moving from state init -> state preparing >Thread-15912279::DEBUG::2013-11-20 02:24:47,339::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912279::DEBUG::2013-11-20 02:24:47,340::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912279::DEBUG::2013-11-20 02:24:47,340::task::495::TaskManager.Task::(_debug) Task ec92d3c8-33bf-425d-989c-6aea6ed1e9a5: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912279::DEBUG::2013-11-20 02:24:47,341::task::495::TaskManager.Task::(_debug) Task ec92d3c8-33bf-425d-989c-6aea6ed1e9a5: moving from state preparing -> state finished >Thread-15912279::DEBUG::2013-11-20 02:24:47,341::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912279::DEBUG::2013-11-20 02:24:47,342::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912279::DEBUG::2013-11-20 02:24:47,342::task::495::TaskManager.Task::(_debug) Task ec92d3c8-33bf-425d-989c-6aea6ed1e9a5: ref 0 aborting False >Thread-15912279::INFO::2013-11-20 02:24:47,343::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912281::DEBUG::2013-11-20 02:24:48,358::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912281::INFO::2013-11-20 02:24:48,359::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912281::DEBUG::2013-11-20 02:24:48,360::task::495::TaskManager.Task::(_debug) Task 97ab8a29-8039-42bf-abe3-c6bbebbe45db: moving from state init -> state preparing >Thread-15912281::DEBUG::2013-11-20 02:24:48,360::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912281::DEBUG::2013-11-20 02:24:48,361::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912281::DEBUG::2013-11-20 02:24:48,362::task::495::TaskManager.Task::(_debug) Task 97ab8a29-8039-42bf-abe3-c6bbebbe45db: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912281::DEBUG::2013-11-20 02:24:48,362::task::495::TaskManager.Task::(_debug) Task 97ab8a29-8039-42bf-abe3-c6bbebbe45db: moving from state preparing -> state finished >Thread-15912281::DEBUG::2013-11-20 02:24:48,363::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912281::DEBUG::2013-11-20 02:24:48,363::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912281::DEBUG::2013-11-20 02:24:48,364::task::495::TaskManager.Task::(_debug) Task 97ab8a29-8039-42bf-abe3-c6bbebbe45db: ref 0 aborting False >Thread-15912281::INFO::2013-11-20 02:24:48,364::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912282::DEBUG::2013-11-20 02:24:49,380::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912282::INFO::2013-11-20 02:24:49,381::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912282::DEBUG::2013-11-20 02:24:49,382::task::495::TaskManager.Task::(_debug) Task 95ef8e57-950b-430a-b5b1-299343041332: moving from state init -> state preparing >Thread-15912282::DEBUG::2013-11-20 02:24:49,382::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912282::DEBUG::2013-11-20 02:24:49,383::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912282::DEBUG::2013-11-20 02:24:49,383::task::495::TaskManager.Task::(_debug) Task 95ef8e57-950b-430a-b5b1-299343041332: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912282::DEBUG::2013-11-20 02:24:49,384::task::495::TaskManager.Task::(_debug) Task 95ef8e57-950b-430a-b5b1-299343041332: moving from state preparing -> state finished >Thread-15912282::DEBUG::2013-11-20 02:24:49,385::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912282::DEBUG::2013-11-20 02:24:49,385::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912282::DEBUG::2013-11-20 02:24:49,386::task::495::TaskManager.Task::(_debug) Task 95ef8e57-950b-430a-b5b1-299343041332: ref 0 aborting False >Thread-15912282::INFO::2013-11-20 02:24:49,386::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912283::DEBUG::2013-11-20 02:24:50,402::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912283::INFO::2013-11-20 02:24:50,402::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912283::DEBUG::2013-11-20 02:24:50,403::task::495::TaskManager.Task::(_debug) Task ebdf7d62-c332-4b45-86fb-d40227ae4a90: moving from state init -> state preparing >Thread-15912283::DEBUG::2013-11-20 02:24:50,403::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912283::DEBUG::2013-11-20 02:24:50,404::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912283::DEBUG::2013-11-20 02:24:50,405::task::495::TaskManager.Task::(_debug) Task ebdf7d62-c332-4b45-86fb-d40227ae4a90: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912283::DEBUG::2013-11-20 02:24:50,405::task::495::TaskManager.Task::(_debug) Task ebdf7d62-c332-4b45-86fb-d40227ae4a90: moving from state preparing -> state finished >Thread-15912283::DEBUG::2013-11-20 02:24:50,406::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912283::DEBUG::2013-11-20 02:24:50,406::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912283::DEBUG::2013-11-20 02:24:50,407::task::495::TaskManager.Task::(_debug) Task ebdf7d62-c332-4b45-86fb-d40227ae4a90: ref 0 aborting False >Thread-15912283::INFO::2013-11-20 02:24:50,407::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912284::INFO::2013-11-20 02:24:50,431::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912284::DEBUG::2013-11-20 02:24:50,431::task::495::TaskManager.Task::(_debug) Task 0a7413ae-1464-4319-8d92-3ec2960ea7ec: moving from state init -> state preparing >Thread-15912284::DEBUG::2013-11-20 02:24:50,432::task::495::TaskManager.Task::(_debug) Task 0a7413ae-1464-4319-8d92-3ec2960ea7ec: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000280141830444', 'lastCheck': 1384914287.5072241, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00153803825378', 'lastCheck': 1384914281.7575071, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384914283.1459229, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000362873077393', 'lastCheck': 1384914285.825582, 'valid': True, 'code': 0}} >Thread-15912284::DEBUG::2013-11-20 02:24:50,443::task::495::TaskManager.Task::(_debug) Task 0a7413ae-1464-4319-8d92-3ec2960ea7ec: moving from state preparing -> state finished >Thread-15912284::DEBUG::2013-11-20 02:24:50,443::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912284::DEBUG::2013-11-20 02:24:50,444::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912284::DEBUG::2013-11-20 02:24:50,444::task::495::TaskManager.Task::(_debug) Task 0a7413ae-1464-4319-8d92-3ec2960ea7ec: ref 0 aborting False >Thread-15912284::INFO::2013-11-20 02:24:50,445::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000280141830444', 'lastCheck': 1384914287.5072241, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00153803825378', 'lastCheck': 1384914281.7575071, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0', 'lastCheck': 1384914283.1459229, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000362873077393', 'lastCheck': 1384914285.825582, 'valid': True, 'code': 0}} >Thread-15912286::DEBUG::2013-11-20 02:24:51,465::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912286::INFO::2013-11-20 02:24:51,466::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912286::DEBUG::2013-11-20 02:24:51,467::task::495::TaskManager.Task::(_debug) Task 9ded8c59-9984-452e-bade-c6939ce25bf3: moving from state init -> state preparing >Thread-15912286::DEBUG::2013-11-20 02:24:51,467::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912286::DEBUG::2013-11-20 02:24:51,468::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912286::DEBUG::2013-11-20 02:24:51,469::task::495::TaskManager.Task::(_debug) Task 9ded8c59-9984-452e-bade-c6939ce25bf3: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912286::DEBUG::2013-11-20 02:24:51,469::task::495::TaskManager.Task::(_debug) Task 9ded8c59-9984-452e-bade-c6939ce25bf3: moving from state preparing -> state finished >Thread-15912286::DEBUG::2013-11-20 02:24:51,470::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912286::DEBUG::2013-11-20 02:24:51,470::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912286::DEBUG::2013-11-20 02:24:51,471::task::495::TaskManager.Task::(_debug) Task 9ded8c59-9984-452e-bade-c6939ce25bf3: ref 0 aborting False >Thread-15912286::INFO::2013-11-20 02:24:51,471::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912287::DEBUG::2013-11-20 02:24:52,486::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912287::INFO::2013-11-20 02:24:52,487::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912287::DEBUG::2013-11-20 02:24:52,495::task::495::TaskManager.Task::(_debug) Task 81828ba9-dfae-4b2d-bfb2-d9a70d7984d3: moving from state init -> state preparing >Thread-15912287::DEBUG::2013-11-20 02:24:52,495::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912287::DEBUG::2013-11-20 02:24:52,496::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912287::DEBUG::2013-11-20 02:24:52,496::task::495::TaskManager.Task::(_debug) Task 81828ba9-dfae-4b2d-bfb2-d9a70d7984d3: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912287::DEBUG::2013-11-20 02:24:52,497::task::495::TaskManager.Task::(_debug) Task 81828ba9-dfae-4b2d-bfb2-d9a70d7984d3: moving from state preparing -> state finished >Thread-15912287::DEBUG::2013-11-20 02:24:52,498::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912287::DEBUG::2013-11-20 02:24:52,498::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912287::DEBUG::2013-11-20 02:24:52,499::task::495::TaskManager.Task::(_debug) Task 81828ba9-dfae-4b2d-bfb2-d9a70d7984d3: ref 0 aborting False >Thread-15912287::INFO::2013-11-20 02:24:52,499::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-1024767::ERROR::2013-11-20 02:24:53,158::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` ([Errno 2] No such file or directory: '/rhev/data-center/mnt/kairo.brd-klee.dk:_export/5d7b5178-3790-472c-a13b-b481eb27a61e/dom_md/metadata') >Thread-15912288::DEBUG::2013-11-20 02:24:53,515::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912288::INFO::2013-11-20 02:24:53,516::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912288::DEBUG::2013-11-20 02:24:53,516::task::495::TaskManager.Task::(_debug) Task 499c46e5-41d5-4422-a17f-b008a0619c3d: moving from state init -> state preparing >Thread-15912288::DEBUG::2013-11-20 02:24:53,517::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912288::DEBUG::2013-11-20 02:24:53,517::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912288::DEBUG::2013-11-20 02:24:53,518::task::495::TaskManager.Task::(_debug) Task 499c46e5-41d5-4422-a17f-b008a0619c3d: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912288::DEBUG::2013-11-20 02:24:53,519::task::495::TaskManager.Task::(_debug) Task 499c46e5-41d5-4422-a17f-b008a0619c3d: moving from state preparing -> state finished >Thread-15912288::DEBUG::2013-11-20 02:24:53,519::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912288::DEBUG::2013-11-20 02:24:53,520::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912288::DEBUG::2013-11-20 02:24:53,520::task::495::TaskManager.Task::(_debug) Task 499c46e5-41d5-4422-a17f-b008a0619c3d: ref 0 aborting False >Thread-15912288::INFO::2013-11-20 02:24:53,521::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912290::DEBUG::2013-11-20 02:24:54,536::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912290::INFO::2013-11-20 02:24:54,537::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912290::DEBUG::2013-11-20 02:24:54,538::task::495::TaskManager.Task::(_debug) Task 55bb00d8-52ec-4ef9-a5af-94ed9f57b66f: moving from state init -> state preparing >Thread-15912290::DEBUG::2013-11-20 02:24:54,538::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912290::DEBUG::2013-11-20 02:24:54,539::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912290::DEBUG::2013-11-20 02:24:54,539::task::495::TaskManager.Task::(_debug) Task 55bb00d8-52ec-4ef9-a5af-94ed9f57b66f: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912290::DEBUG::2013-11-20 02:24:54,540::task::495::TaskManager.Task::(_debug) Task 55bb00d8-52ec-4ef9-a5af-94ed9f57b66f: moving from state preparing -> state finished >Thread-15912290::DEBUG::2013-11-20 02:24:54,541::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912290::DEBUG::2013-11-20 02:24:54,541::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912290::DEBUG::2013-11-20 02:24:54,542::task::495::TaskManager.Task::(_debug) Task 55bb00d8-52ec-4ef9-a5af-94ed9f57b66f: ref 0 aborting False >Thread-15912290::INFO::2013-11-20 02:24:54,542::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912291::DEBUG::2013-11-20 02:24:55,558::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912291::INFO::2013-11-20 02:24:55,559::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912291::DEBUG::2013-11-20 02:24:55,560::task::495::TaskManager.Task::(_debug) Task fe9f98e9-ec15-4baf-8554-09e46fee438a: moving from state init -> state preparing >Thread-15912291::DEBUG::2013-11-20 02:24:55,560::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912291::DEBUG::2013-11-20 02:24:55,561::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912291::DEBUG::2013-11-20 02:24:55,561::task::495::TaskManager.Task::(_debug) Task fe9f98e9-ec15-4baf-8554-09e46fee438a: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912291::DEBUG::2013-11-20 02:24:55,562::task::495::TaskManager.Task::(_debug) Task fe9f98e9-ec15-4baf-8554-09e46fee438a: moving from state preparing -> state finished >Thread-15912291::DEBUG::2013-11-20 02:24:55,563::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912291::DEBUG::2013-11-20 02:24:55,563::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912291::DEBUG::2013-11-20 02:24:55,564::task::495::TaskManager.Task::(_debug) Task fe9f98e9-ec15-4baf-8554-09e46fee438a: ref 0 aborting False >Thread-15912291::INFO::2013-11-20 02:24:55,564::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912293::DEBUG::2013-11-20 02:24:56,580::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912293::INFO::2013-11-20 02:24:56,581::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912293::DEBUG::2013-11-20 02:24:56,581::task::495::TaskManager.Task::(_debug) Task 65955b54-f7cb-47c9-8811-476144c80bde: moving from state init -> state preparing >Thread-15912293::DEBUG::2013-11-20 02:24:56,582::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912293::DEBUG::2013-11-20 02:24:56,582::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912293::DEBUG::2013-11-20 02:24:56,583::task::495::TaskManager.Task::(_debug) Task 65955b54-f7cb-47c9-8811-476144c80bde: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912293::DEBUG::2013-11-20 02:24:56,583::task::495::TaskManager.Task::(_debug) Task 65955b54-f7cb-47c9-8811-476144c80bde: moving from state preparing -> state finished >Thread-15912293::DEBUG::2013-11-20 02:24:56,584::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912293::DEBUG::2013-11-20 02:24:56,584::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912293::DEBUG::2013-11-20 02:24:56,585::task::495::TaskManager.Task::(_debug) Task 65955b54-f7cb-47c9-8811-476144c80bde: ref 0 aborting False >Thread-15912293::INFO::2013-11-20 02:24:56,586::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912294::DEBUG::2013-11-20 02:24:57,601::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912294::INFO::2013-11-20 02:24:57,602::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912294::DEBUG::2013-11-20 02:24:57,613::task::495::TaskManager.Task::(_debug) Task 11b3bab9-c008-4bbc-9340-cbe0bae697ed: moving from state init -> state preparing >Thread-15912294::DEBUG::2013-11-20 02:24:57,614::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912294::DEBUG::2013-11-20 02:24:57,614::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912294::DEBUG::2013-11-20 02:24:57,615::task::495::TaskManager.Task::(_debug) Task 11b3bab9-c008-4bbc-9340-cbe0bae697ed: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912294::DEBUG::2013-11-20 02:24:57,615::task::495::TaskManager.Task::(_debug) Task 11b3bab9-c008-4bbc-9340-cbe0bae697ed: moving from state preparing -> state finished >Thread-15912294::DEBUG::2013-11-20 02:24:57,616::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912294::DEBUG::2013-11-20 02:24:57,617::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912294::DEBUG::2013-11-20 02:24:57,617::task::495::TaskManager.Task::(_debug) Task 11b3bab9-c008-4bbc-9340-cbe0bae697ed: ref 0 aborting False >Thread-15912294::INFO::2013-11-20 02:24:57,618::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912295::DEBUG::2013-11-20 02:24:58,633::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912295::INFO::2013-11-20 02:24:58,634::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912295::DEBUG::2013-11-20 02:24:58,634::task::495::TaskManager.Task::(_debug) Task 6b406e14-7fe3-4235-aa39-9de713af0968: moving from state init -> state preparing >Thread-15912295::DEBUG::2013-11-20 02:24:58,635::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912295::DEBUG::2013-11-20 02:24:58,635::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912295::DEBUG::2013-11-20 02:24:58,636::task::495::TaskManager.Task::(_debug) Task 6b406e14-7fe3-4235-aa39-9de713af0968: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912295::DEBUG::2013-11-20 02:24:58,637::task::495::TaskManager.Task::(_debug) Task 6b406e14-7fe3-4235-aa39-9de713af0968: moving from state preparing -> state finished >Thread-15912295::DEBUG::2013-11-20 02:24:58,637::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912295::DEBUG::2013-11-20 02:24:58,638::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912295::DEBUG::2013-11-20 02:24:58,638::task::495::TaskManager.Task::(_debug) Task 6b406e14-7fe3-4235-aa39-9de713af0968: ref 0 aborting False >Thread-15912295::INFO::2013-11-20 02:24:58,639::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912297::DEBUG::2013-11-20 02:24:59,654::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912297::INFO::2013-11-20 02:24:59,655::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912297::DEBUG::2013-11-20 02:24:59,656::task::495::TaskManager.Task::(_debug) Task d47d456f-7856-4473-9d58-29dc8391311a: moving from state init -> state preparing >Thread-15912297::DEBUG::2013-11-20 02:24:59,656::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912297::DEBUG::2013-11-20 02:24:59,657::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912297::DEBUG::2013-11-20 02:24:59,657::task::495::TaskManager.Task::(_debug) Task d47d456f-7856-4473-9d58-29dc8391311a: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912297::DEBUG::2013-11-20 02:24:59,658::task::495::TaskManager.Task::(_debug) Task d47d456f-7856-4473-9d58-29dc8391311a: moving from state preparing -> state finished >Thread-15912297::DEBUG::2013-11-20 02:24:59,658::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912297::DEBUG::2013-11-20 02:24:59,659::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912297::DEBUG::2013-11-20 02:24:59,659::task::495::TaskManager.Task::(_debug) Task d47d456f-7856-4473-9d58-29dc8391311a: ref 0 aborting False >Thread-15912297::INFO::2013-11-20 02:24:59,660::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912298::DEBUG::2013-11-20 02:25:00,247::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912298::INFO::2013-11-20 02:25:00,248::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: validateStorageServerConnection, args: (domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{'connection': 'kairo.brd-klee.dk:/export', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36', 'port': ''}]) >Thread-15912298::DEBUG::2013-11-20 02:25:00,248::task::495::TaskManager.Task::(_debug) Task 9429da2b-2c38-4b6c-8593-9b9fae6aca89: moving from state init -> state preparing >Thread-15912298::INFO::2013-11-20 02:25:00,249::storage_connection::127::Storage.ServerConnection::(validate) Request to validate NFS storage server >Thread-15912298::INFO::2013-11-20 02:25:00,250::storage_connection::53::Storage.ServerConnection::(__validateConnectionParams) conList=[{'connection': 'kairo.brd-klee.dk:/export', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36', 'port': ''}] >Thread-15912298::DEBUG::2013-11-20 02:25:00,254::fileUtils::83::Storage.Misc.excCmd::(mount) '/usr/bin/sudo -n /bin/mount -o soft,timeo=600,retrans=6,nosharecache,vers=3 -t nfs kairo.brd-klee.dk:/export /tmp/tmpWUQ14N' (cwd None) >Thread-15912299::DEBUG::2013-11-20 02:25:00,676::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912299::INFO::2013-11-20 02:25:00,677::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912299::DEBUG::2013-11-20 02:25:00,677::task::495::TaskManager.Task::(_debug) Task 1729bdc2-a130-4258-8b70-9334530fce68: moving from state init -> state preparing >Thread-15912299::DEBUG::2013-11-20 02:25:00,677::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912299::DEBUG::2013-11-20 02:25:00,678::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912299::DEBUG::2013-11-20 02:25:00,678::task::495::TaskManager.Task::(_debug) Task 1729bdc2-a130-4258-8b70-9334530fce68: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912299::DEBUG::2013-11-20 02:25:00,678::task::495::TaskManager.Task::(_debug) Task 1729bdc2-a130-4258-8b70-9334530fce68: moving from state preparing -> state finished >Thread-15912299::DEBUG::2013-11-20 02:25:00,679::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912299::DEBUG::2013-11-20 02:25:00,679::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912299::DEBUG::2013-11-20 02:25:00,679::task::495::TaskManager.Task::(_debug) Task 1729bdc2-a130-4258-8b70-9334530fce68: ref 0 aborting False >Thread-15912299::INFO::2013-11-20 02:25:00,680::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912298::DEBUG::2013-11-20 02:25:01,017::fileUtils::83::Storage.Misc.excCmd::(mount) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912298::DEBUG::2013-11-20 02:25:01,352::fileUtils::112::Storage.Misc.excCmd::(umount) '/usr/bin/sudo -n /bin/umount -t nfs -f /tmp/tmpWUQ14N' (cwd None) >Thread-15912300::DEBUG::2013-11-20 02:25:01,695::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912298::DEBUG::2013-11-20 02:25:01,694::fileUtils::112::Storage.Misc.excCmd::(umount) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912300::INFO::2013-11-20 02:25:01,696::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912300::DEBUG::2013-11-20 02:25:01,697::task::495::TaskManager.Task::(_debug) Task f96e9d8d-cdca-4c98-b0ce-ac4e1f5cf2d1: moving from state init -> state preparing >Thread-15912300::DEBUG::2013-11-20 02:25:01,698::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912298::DEBUG::2013-11-20 02:25:01,698::task::495::TaskManager.Task::(_debug) Task 9429da2b-2c38-4b6c-8593-9b9fae6aca89: finished: {'statuslist': [{'status': 0, 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36'}]} >Thread-15912300::DEBUG::2013-11-20 02:25:01,698::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912298::DEBUG::2013-11-20 02:25:01,699::task::495::TaskManager.Task::(_debug) Task 9429da2b-2c38-4b6c-8593-9b9fae6aca89: moving from state preparing -> state finished >Thread-15912300::DEBUG::2013-11-20 02:25:01,699::task::495::TaskManager.Task::(_debug) Task f96e9d8d-cdca-4c98-b0ce-ac4e1f5cf2d1: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912298::DEBUG::2013-11-20 02:25:01,700::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912300::DEBUG::2013-11-20 02:25:01,700::task::495::TaskManager.Task::(_debug) Task f96e9d8d-cdca-4c98-b0ce-ac4e1f5cf2d1: moving from state preparing -> state finished >Thread-15912298::DEBUG::2013-11-20 02:25:01,701::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912300::DEBUG::2013-11-20 02:25:01,701::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912298::DEBUG::2013-11-20 02:25:01,701::task::495::TaskManager.Task::(_debug) Task 9429da2b-2c38-4b6c-8593-9b9fae6aca89: ref 0 aborting False >Thread-15912300::DEBUG::2013-11-20 02:25:01,702::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912298::INFO::2013-11-20 02:25:01,702::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: validateStorageServerConnection, Return response: {'status': {'message': 'OK', 'code': 0}, 'statuslist': [{'status': 0, 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36'}]} >Thread-15912300::DEBUG::2013-11-20 02:25:01,703::task::495::TaskManager.Task::(_debug) Task f96e9d8d-cdca-4c98-b0ce-ac4e1f5cf2d1: ref 0 aborting False >Thread-15912300::INFO::2013-11-20 02:25:01,709::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912301::DEBUG::2013-11-20 02:25:01,729::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912301::INFO::2013-11-20 02:25:01,730::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: connectStorageServer, args: (domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{'connection': 'kairo.brd-klee.dk:/export', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36', 'port': ''}]) >Thread-15912301::DEBUG::2013-11-20 02:25:01,731::task::495::TaskManager.Task::(_debug) Task e48a8e9b-62e8-4414-9428-86fb4eaeb3a7: moving from state init -> state preparing >Thread-15912301::INFO::2013-11-20 02:25:01,731::storage_connection::95::Storage.ServerConnection::(connect) Request to connect NFS storage server >Thread-15912301::INFO::2013-11-20 02:25:01,732::storage_connection::53::Storage.ServerConnection::(__validateConnectionParams) conList=[{'connection': 'kairo.brd-klee.dk:/export', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36', 'port': ''}] >Thread-15912301::DEBUG::2013-11-20 02:25:01,735::fileUtils::83::Storage.Misc.excCmd::(mount) '/usr/bin/sudo -n /bin/mount -o soft,timeo=600,retrans=6,nosharecache,vers=3 -t nfs kairo.brd-klee.dk:/export /rhev/data-center/mnt/kairo.brd-klee.dk:_export' (cwd None) >Thread-15912301::DEBUG::2013-11-20 02:25:02,125::fileUtils::83::Storage.Misc.excCmd::(mount) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912301::DEBUG::2013-11-20 02:25:02,135::lvm::547::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15912301::DEBUG::2013-11-20 02:25:02,136::lvm::549::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15912301::DEBUG::2013-11-20 02:25:02,137::lvm::559::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15912301::DEBUG::2013-11-20 02:25:02,137::lvm::561::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15912301::DEBUG::2013-11-20 02:25:02,138::lvm::580::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15912301::DEBUG::2013-11-20 02:25:02,139::lvm::582::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15912301::DEBUG::2013-11-20 02:25:02,139::task::495::TaskManager.Task::(_debug) Task e48a8e9b-62e8-4414-9428-86fb4eaeb3a7: finished: {'statuslist': [{'status': 0, 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36'}]} >Thread-15912301::DEBUG::2013-11-20 02:25:02,140::task::495::TaskManager.Task::(_debug) Task e48a8e9b-62e8-4414-9428-86fb4eaeb3a7: moving from state preparing -> state finished >Thread-15912301::DEBUG::2013-11-20 02:25:02,141::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912301::DEBUG::2013-11-20 02:25:02,141::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912301::DEBUG::2013-11-20 02:25:02,142::task::495::TaskManager.Task::(_debug) Task e48a8e9b-62e8-4414-9428-86fb4eaeb3a7: ref 0 aborting False >Thread-15912301::INFO::2013-11-20 02:25:02,142::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: connectStorageServer, Return response: {'status': {'message': 'OK', 'code': 0}, 'statuslist': [{'status': 0, 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36'}]} >Thread-15912303::DEBUG::2013-11-20 02:25:02,724::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912303::INFO::2013-11-20 02:25:02,725::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912303::DEBUG::2013-11-20 02:25:02,726::task::495::TaskManager.Task::(_debug) Task fc393201-a466-4491-af0d-38c5f25b1747: moving from state init -> state preparing >Thread-15912303::DEBUG::2013-11-20 02:25:02,726::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912303::DEBUG::2013-11-20 02:25:02,727::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912303::DEBUG::2013-11-20 02:25:02,728::task::495::TaskManager.Task::(_debug) Task fc393201-a466-4491-af0d-38c5f25b1747: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912303::DEBUG::2013-11-20 02:25:02,728::task::495::TaskManager.Task::(_debug) Task fc393201-a466-4491-af0d-38c5f25b1747: moving from state preparing -> state finished >Thread-15912303::DEBUG::2013-11-20 02:25:02,729::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912303::DEBUG::2013-11-20 02:25:02,729::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912303::DEBUG::2013-11-20 02:25:02,730::task::495::TaskManager.Task::(_debug) Task fc393201-a466-4491-af0d-38c5f25b1747: ref 0 aborting False >Thread-15912303::INFO::2013-11-20 02:25:02,730::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-1024767::ERROR::2013-11-20 02:25:03,187::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912304::DEBUG::2013-11-20 02:25:03,750::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912304::INFO::2013-11-20 02:25:03,751::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912304::DEBUG::2013-11-20 02:25:03,752::task::495::TaskManager.Task::(_debug) Task 54e796b1-8186-4caa-8e4e-c949fe80a01f: moving from state init -> state preparing >Thread-15912304::DEBUG::2013-11-20 02:25:03,752::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912304::DEBUG::2013-11-20 02:25:03,753::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912304::DEBUG::2013-11-20 02:25:03,753::task::495::TaskManager.Task::(_debug) Task 54e796b1-8186-4caa-8e4e-c949fe80a01f: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912304::DEBUG::2013-11-20 02:25:03,754::task::495::TaskManager.Task::(_debug) Task 54e796b1-8186-4caa-8e4e-c949fe80a01f: moving from state preparing -> state finished >Thread-15912304::DEBUG::2013-11-20 02:25:03,754::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912304::DEBUG::2013-11-20 02:25:03,755::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912304::DEBUG::2013-11-20 02:25:03,755::task::495::TaskManager.Task::(_debug) Task 54e796b1-8186-4caa-8e4e-c949fe80a01f: ref 0 aborting False >Thread-15912304::INFO::2013-11-20 02:25:03,756::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912305::DEBUG::2013-11-20 02:25:04,776::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912305::INFO::2013-11-20 02:25:04,778::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912305::DEBUG::2013-11-20 02:25:04,780::task::495::TaskManager.Task::(_debug) Task f882fff1-98d3-4968-be64-76547d8c0e97: moving from state init -> state preparing >Thread-15912305::DEBUG::2013-11-20 02:25:04,781::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912305::DEBUG::2013-11-20 02:25:04,781::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912305::DEBUG::2013-11-20 02:25:04,782::task::495::TaskManager.Task::(_debug) Task f882fff1-98d3-4968-be64-76547d8c0e97: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912305::DEBUG::2013-11-20 02:25:04,783::task::495::TaskManager.Task::(_debug) Task f882fff1-98d3-4968-be64-76547d8c0e97: moving from state preparing -> state finished >Thread-15912305::DEBUG::2013-11-20 02:25:04,785::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912305::DEBUG::2013-11-20 02:25:04,785::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912305::DEBUG::2013-11-20 02:25:04,786::task::495::TaskManager.Task::(_debug) Task f882fff1-98d3-4968-be64-76547d8c0e97: ref 0 aborting False >Thread-15912305::INFO::2013-11-20 02:25:04,787::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912306::INFO::2013-11-20 02:25:05,634::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912306::DEBUG::2013-11-20 02:25:05,635::task::495::TaskManager.Task::(_debug) Task a5363760-d2e1-4e78-9017-1bffe30bcef9: moving from state init -> state preparing >Thread-15912306::DEBUG::2013-11-20 02:25:05,635::task::495::TaskManager.Task::(_debug) Task a5363760-d2e1-4e78-9017-1bffe30bcef9: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000324010848999', 'lastCheck': 1384914297.5202911, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148391723633', 'lastCheck': 1384914301.7890749, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0251250267029', 'lastCheck': 1384914303.2140191, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000370025634766', 'lastCheck': 1384914295.8369949, 'valid': True, 'code': 0}} >Thread-15912306::DEBUG::2013-11-20 02:25:05,636::task::495::TaskManager.Task::(_debug) Task a5363760-d2e1-4e78-9017-1bffe30bcef9: moving from state preparing -> state finished >Thread-15912306::DEBUG::2013-11-20 02:25:05,636::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912306::DEBUG::2013-11-20 02:25:05,637::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912306::DEBUG::2013-11-20 02:25:05,638::task::495::TaskManager.Task::(_debug) Task a5363760-d2e1-4e78-9017-1bffe30bcef9: ref 0 aborting False >Thread-15912306::INFO::2013-11-20 02:25:05,638::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000324010848999', 'lastCheck': 1384914297.5202911, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148391723633', 'lastCheck': 1384914301.7890749, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0251250267029', 'lastCheck': 1384914303.2140191, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000370025634766', 'lastCheck': 1384914295.8369949, 'valid': True, 'code': 0}} >Thread-15912308::DEBUG::2013-11-20 02:25:05,805::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912308::INFO::2013-11-20 02:25:05,805::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912308::DEBUG::2013-11-20 02:25:05,806::task::495::TaskManager.Task::(_debug) Task 30e5c063-315b-4894-94f6-5f793ac8649d: moving from state init -> state preparing >Thread-15912308::DEBUG::2013-11-20 02:25:05,806::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912308::DEBUG::2013-11-20 02:25:05,807::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912308::DEBUG::2013-11-20 02:25:05,808::task::495::TaskManager.Task::(_debug) Task 30e5c063-315b-4894-94f6-5f793ac8649d: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912308::DEBUG::2013-11-20 02:25:05,808::task::495::TaskManager.Task::(_debug) Task 30e5c063-315b-4894-94f6-5f793ac8649d: moving from state preparing -> state finished >Thread-15912308::DEBUG::2013-11-20 02:25:05,809::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912308::DEBUG::2013-11-20 02:25:05,809::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912308::DEBUG::2013-11-20 02:25:05,810::task::495::TaskManager.Task::(_debug) Task 30e5c063-315b-4894-94f6-5f793ac8649d: ref 0 aborting False >Thread-15912308::INFO::2013-11-20 02:25:05,810::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-42840::DEBUG::2013-11-20 02:25:05,847::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-42840::DEBUG::2013-11-20 02:25:05,854::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 02:25:06,564::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-42840::DEBUG::2013-11-20 02:25:06,570::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15912309::DEBUG::2013-11-20 02:25:06,827::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912309::INFO::2013-11-20 02:25:06,828::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912309::DEBUG::2013-11-20 02:25:06,845::task::495::TaskManager.Task::(_debug) Task 20cf55b2-3a7e-419a-9ceb-e51ed146d4e6: moving from state init -> state preparing >Thread-15912309::DEBUG::2013-11-20 02:25:06,846::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912309::DEBUG::2013-11-20 02:25:06,846::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912309::DEBUG::2013-11-20 02:25:06,847::task::495::TaskManager.Task::(_debug) Task 20cf55b2-3a7e-419a-9ceb-e51ed146d4e6: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912309::DEBUG::2013-11-20 02:25:06,847::task::495::TaskManager.Task::(_debug) Task 20cf55b2-3a7e-419a-9ceb-e51ed146d4e6: moving from state preparing -> state finished >Thread-15912309::DEBUG::2013-11-20 02:25:06,848::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912309::DEBUG::2013-11-20 02:25:06,848::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912309::DEBUG::2013-11-20 02:25:06,849::task::495::TaskManager.Task::(_debug) Task 20cf55b2-3a7e-419a-9ceb-e51ed146d4e6: ref 0 aborting False >Thread-15912309::INFO::2013-11-20 02:25:06,849::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-42838::DEBUG::2013-11-20 02:25:07,530::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-42838::DEBUG::2013-11-20 02:25:07,531::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15912310::DEBUG::2013-11-20 02:25:07,867::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912310::INFO::2013-11-20 02:25:07,869::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912310::DEBUG::2013-11-20 02:25:07,872::task::495::TaskManager.Task::(_debug) Task 7fac510f-67f6-41b1-92fa-a4bfd57e63ab: moving from state init -> state preparing >Thread-15912310::DEBUG::2013-11-20 02:25:07,874::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912310::DEBUG::2013-11-20 02:25:07,875::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912310::DEBUG::2013-11-20 02:25:07,876::task::495::TaskManager.Task::(_debug) Task 7fac510f-67f6-41b1-92fa-a4bfd57e63ab: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912310::DEBUG::2013-11-20 02:25:07,876::task::495::TaskManager.Task::(_debug) Task 7fac510f-67f6-41b1-92fa-a4bfd57e63ab: moving from state preparing -> state finished >Thread-15912310::DEBUG::2013-11-20 02:25:07,877::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912310::DEBUG::2013-11-20 02:25:07,877::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912310::DEBUG::2013-11-20 02:25:07,878::task::495::TaskManager.Task::(_debug) Task 7fac510f-67f6-41b1-92fa-a4bfd57e63ab: ref 0 aborting False >Thread-15912310::INFO::2013-11-20 02:25:07,884::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-42838::DEBUG::2013-11-20 02:25:08,193::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-42838::DEBUG::2013-11-20 02:25:08,196::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15912312::DEBUG::2013-11-20 02:25:08,903::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912312::INFO::2013-11-20 02:25:08,904::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912312::DEBUG::2013-11-20 02:25:08,905::task::495::TaskManager.Task::(_debug) Task d79303c4-4029-46a5-b288-20a9f0467e37: moving from state init -> state preparing >Thread-15912312::DEBUG::2013-11-20 02:25:08,905::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912312::DEBUG::2013-11-20 02:25:08,906::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912312::DEBUG::2013-11-20 02:25:08,906::task::495::TaskManager.Task::(_debug) Task d79303c4-4029-46a5-b288-20a9f0467e37: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912312::DEBUG::2013-11-20 02:25:08,907::task::495::TaskManager.Task::(_debug) Task d79303c4-4029-46a5-b288-20a9f0467e37: moving from state preparing -> state finished >Thread-15912312::DEBUG::2013-11-20 02:25:08,907::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912312::DEBUG::2013-11-20 02:25:08,908::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912312::DEBUG::2013-11-20 02:25:08,908::task::495::TaskManager.Task::(_debug) Task d79303c4-4029-46a5-b288-20a9f0467e37: ref 0 aborting False >Thread-15912312::INFO::2013-11-20 02:25:08,909::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912313::DEBUG::2013-11-20 02:25:09,924::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912313::INFO::2013-11-20 02:25:09,925::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912313::DEBUG::2013-11-20 02:25:09,925::task::495::TaskManager.Task::(_debug) Task ac8f9089-6df9-4fb4-aabf-4dd8137c94c9: moving from state init -> state preparing >Thread-15912313::DEBUG::2013-11-20 02:25:09,926::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912313::DEBUG::2013-11-20 02:25:09,927::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912313::DEBUG::2013-11-20 02:25:09,927::task::495::TaskManager.Task::(_debug) Task ac8f9089-6df9-4fb4-aabf-4dd8137c94c9: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912313::DEBUG::2013-11-20 02:25:09,928::task::495::TaskManager.Task::(_debug) Task ac8f9089-6df9-4fb4-aabf-4dd8137c94c9: moving from state preparing -> state finished >Thread-15912313::DEBUG::2013-11-20 02:25:09,928::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912313::DEBUG::2013-11-20 02:25:09,929::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912313::DEBUG::2013-11-20 02:25:09,929::task::495::TaskManager.Task::(_debug) Task ac8f9089-6df9-4fb4-aabf-4dd8137c94c9: ref 0 aborting False >Thread-15912313::INFO::2013-11-20 02:25:09,930::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912314::DEBUG::2013-11-20 02:25:10,945::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912314::INFO::2013-11-20 02:25:10,946::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912314::DEBUG::2013-11-20 02:25:10,946::task::495::TaskManager.Task::(_debug) Task 7fcd52ca-9bcd-4c83-98f9-dcc8efb1086f: moving from state init -> state preparing >Thread-15912314::DEBUG::2013-11-20 02:25:10,947::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912314::DEBUG::2013-11-20 02:25:10,947::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912314::DEBUG::2013-11-20 02:25:10,948::task::495::TaskManager.Task::(_debug) Task 7fcd52ca-9bcd-4c83-98f9-dcc8efb1086f: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912314::DEBUG::2013-11-20 02:25:10,948::task::495::TaskManager.Task::(_debug) Task 7fcd52ca-9bcd-4c83-98f9-dcc8efb1086f: moving from state preparing -> state finished >Thread-15912314::DEBUG::2013-11-20 02:25:10,949::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912314::DEBUG::2013-11-20 02:25:10,949::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912314::DEBUG::2013-11-20 02:25:10,950::task::495::TaskManager.Task::(_debug) Task 7fcd52ca-9bcd-4c83-98f9-dcc8efb1086f: ref 0 aborting False >Thread-15912314::INFO::2013-11-20 02:25:10,950::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912316::DEBUG::2013-11-20 02:25:11,965::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912316::INFO::2013-11-20 02:25:11,966::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912316::DEBUG::2013-11-20 02:25:11,979::task::495::TaskManager.Task::(_debug) Task 0995682d-648a-4def-9f23-e3a50beac70d: moving from state init -> state preparing >Thread-15912316::DEBUG::2013-11-20 02:25:11,980::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912316::DEBUG::2013-11-20 02:25:11,980::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912316::DEBUG::2013-11-20 02:25:11,981::task::495::TaskManager.Task::(_debug) Task 0995682d-648a-4def-9f23-e3a50beac70d: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912316::DEBUG::2013-11-20 02:25:11,981::task::495::TaskManager.Task::(_debug) Task 0995682d-648a-4def-9f23-e3a50beac70d: moving from state preparing -> state finished >Thread-15912316::DEBUG::2013-11-20 02:25:11,982::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912316::DEBUG::2013-11-20 02:25:11,982::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912316::DEBUG::2013-11-20 02:25:11,983::task::495::TaskManager.Task::(_debug) Task 0995682d-648a-4def-9f23-e3a50beac70d: ref 0 aborting False >Thread-15912316::INFO::2013-11-20 02:25:11,984::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912317::DEBUG::2013-11-20 02:25:12,999::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912317::INFO::2013-11-20 02:25:12,999::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912317::DEBUG::2013-11-20 02:25:13,000::task::495::TaskManager.Task::(_debug) Task 90586aeb-34e6-4028-b43c-618496c53c2f: moving from state init -> state preparing >Thread-15912317::DEBUG::2013-11-20 02:25:13,000::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912317::DEBUG::2013-11-20 02:25:13,001::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912317::DEBUG::2013-11-20 02:25:13,002::task::495::TaskManager.Task::(_debug) Task 90586aeb-34e6-4028-b43c-618496c53c2f: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912317::DEBUG::2013-11-20 02:25:13,002::task::495::TaskManager.Task::(_debug) Task 90586aeb-34e6-4028-b43c-618496c53c2f: moving from state preparing -> state finished >Thread-15912317::DEBUG::2013-11-20 02:25:13,003::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912317::DEBUG::2013-11-20 02:25:13,003::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912317::DEBUG::2013-11-20 02:25:13,004::task::495::TaskManager.Task::(_debug) Task 90586aeb-34e6-4028-b43c-618496c53c2f: ref 0 aborting False >Thread-15912317::INFO::2013-11-20 02:25:13,004::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-1024767::ERROR::2013-11-20 02:25:13,228::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912318::DEBUG::2013-11-20 02:25:14,019::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912318::INFO::2013-11-20 02:25:14,020::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912318::DEBUG::2013-11-20 02:25:14,021::task::495::TaskManager.Task::(_debug) Task b77f9237-bc39-4fed-8e08-b4a710cee321: moving from state init -> state preparing >Thread-15912318::DEBUG::2013-11-20 02:25:14,021::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912318::DEBUG::2013-11-20 02:25:14,022::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912318::DEBUG::2013-11-20 02:25:14,022::task::495::TaskManager.Task::(_debug) Task b77f9237-bc39-4fed-8e08-b4a710cee321: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912318::DEBUG::2013-11-20 02:25:14,023::task::495::TaskManager.Task::(_debug) Task b77f9237-bc39-4fed-8e08-b4a710cee321: moving from state preparing -> state finished >Thread-15912318::DEBUG::2013-11-20 02:25:14,023::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912318::DEBUG::2013-11-20 02:25:14,024::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912318::DEBUG::2013-11-20 02:25:14,024::task::495::TaskManager.Task::(_debug) Task b77f9237-bc39-4fed-8e08-b4a710cee321: ref 0 aborting False >Thread-15912318::INFO::2013-11-20 02:25:14,025::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912320::DEBUG::2013-11-20 02:25:15,040::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912320::INFO::2013-11-20 02:25:15,041::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912320::DEBUG::2013-11-20 02:25:15,042::task::495::TaskManager.Task::(_debug) Task 68b2d574-2a14-482f-aead-e87309927392: moving from state init -> state preparing >Thread-15912320::DEBUG::2013-11-20 02:25:15,042::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912320::DEBUG::2013-11-20 02:25:15,043::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912320::DEBUG::2013-11-20 02:25:15,043::task::495::TaskManager.Task::(_debug) Task 68b2d574-2a14-482f-aead-e87309927392: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912320::DEBUG::2013-11-20 02:25:15,044::task::495::TaskManager.Task::(_debug) Task 68b2d574-2a14-482f-aead-e87309927392: moving from state preparing -> state finished >Thread-15912320::DEBUG::2013-11-20 02:25:15,044::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912320::DEBUG::2013-11-20 02:25:15,045::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912320::DEBUG::2013-11-20 02:25:15,045::task::495::TaskManager.Task::(_debug) Task 68b2d574-2a14-482f-aead-e87309927392: ref 0 aborting False >Thread-15912320::INFO::2013-11-20 02:25:15,046::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912321::DEBUG::2013-11-20 02:25:16,061::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912321::INFO::2013-11-20 02:25:16,062::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912321::DEBUG::2013-11-20 02:25:16,063::task::495::TaskManager.Task::(_debug) Task 35b28066-4451-4da4-aac6-6ab4e8ef620e: moving from state init -> state preparing >Thread-15912321::DEBUG::2013-11-20 02:25:16,063::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912321::DEBUG::2013-11-20 02:25:16,064::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912321::DEBUG::2013-11-20 02:25:16,064::task::495::TaskManager.Task::(_debug) Task 35b28066-4451-4da4-aac6-6ab4e8ef620e: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912321::DEBUG::2013-11-20 02:25:16,065::task::495::TaskManager.Task::(_debug) Task 35b28066-4451-4da4-aac6-6ab4e8ef620e: moving from state preparing -> state finished >Thread-15912321::DEBUG::2013-11-20 02:25:16,065::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912321::DEBUG::2013-11-20 02:25:16,066::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912321::DEBUG::2013-11-20 02:25:16,066::task::495::TaskManager.Task::(_debug) Task 35b28066-4451-4da4-aac6-6ab4e8ef620e: ref 0 aborting False >Thread-15912321::INFO::2013-11-20 02:25:16,067::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912322::DEBUG::2013-11-20 02:25:17,082::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912322::INFO::2013-11-20 02:25:17,083::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912322::DEBUG::2013-11-20 02:25:17,103::task::495::TaskManager.Task::(_debug) Task 1dbf3482-6fec-472f-84ce-b083df4a7f7f: moving from state init -> state preparing >Thread-15912322::DEBUG::2013-11-20 02:25:17,104::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912322::DEBUG::2013-11-20 02:25:17,104::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912322::DEBUG::2013-11-20 02:25:17,105::task::495::TaskManager.Task::(_debug) Task 1dbf3482-6fec-472f-84ce-b083df4a7f7f: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912322::DEBUG::2013-11-20 02:25:17,105::task::495::TaskManager.Task::(_debug) Task 1dbf3482-6fec-472f-84ce-b083df4a7f7f: moving from state preparing -> state finished >Thread-15912322::DEBUG::2013-11-20 02:25:17,106::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912322::DEBUG::2013-11-20 02:25:17,106::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912322::DEBUG::2013-11-20 02:25:17,107::task::495::TaskManager.Task::(_debug) Task 1dbf3482-6fec-472f-84ce-b083df4a7f7f: ref 0 aborting False >Thread-15912322::INFO::2013-11-20 02:25:17,107::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912324::DEBUG::2013-11-20 02:25:18,122::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912324::INFO::2013-11-20 02:25:18,123::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=378b55ad-db2a-4978-8944-3175e4685c11) >Thread-15912324::DEBUG::2013-11-20 02:25:18,124::task::495::TaskManager.Task::(_debug) Task d708170d-2708-47ae-b6a5-30bdd8cd32a2: moving from state init -> state preparing >Thread-15912324::DEBUG::2013-11-20 02:25:18,124::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 378b55ad-db2a-4978-8944-3175e4685c11 >Thread-15912324::DEBUG::2013-11-20 02:25:18,125::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'} >Thread-15912324::DEBUG::2013-11-20 02:25:18,125::task::495::TaskManager.Task::(_debug) Task d708170d-2708-47ae-b6a5-30bdd8cd32a2: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >Thread-15912324::DEBUG::2013-11-20 02:25:18,126::task::495::TaskManager.Task::(_debug) Task d708170d-2708-47ae-b6a5-30bdd8cd32a2: moving from state preparing -> state finished >Thread-15912324::DEBUG::2013-11-20 02:25:18,127::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912324::DEBUG::2013-11-20 02:25:18,127::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912324::DEBUG::2013-11-20 02:25:18,128::task::495::TaskManager.Task::(_debug) Task d708170d-2708-47ae-b6a5-30bdd8cd32a2: ref 0 aborting False >Thread-15912324::INFO::2013-11-20 02:25:18,128::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '378b55ad-db2a-4978-8944-3175e4685c11'}} >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,259::safelease::61::Storage.Misc.excCmd::(acquire) FAILED: <err> = ''; <rc> = 1 >378b55ad-db2a-4978-8944-3175e4685c11::ERROR::2013-11-20 02:25:40,260::task::868::TaskManager.Task::(_setError) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 876, in _run > File "/usr/share/vdsm/storage/task.py", line 303, in run > File "/usr/share/vdsm/storage/spm.py", line 463, in start > File "/usr/share/vdsm/storage/sp.py", line 203, in acquireClusterLock > File "/usr/share/vdsm/storage/sd.py", line 376, in acquireClusterLock > File "/usr/share/vdsm/storage/safelease.py", line 63, in acquire >AcquireLockFailure: Could not obtain lock: 'id=6b892067-ac7e-4039-a7be-5b7387525b16, rc=1, out=[], err=[]' >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,279::task::495::TaskManager.Task::(_debug) Task 378b55ad-db2a-4978-8944-3175e4685c11: Task._run: 378b55ad-db2a-4978-8944-3175e4685c11 () {} failed - stopping task >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,279::task::495::TaskManager.Task::(_debug) Task 378b55ad-db2a-4978-8944-3175e4685c11: stopping in state running (force False) >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,280::task::495::TaskManager.Task::(_debug) Task 378b55ad-db2a-4978-8944-3175e4685c11: ref 1 aborting True >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,280::task::918::TaskManager.Task::(_runJobs) aborting: Task is aborted: 'Could not obtain lock' - code 651 >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,281::task::495::TaskManager.Task::(_debug) Task 378b55ad-db2a-4978-8944-3175e4685c11: ref 0 aborting True >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,281::task::495::TaskManager.Task::(_debug) Task 378b55ad-db2a-4978-8944-3175e4685c11: Task._doAbort: force False >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,282::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,282::task::495::TaskManager.Task::(_debug) Task 378b55ad-db2a-4978-8944-3175e4685c11: moving from state running -> state aborting >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,283::task::495::TaskManager.Task::(_debug) Task 378b55ad-db2a-4978-8944-3175e4685c11: _aborting: recover policy auto >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,283::task::495::TaskManager.Task::(_debug) Task 378b55ad-db2a-4978-8944-3175e4685c11: moving from state aborting -> state racquiring >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,284::task::495::TaskManager.Task::(_debug) Task 378b55ad-db2a-4978-8944-3175e4685c11: moving from state racquiring -> state recovering >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,284::task::495::TaskManager.Task::(_debug) Task 378b55ad-db2a-4978-8944-3175e4685c11: _recover >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,285::task::495::TaskManager.Task::(_debug) Task 378b55ad-db2a-4978-8944-3175e4685c11: running recovery None >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,285::task::495::TaskManager.Task::(_debug) Task 378b55ad-db2a-4978-8944-3175e4685c11: Recover Done: state recovering >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,286::task::495::TaskManager.Task::(_debug) Task 378b55ad-db2a-4978-8944-3175e4685c11: moving from state recovering -> state recovered >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,286::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,287::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,287::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,288::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,289::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,289::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >378b55ad-db2a-4978-8944-3175e4685c11::DEBUG::2013-11-20 02:25:40,290::threadPool::47::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 >Thread-42838::DEBUG::2013-11-20 02:25:58,261::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 02:25:58,907::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 02:27:27,590::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `5d7b5178-3790-472c-a13b-b481eb27a61e` (Operation Stuck) >Thread-42839::ERROR::2013-11-20 02:27:36,100::sp::113::Storage.StatsThread::(run) Could not figure out delay for domain `45cdff8b-89c0-4926-a8c7-80354064bfa6` (Operation Stuck) >Thread-1024767::ERROR::2013-11-20 02:28:02,248::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-1024767::ERROR::2013-11-20 02:28:12,268::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-1024767::ERROR::2013-11-20 02:28:22,285::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-42840::DEBUG::2013-11-20 02:28:26,983::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 02:28:27,671::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 02:28:32,302::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-1024767::ERROR::2013-11-20 02:28:42,318::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-1024767::ERROR::2013-11-20 02:28:52,335::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-1024767::ERROR::2013-11-20 02:29:02,353::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-1024767::ERROR::2013-11-20 02:29:12,370::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-1024767::ERROR::2013-11-20 02:29:22,387::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912329::INFO::2013-11-20 02:29:29,144::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912329::DEBUG::2013-11-20 02:29:29,145::task::495::TaskManager.Task::(_debug) Task 0fcb44b0-69b2-47ca-be34-9392c01f5df8: moving from state init -> state preparing >Thread-15912329::DEBUG::2013-11-20 02:29:29,145::task::495::TaskManager.Task::(_debug) Task 0fcb44b0-69b2-47ca-be34-9392c01f5df8: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000340938568115', 'lastCheck': 1384914559.4371641, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00183796882629', 'lastCheck': 1384914562.3889771, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00139403343201', 'lastCheck': 1384914562.3900311, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000348091125488', 'lastCheck': 1384914567.7438021, 'valid': True, 'code': 0}} >Thread-15912329::DEBUG::2013-11-20 02:29:29,146::task::495::TaskManager.Task::(_debug) Task 0fcb44b0-69b2-47ca-be34-9392c01f5df8: moving from state preparing -> state finished >Thread-15912329::DEBUG::2013-11-20 02:29:29,147::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912329::DEBUG::2013-11-20 02:29:29,147::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912329::DEBUG::2013-11-20 02:29:29,148::task::495::TaskManager.Task::(_debug) Task 0fcb44b0-69b2-47ca-be34-9392c01f5df8: ref 0 aborting False >Thread-15912329::INFO::2013-11-20 02:29:29,148::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000340938568115', 'lastCheck': 1384914559.4371641, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00183796882629', 'lastCheck': 1384914562.3889771, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00139403343201', 'lastCheck': 1384914562.3900311, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000348091125488', 'lastCheck': 1384914567.7438021, 'valid': True, 'code': 0}} >Thread-15912330::DEBUG::2013-11-20 02:29:29,402::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVdsCapabilities with () {} >Thread-15912330::DEBUG::2013-11-20 02:29:29,781::utils::579::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-img-rhev' (cwd None) >Thread-15912330::DEBUG::2013-11-20 02:29:30,236::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912330::DEBUG::2013-11-20 02:29:30,238::utils::579::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" qemu-kvm-rhev' (cwd None) >Thread-15912330::DEBUG::2013-11-20 02:29:30,645::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912330::DEBUG::2013-11-20 02:29:30,646::utils::579::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" vdsm' (cwd None) >Thread-15912330::DEBUG::2013-11-20 02:29:31,088::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912330::DEBUG::2013-11-20 02:29:31,090::utils::579::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" spice-server' (cwd None) >Thread-15912330::DEBUG::2013-11-20 02:29:31,502::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912330::DEBUG::2013-11-20 02:29:31,503::utils::579::Storage.Misc.excCmd::(execCmd) '/bin/rpm -q --qf "%{NAME}\t%{VERSION}\t%{RELEASE}\t%{BUILDTIME}\n" libvirt' (cwd None) >Thread-15912330::DEBUG::2013-11-20 02:29:31,928::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912330::DEBUG::2013-11-20 02:29:31,930::clientIF::59::vds::(wrapper) return getVdsCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:c936ebb147a8'}], 'FC': []}, 'packages2': {'kernel': {'release': '279.11.1.el6.x86_64', 'buildtime': 1348297826.0, 'version': '2.6.32'}, 'spice-server': {'release': '10.el6', 'buildtime': '1337611492', 'version': '0.10.1'}, 'vdsm': {'release': '113.4.el6_3', 'buildtime': '1347534072', 'version': '4.9'}, 'qemu-kvm': {'release': '2.295.el6_3.2', 'buildtime': '1346673796', 'version': '0.12.1.2'}, 'libvirt': {'release': '21.el6_3.5', 'buildtime': '1348687577', 'version': '0.9.10'}, 'qemu-img': {'release': '2.295.el6_3.2', 'buildtime': '1346673796', 'version': '0.12.1.2'}}, 'cpuModel': 'Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz', 'hooks': {'before_vm_start': {'50_vhostmd': {'md5': '1635e45584995d9ad2629939cb0bcdbf'}}, 'after_vm_destroy': {'50_vhostmd': {'md5': '7e7509430304099f38e4a9de966af090'}}, 'before_vm_dehibernate': {'50_vhostmd': {'md5': '1635e45584995d9ad2629939cb0bcdbf'}}, 'before_vm_migrate_destination': {'50_vhostmd': {'md5': '1635e45584995d9ad2629939cb0bcdbf'}}}, 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'addr': '192.168.100.133', 'cfg': {'IPADDR': '192.168.100.133', 'GATEWAY': '192.168.100.1', 'DELAY': '0', 'NETMASK': '255.255.128.0', 'STP': 'no', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ports': ['bond0'], 'netmask': '255.255.128.0', 'stp': 'off', 'gateway': '192.168.100.1'}, 'DMZ': {'addr': '172.16.1.133', 'cfg': {'IPADDR': '172.16.1.133', 'DELAY': '0', 'NETMASK': '255.255.255.0', 'STP': 'no', 'DEVICE': 'DMZ', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ports': ['eth0'], 'netmask': '255.255.255.0', 'stp': 'off', 'gateway': '0.0.0.0'}}, 'uuid': '30343536-3138-5A43-4A32-333630373442_AC:16:2D:76:EB:F0', 'lastClientIface': 'rhevm', 'nics': {'eth3': {'hwaddr': 'AC:16:2D:76:EB:F2', 'netmask': '', 'speed': 1000, 'addr': '', 'permhwaddr': 'AC:16:2D:76:EB:F3'}, 'eth2': {'hwaddr': 'AC:16:2D:76:EB:F2', 'netmask': '', 'speed': 1000, 'addr': '', 'permhwaddr': 'AC:16:2D:76:EB:F2'}, 'eth1': {'hwaddr': 'AC:16:2D:76:EB:F1', 'netmask': '', 'speed': 0, 'addr': ''}, 'eth0': {'hwaddr': 'AC:16:2D:76:EB:F0', 'netmask': '', 'speed': 1000, 'addr': ''}}, 'software_revision': '113.4', 'management_ip': '192.168.100.133', 'clusterLevels': ['3.0'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,dca,sse4_1,sse4_2,x2apic,popcnt,aes,xsave,avx,lahf_lm,ida,arat,epb,xsaveopt,pln,pts,dts,tpr_shadow,vnmi,flexpriority,ept,vpid,model_486,model_pentium,model_pentium2,model_pentium3,model_pentiumpro,model_qemu32,model_cpu64-rhel5,model_cpu64-rhel6,model_kvm32,model_coreduo,model_kvm64,model_core2duo,model_n270,model_Conroe,model_Penryn,model_Nehalem,model_Westmere,model_SandyBridge,model_Opteron_G1', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:c936ebb147a8', 'memSize': '64392', 'reservedMem': '256', 'bondings': {'bond4': {'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'netmask': '', 'addr': '', 'slaves': []}, 'bond0': {'hwaddr': 'AC:16:2D:76:EB:F2', 'cfg': {'DEVICE': 'bond0', 'BRIDGE': 'rhevm', 'BOOTPROTO': 'none', 'ONBOOT': 'yes', 'BONDING_OPTS': 'mode=1 miimon=100'}, 'netmask': '', 'addr': '', 'slaves': ['eth2', 'eth3']}, 'bond1': {'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'netmask': '', 'addr': '', 'slaves': []}, 'bond2': {'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'netmask': '', 'addr': '', 'slaves': []}, 'bond3': {'hwaddr': '00:00:00:00:00:00', 'cfg': {}, 'netmask': '', 'addr': '', 'slaves': []}}, 'software_version': '4.9', 'cpuSpeed': '1995.126', 'cpuSockets': '2', 'vlans': {}, 'cpuCores': '12', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'version_name': 'Snow Man', 'emulatedMachines': [u'rhel6.3.0', u'pc', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '20121012.0.el6_3', 'version': '6.3', 'name': 'RHEV Hypervisor'}, 'lastClient': '192.168.100.135'}} >Thread-1024767::ERROR::2013-11-20 02:29:32,443::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912332::DEBUG::2013-11-20 02:29:36,180::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912332::INFO::2013-11-20 02:29:36,181::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912332::DEBUG::2013-11-20 02:29:36,182::task::495::TaskManager.Task::(_debug) Task c5c3141c-20c3-4b79-af5a-0d149ca1316e: moving from state init -> state preparing >Thread-15912332::DEBUG::2013-11-20 02:29:36,183::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=Free spmLver=286 spmId=1 >Thread-15912332::DEBUG::2013-11-20 02:29:36,183::task::495::TaskManager.Task::(_debug) Task c5c3141c-20c3-4b79-af5a-0d149ca1316e: finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 286}} >Thread-15912332::DEBUG::2013-11-20 02:29:36,184::task::495::TaskManager.Task::(_debug) Task c5c3141c-20c3-4b79-af5a-0d149ca1316e: moving from state preparing -> state finished >Thread-15912332::DEBUG::2013-11-20 02:29:36,184::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912332::DEBUG::2013-11-20 02:29:36,185::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912332::DEBUG::2013-11-20 02:29:36,186::task::495::TaskManager.Task::(_debug) Task c5c3141c-20c3-4b79-af5a-0d149ca1316e: ref 0 aborting False >Thread-15912332::INFO::2013-11-20 02:29:36,186::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 286}} >Thread-1024767::ERROR::2013-11-20 02:29:42,461::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912337::INFO::2013-11-20 02:29:48,344::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912337::DEBUG::2013-11-20 02:29:48,345::task::495::TaskManager.Task::(_debug) Task 5fbf9b8e-4d22-4348-b3ac-f95ce131f0b6: moving from state init -> state preparing >Thread-15912337::DEBUG::2013-11-20 02:29:48,346::task::495::TaskManager.Task::(_debug) Task 5fbf9b8e-4d22-4348-b3ac-f95ce131f0b6: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000386953353882', 'lastCheck': 1384914579.736455, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00133490562439', 'lastCheck': 1384914582.4611671, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00112891197205', 'lastCheck': 1384914582.4634669, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00032114982605', 'lastCheck': 1384914587.8878109, 'valid': True, 'code': 0}} >Thread-15912337::DEBUG::2013-11-20 02:29:48,346::task::495::TaskManager.Task::(_debug) Task 5fbf9b8e-4d22-4348-b3ac-f95ce131f0b6: moving from state preparing -> state finished >Thread-15912337::DEBUG::2013-11-20 02:29:48,347::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912337::DEBUG::2013-11-20 02:29:48,347::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912337::DEBUG::2013-11-20 02:29:48,348::task::495::TaskManager.Task::(_debug) Task 5fbf9b8e-4d22-4348-b3ac-f95ce131f0b6: ref 0 aborting False >Thread-15912337::INFO::2013-11-20 02:29:48,349::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000386953353882', 'lastCheck': 1384914579.736455, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00133490562439', 'lastCheck': 1384914582.4611671, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00112891197205', 'lastCheck': 1384914582.4634669, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00032114982605', 'lastCheck': 1384914587.8878109, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:29:52,482::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912342::DEBUG::2013-11-20 02:30:00,607::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912342::INFO::2013-11-20 02:30:00,608::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: validateStorageServerConnection, args: (domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{'connection': 'kairo.brd-klee.dk:/export', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36', 'port': ''}]) >Thread-15912342::DEBUG::2013-11-20 02:30:00,609::task::495::TaskManager.Task::(_debug) Task 95b15c31-2160-4309-8000-70e25468915c: moving from state init -> state preparing >Thread-15912342::INFO::2013-11-20 02:30:00,609::storage_connection::127::Storage.ServerConnection::(validate) Request to validate NFS storage server >Thread-15912342::INFO::2013-11-20 02:30:00,610::storage_connection::53::Storage.ServerConnection::(__validateConnectionParams) conList=[{'connection': 'kairo.brd-klee.dk:/export', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36', 'port': ''}] >Thread-15912342::DEBUG::2013-11-20 02:30:00,614::fileUtils::83::Storage.Misc.excCmd::(mount) '/usr/bin/sudo -n /bin/mount -o soft,timeo=600,retrans=6,nosharecache,vers=3 -t nfs kairo.brd-klee.dk:/export /tmp/tmpffZdt0' (cwd None) >Thread-15912342::DEBUG::2013-11-20 02:30:01,089::fileUtils::83::Storage.Misc.excCmd::(mount) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912342::DEBUG::2013-11-20 02:30:01,102::fileUtils::112::Storage.Misc.excCmd::(umount) '/usr/bin/sudo -n /bin/umount -t nfs -f /tmp/tmpffZdt0' (cwd None) >Thread-15912342::DEBUG::2013-11-20 02:30:01,522::fileUtils::112::Storage.Misc.excCmd::(umount) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912342::DEBUG::2013-11-20 02:30:01,526::task::495::TaskManager.Task::(_debug) Task 95b15c31-2160-4309-8000-70e25468915c: finished: {'statuslist': [{'status': 0, 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36'}]} >Thread-15912342::DEBUG::2013-11-20 02:30:01,533::task::495::TaskManager.Task::(_debug) Task 95b15c31-2160-4309-8000-70e25468915c: moving from state preparing -> state finished >Thread-15912342::DEBUG::2013-11-20 02:30:01,533::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912342::DEBUG::2013-11-20 02:30:01,534::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912342::DEBUG::2013-11-20 02:30:01,534::task::495::TaskManager.Task::(_debug) Task 95b15c31-2160-4309-8000-70e25468915c: ref 0 aborting False >Thread-15912342::INFO::2013-11-20 02:30:01,535::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: validateStorageServerConnection, Return response: {'status': {'message': 'OK', 'code': 0}, 'statuslist': [{'status': 0, 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36'}]} >Thread-15912344::DEBUG::2013-11-20 02:30:01,565::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912344::INFO::2013-11-20 02:30:01,566::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: connectStorageServer, args: (domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{'connection': 'kairo.brd-klee.dk:/export', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36', 'port': ''}]) >Thread-15912344::DEBUG::2013-11-20 02:30:01,566::task::495::TaskManager.Task::(_debug) Task dcffbae9-8513-4685-8229-b33daf6d5b29: moving from state init -> state preparing >Thread-15912344::INFO::2013-11-20 02:30:01,567::storage_connection::95::Storage.ServerConnection::(connect) Request to connect NFS storage server >Thread-15912344::INFO::2013-11-20 02:30:01,567::storage_connection::53::Storage.ServerConnection::(__validateConnectionParams) conList=[{'connection': 'kairo.brd-klee.dk:/export', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36', 'port': ''}] >Thread-15912344::DEBUG::2013-11-20 02:30:01,581::lvm::547::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15912344::DEBUG::2013-11-20 02:30:01,582::lvm::549::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15912344::DEBUG::2013-11-20 02:30:01,583::lvm::559::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15912344::DEBUG::2013-11-20 02:30:01,583::lvm::561::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15912344::DEBUG::2013-11-20 02:30:01,584::lvm::580::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15912344::DEBUG::2013-11-20 02:30:01,585::lvm::582::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15912344::DEBUG::2013-11-20 02:30:01,585::task::495::TaskManager.Task::(_debug) Task dcffbae9-8513-4685-8229-b33daf6d5b29: finished: {'statuslist': [{'status': 0, 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36'}]} >Thread-15912344::DEBUG::2013-11-20 02:30:01,586::task::495::TaskManager.Task::(_debug) Task dcffbae9-8513-4685-8229-b33daf6d5b29: moving from state preparing -> state finished >Thread-15912344::DEBUG::2013-11-20 02:30:01,586::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912344::DEBUG::2013-11-20 02:30:01,587::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912344::DEBUG::2013-11-20 02:30:01,587::task::495::TaskManager.Task::(_debug) Task dcffbae9-8513-4685-8229-b33daf6d5b29: ref 0 aborting False >Thread-15912344::INFO::2013-11-20 02:30:01,588::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: connectStorageServer, Return response: {'status': {'message': 'OK', 'code': 0}, 'statuslist': [{'status': 0, 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36'}]} >Thread-1024767::ERROR::2013-11-20 02:30:02,501::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912345::INFO::2013-11-20 02:30:03,719::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912345::DEBUG::2013-11-20 02:30:03,720::task::495::TaskManager.Task::(_debug) Task 8d33322f-577e-4b7b-9479-b12d805f7651: moving from state init -> state preparing >Thread-15912345::DEBUG::2013-11-20 02:30:03,720::task::495::TaskManager.Task::(_debug) Task 8d33322f-577e-4b7b-9479-b12d805f7651: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000412940979004', 'lastCheck': 1384914599.7623041, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00247693061829', 'lastCheck': 1384914602.5034449, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00165390968323', 'lastCheck': 1384914602.504374, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000339031219482', 'lastCheck': 1384914597.899472, 'valid': True, 'code': 0}} >Thread-15912345::DEBUG::2013-11-20 02:30:03,721::task::495::TaskManager.Task::(_debug) Task 8d33322f-577e-4b7b-9479-b12d805f7651: moving from state preparing -> state finished >Thread-15912345::DEBUG::2013-11-20 02:30:03,722::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912345::DEBUG::2013-11-20 02:30:03,722::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912345::DEBUG::2013-11-20 02:30:03,723::task::495::TaskManager.Task::(_debug) Task 8d33322f-577e-4b7b-9479-b12d805f7651: ref 0 aborting False >Thread-15912345::INFO::2013-11-20 02:30:03,723::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000412940979004', 'lastCheck': 1384914599.7623041, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00247693061829', 'lastCheck': 1384914602.5034449, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00165390968323', 'lastCheck': 1384914602.504374, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000339031219482', 'lastCheck': 1384914597.899472, 'valid': True, 'code': 0}} >Thread-42840::DEBUG::2013-11-20 02:30:07,911::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-42840::DEBUG::2013-11-20 02:30:07,919::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 02:30:08,570::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-42840::DEBUG::2013-11-20 02:30:08,575::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-42838::DEBUG::2013-11-20 02:30:09,773::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-42838::DEBUG::2013-11-20 02:30:09,774::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 02:30:10,393::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-42838::DEBUG::2013-11-20 02:30:10,396::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1024767::ERROR::2013-11-20 02:30:12,520::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912351::INFO::2013-11-20 02:30:19,069::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912351::DEBUG::2013-11-20 02:30:19,070::task::495::TaskManager.Task::(_debug) Task 9c55bdce-6f76-44bc-9b1c-5fb5b7bf7de5: moving from state init -> state preparing >Thread-15912351::DEBUG::2013-11-20 02:30:19,070::task::495::TaskManager.Task::(_debug) Task 9c55bdce-6f76-44bc-9b1c-5fb5b7bf7de5: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000403881072998', 'lastCheck': 1384914610.3995609, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00163793563843', 'lastCheck': 1384914612.5220251, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00124192237854', 'lastCheck': 1384914612.523622, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000380039215088', 'lastCheck': 1384914618.5887921, 'valid': True, 'code': 0}} >Thread-15912351::DEBUG::2013-11-20 02:30:19,071::task::495::TaskManager.Task::(_debug) Task 9c55bdce-6f76-44bc-9b1c-5fb5b7bf7de5: moving from state preparing -> state finished >Thread-15912351::DEBUG::2013-11-20 02:30:19,072::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912351::DEBUG::2013-11-20 02:30:19,072::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912351::DEBUG::2013-11-20 02:30:19,073::task::495::TaskManager.Task::(_debug) Task 9c55bdce-6f76-44bc-9b1c-5fb5b7bf7de5: ref 0 aborting False >Thread-15912351::INFO::2013-11-20 02:30:19,073::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000403881072998', 'lastCheck': 1384914610.3995609, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00163793563843', 'lastCheck': 1384914612.5220251, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00124192237854', 'lastCheck': 1384914612.523622, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000380039215088', 'lastCheck': 1384914618.5887921, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:30:22,538::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-1024767::ERROR::2013-11-20 02:30:32,555::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912357::INFO::2013-11-20 02:30:34,536::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912357::DEBUG::2013-11-20 02:30:34,537::task::495::TaskManager.Task::(_debug) Task 186a33ad-b9c9-40d4-8078-c1d6cc06d437: moving from state init -> state preparing >Thread-15912357::DEBUG::2013-11-20 02:30:34,538::task::495::TaskManager.Task::(_debug) Task 186a33ad-b9c9-40d4-8078-c1d6cc06d437: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000308036804199', 'lastCheck': 1384914630.4250901, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0014340877533', 'lastCheck': 1384914632.5565829, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0014340877533', 'lastCheck': 1384914632.5583401, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000425815582275', 'lastCheck': 1384914628.6003871, 'valid': True, 'code': 0}} >Thread-15912357::DEBUG::2013-11-20 02:30:34,538::task::495::TaskManager.Task::(_debug) Task 186a33ad-b9c9-40d4-8078-c1d6cc06d437: moving from state preparing -> state finished >Thread-15912357::DEBUG::2013-11-20 02:30:34,539::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912357::DEBUG::2013-11-20 02:30:34,540::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912357::DEBUG::2013-11-20 02:30:34,540::task::495::TaskManager.Task::(_debug) Task 186a33ad-b9c9-40d4-8078-c1d6cc06d437: ref 0 aborting False >Thread-15912357::INFO::2013-11-20 02:30:34,541::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000308036804199', 'lastCheck': 1384914630.4250901, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0014340877533', 'lastCheck': 1384914632.5565829, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0014340877533', 'lastCheck': 1384914632.5583401, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000425815582275', 'lastCheck': 1384914628.6003871, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:30:42,573::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912363::INFO::2013-11-20 02:30:49,885::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912363::DEBUG::2013-11-20 02:30:49,886::task::495::TaskManager.Task::(_debug) Task fde00388-bc36-4f2a-b3e7-e313b8e79f26: moving from state init -> state preparing >Thread-15912363::DEBUG::2013-11-20 02:30:49,887::task::495::TaskManager.Task::(_debug) Task fde00388-bc36-4f2a-b3e7-e313b8e79f26: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00033712387085', 'lastCheck': 1384914640.437886, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00182199478149', 'lastCheck': 1384914642.5733891, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00113701820374', 'lastCheck': 1384914642.5751951, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000377893447876', 'lastCheck': 1384914648.6234379, 'valid': True, 'code': 0}} >Thread-15912363::DEBUG::2013-11-20 02:30:49,888::task::495::TaskManager.Task::(_debug) Task fde00388-bc36-4f2a-b3e7-e313b8e79f26: moving from state preparing -> state finished >Thread-15912363::DEBUG::2013-11-20 02:30:49,888::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912363::DEBUG::2013-11-20 02:30:49,889::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912363::DEBUG::2013-11-20 02:30:49,889::task::495::TaskManager.Task::(_debug) Task fde00388-bc36-4f2a-b3e7-e313b8e79f26: ref 0 aborting False >Thread-15912363::INFO::2013-11-20 02:30:49,890::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00033712387085', 'lastCheck': 1384914640.437886, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00182199478149', 'lastCheck': 1384914642.5733891, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00113701820374', 'lastCheck': 1384914642.5751951, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000377893447876', 'lastCheck': 1384914648.6234379, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:30:52,589::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-42838::DEBUG::2013-11-20 02:31:00,462::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 02:31:01,104::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 02:31:02,677::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912369::INFO::2013-11-20 02:31:05,385::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912369::DEBUG::2013-11-20 02:31:05,386::task::495::TaskManager.Task::(_debug) Task ae2075bc-10d8-4cf8-ad05-2bed325797df: moving from state init -> state preparing >Thread-15912369::DEBUG::2013-11-20 02:31:05,387::task::495::TaskManager.Task::(_debug) Task ae2075bc-10d8-4cf8-ad05-2bed325797df: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000346899032593', 'lastCheck': 1384914661.108465, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00259017944336', 'lastCheck': 1384914662.6792359, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00160193443298', 'lastCheck': 1384914662.680382, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000383853912354', 'lastCheck': 1384914658.6350751, 'valid': True, 'code': 0}} >Thread-15912369::DEBUG::2013-11-20 02:31:05,388::task::495::TaskManager.Task::(_debug) Task ae2075bc-10d8-4cf8-ad05-2bed325797df: moving from state preparing -> state finished >Thread-15912369::DEBUG::2013-11-20 02:31:05,388::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912369::DEBUG::2013-11-20 02:31:05,389::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912369::DEBUG::2013-11-20 02:31:05,389::task::495::TaskManager.Task::(_debug) Task ae2075bc-10d8-4cf8-ad05-2bed325797df: ref 0 aborting False >Thread-15912369::INFO::2013-11-20 02:31:05,390::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000346899032593', 'lastCheck': 1384914661.108465, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00259017944336', 'lastCheck': 1384914662.6792359, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00160193443298', 'lastCheck': 1384914662.680382, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000383853912354', 'lastCheck': 1384914658.6350751, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:31:12,695::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912375::INFO::2013-11-20 02:31:20,852::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912375::DEBUG::2013-11-20 02:31:20,853::task::495::TaskManager.Task::(_debug) Task f4076b7c-d76c-4790-aeb9-7ed3b3f01c08: moving from state init -> state preparing >Thread-15912375::DEBUG::2013-11-20 02:31:20,853::task::495::TaskManager.Task::(_debug) Task f4076b7c-d76c-4790-aeb9-7ed3b3f01c08: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00031304359436', 'lastCheck': 1384914671.1212211, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00165295600891', 'lastCheck': 1384914672.6964979, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00109505653381', 'lastCheck': 1384914672.6976659, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000471115112305', 'lastCheck': 1384914678.6884501, 'valid': True, 'code': 0}} >Thread-15912375::DEBUG::2013-11-20 02:31:20,854::task::495::TaskManager.Task::(_debug) Task f4076b7c-d76c-4790-aeb9-7ed3b3f01c08: moving from state preparing -> state finished >Thread-15912375::DEBUG::2013-11-20 02:31:20,855::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912375::DEBUG::2013-11-20 02:31:20,855::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912375::DEBUG::2013-11-20 02:31:20,856::task::495::TaskManager.Task::(_debug) Task f4076b7c-d76c-4790-aeb9-7ed3b3f01c08: ref 0 aborting False >Thread-15912375::INFO::2013-11-20 02:31:20,856::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00031304359436', 'lastCheck': 1384914671.1212211, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00165295600891', 'lastCheck': 1384914672.6964979, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00109505653381', 'lastCheck': 1384914672.6976659, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000471115112305', 'lastCheck': 1384914678.6884501, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:31:22,712::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-1024767::ERROR::2013-11-20 02:31:32,731::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912381::INFO::2013-11-20 02:31:36,293::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912381::DEBUG::2013-11-20 02:31:36,294::task::495::TaskManager.Task::(_debug) Task c31b5b96-0b63-4b77-b8be-5e0dc1008d7b: moving from state init -> state preparing >Thread-15912381::DEBUG::2013-11-20 02:31:36,295::task::495::TaskManager.Task::(_debug) Task c31b5b96-0b63-4b77-b8be-5e0dc1008d7b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000322103500366', 'lastCheck': 1384914691.1461561, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139403343201', 'lastCheck': 1384914692.7330041, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00122117996216', 'lastCheck': 1384914692.733649, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000478029251099', 'lastCheck': 1384914688.6999321, 'valid': True, 'code': 0}} >Thread-15912381::DEBUG::2013-11-20 02:31:36,296::task::495::TaskManager.Task::(_debug) Task c31b5b96-0b63-4b77-b8be-5e0dc1008d7b: moving from state preparing -> state finished >Thread-15912381::DEBUG::2013-11-20 02:31:36,296::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912381::DEBUG::2013-11-20 02:31:36,297::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912381::DEBUG::2013-11-20 02:31:36,297::task::495::TaskManager.Task::(_debug) Task c31b5b96-0b63-4b77-b8be-5e0dc1008d7b: ref 0 aborting False >Thread-15912381::INFO::2013-11-20 02:31:36,298::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000322103500366', 'lastCheck': 1384914691.1461561, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139403343201', 'lastCheck': 1384914692.7330041, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00122117996216', 'lastCheck': 1384914692.733649, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000478029251099', 'lastCheck': 1384914688.6999321, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:31:42,749::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912385::DEBUG::2013-11-20 02:31:44,350::clientIF::54::vds::(wrapper) [192.168.100.135]::call create with ({'bridge': 'rhevm', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.2.0', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'transparentHugePages': 'true', 'spiceSslCipherSuite': 'DEFAULT', 'cpuType': 'Westmere', 'smp': '4', 'macAddr': '00:1a:4a:a8:64:06', 'boot': 'c', 'custom': {}, 'vmType': 'kvm', 'memSize': 2048, 'smpCoresPerSocket': '2', 'vmName': 'KLEE-DC01', 'spiceMonitors': '1', 'nice': '0', 'drives': [{'index': '0', 'domainID': '6b892067-ac7e-4039-a7be-5b7387525b16', 'format': 'raw', 'boot': 'true', 'volumeID': '038d75ec-9421-4d79-9865-0dbebd5271d9', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}], 'smartcardEnable': 'false', 'nicModel': 'pv', 'keyboardLayout': 'en-us', 'kvmEnable': 'true', 'timeOffset': '3600', 'spiceSecureChannels': 'smain,sinputs', 'display': 'qxl'},) {} >Thread-15912385::INFO::2013-11-20 02:31:44,699::clientIF::523::vds::(_getNetworkIp) network None: using 0 >Thread-15912385::INFO::2013-11-20 02:31:44,700::clientIF::656::vds::(create) vmContainerLock acquired by vm 98ec88b5-2c08-4ff2-b607-c3f4e281f6af >Thread-15912386::DEBUG::2013-11-20 02:31:44,705::vm::342::vm.Vm::(_startUnderlyingVm) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Start >Thread-15912385::DEBUG::2013-11-20 02:31:44,706::clientIF::669::vds::(create) Total desktops after creation of 98ec88b5-2c08-4ff2-b607-c3f4e281f6af is 1 >Thread-15912386::DEBUG::2013-11-20 02:31:44,707::vm::346::vm.Vm::(_startUnderlyingVm) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::_ongoingCreations acquired >Thread-15912385::DEBUG::2013-11-20 02:31:44,708::clientIF::59::vds::(wrapper) return create with {'status': {'message': 'Done', 'code': 0}, 'vmList': {'status': 'WaitForLaunch', 'bridge': 'rhevm', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.2.0', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'pid': '0', 'drives': [{'index': '0', 'domainID': '6b892067-ac7e-4039-a7be-5b7387525b16', 'format': 'raw', 'boot': 'true', 'volumeID': '038d75ec-9421-4d79-9865-0dbebd5271d9', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}], 'transparentHugePages': 'true', 'keyboardLayout': 'en-us', 'displayPort': '-1', 'displaySecurePort': '-1', 'spiceSslCipherSuite': 'DEFAULT', 'cpuType': 'Westmere', 'custom': {}, 'clientIp': '', 'nicModel': 'pv', 'macAddr': '00:1a:4a:a8:64:06', 'kvmEnable': 'true', 'boot': 'c', 'smp': '4', 'vmType': 'kvm', 'smartcardEnable': 'false', 'timeOffset': '3600', 'memSize': 2048, 'displayIp': '0', 'spiceSecureChannels': 'smain,sinputs', 'smpCoresPerSocket': '2', 'vmName': 'KLEE-DC01', 'display': 'qxl', 'spiceMonitors': '1', 'nice': '0'}} >Thread-15912386::INFO::2013-11-20 02:31:44,708::libvirtvm::1061::vm.Vm::(_run) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::VM wrapper has started >Thread-15912386::INFO::2013-11-20 02:31:44,712::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912386::DEBUG::2013-11-20 02:31:44,713::task::495::TaskManager.Task::(_debug) Task c01207ea-4d77-41d5-859c-65b0b57669a2: moving from state init -> state preparing >Thread-15912386::DEBUG::2013-11-20 02:31:44,715::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`36a6b7e3-d132-42d8-8104-2ddfdad4c775`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1690' at 'public_prepareVolume' >Thread-15912386::DEBUG::2013-11-20 02:31:44,716::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912386::DEBUG::2013-11-20 02:31:44,717::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912386::DEBUG::2013-11-20 02:31:44,718::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`36a6b7e3-d132-42d8-8104-2ddfdad4c775`::Granted request >Thread-15912386::DEBUG::2013-11-20 02:31:44,719::task::495::TaskManager.Task::(_debug) Task c01207ea-4d77-41d5-859c-65b0b57669a2: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912386::DEBUG::2013-11-20 02:31:44,719::task::495::TaskManager.Task::(_debug) Task c01207ea-4d77-41d5-859c-65b0b57669a2: ref 1 aborting False >Thread-15912386::DEBUG::2013-11-20 02:31:44,721::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.d8be6e76-2f7f-4b56-9ada-c3fd543a9794`ReqID=`2e86c340-e8bc-435e-9fa0-514cd4843568`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1695' at 'public_prepareVolume' >Thread-15912386::DEBUG::2013-11-20 02:31:44,721::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.d8be6e76-2f7f-4b56-9ada-c3fd543a9794' for lock type 'exclusive' >Thread-15912386::DEBUG::2013-11-20 02:31:44,723::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15912386::DEBUG::2013-11-20 02:31:44,724::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15912386::DEBUG::2013-11-20 02:31:45,447::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912386::DEBUG::2013-11-20 02:31:45,898::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15912386::DEBUG::2013-11-20 02:31:45,899::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=15 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15912386::DEBUG::2013-11-20 02:31:46,263::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.00787782 s, 65.0 kB/s\n'; <rc> = 0 >Thread-15912386::DEBUG::2013-11-20 02:31:46,265::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.00787782 s, 65.0 kB/s'], size: 512 >Thread-15912386::DEBUG::2013-11-20 02:31:46,271::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=15 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15912386::DEBUG::2013-11-20 02:31:46,690::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.0003415 s, 1.5 MB/s\n'; <rc> = 0 >Thread-15912386::DEBUG::2013-11-20 02:31:46,691::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.0003415 s, 1.5 MB/s'], size: 512 >Thread-15912386::INFO::2013-11-20 02:31:46,692::image::340::Storage.Image::(getChain) sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 chain=[<storage.blockVolume.BlockVolume instance at 0x1b9bbd8>] >Thread-15912386::DEBUG::2013-11-20 02:31:46,694::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --autobackup n --available y 6b892067-ac7e-4039-a7be-5b7387525b16/038d75ec-9421-4d79-9865-0dbebd5271d9' (cwd None) >Thread-15912386::DEBUG::2013-11-20 02:31:47,643::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912386::DEBUG::2013-11-20 02:31:47,645::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15912386::DEBUG::2013-11-20 02:31:47,645::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15912386::DEBUG::2013-11-20 02:31:47,647::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.038d75ec-9421-4d79-9865-0dbebd5271d9`ReqID=`6eda7ba4-2742-4867-8644-99f3cc0eab4b`::Request was made in '/usr/share/vdsm/storage/resourceFactories.py' line '149' at '__getResourceCandidatesList' >Thread-15912386::DEBUG::2013-11-20 02:31:47,648::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.038d75ec-9421-4d79-9865-0dbebd5271d9' for lock type 'exclusive' >Thread-15912386::DEBUG::2013-11-20 02:31:47,649::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.038d75ec-9421-4d79-9865-0dbebd5271d9' is free. Now locking as 'exclusive' (1 active user) >Thread-15912386::DEBUG::2013-11-20 02:31:47,649::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.038d75ec-9421-4d79-9865-0dbebd5271d9`ReqID=`6eda7ba4-2742-4867-8644-99f3cc0eab4b`::Granted request >Thread-15912386::DEBUG::2013-11-20 02:31:47,650::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.d8be6e76-2f7f-4b56-9ada-c3fd543a9794' is free. Now locking as 'exclusive' (1 active user) >Thread-15912386::DEBUG::2013-11-20 02:31:47,651::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.d8be6e76-2f7f-4b56-9ada-c3fd543a9794`ReqID=`2e86c340-e8bc-435e-9fa0-514cd4843568`::Granted request >Thread-15912386::DEBUG::2013-11-20 02:31:47,653::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15912386::DEBUG::2013-11-20 02:31:47,654::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15912386::DEBUG::2013-11-20 02:31:48,355::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912386::DEBUG::2013-11-20 02:31:48,609::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15912386::DEBUG::2013-11-20 02:31:48,612::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --refresh 6b892067-ac7e-4039-a7be-5b7387525b16/038d75ec-9421-4d79-9865-0dbebd5271d9' (cwd None) >Thread-15912386::DEBUG::2013-11-20 02:31:49,218::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912386::DEBUG::2013-11-20 02:31:49,220::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15912386::DEBUG::2013-11-20 02:31:49,220::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15912386::INFO::2013-11-20 02:31:49,221::volume::517::Storage.Volume::(prepare) Volume: preparing volume 6b892067-ac7e-4039-a7be-5b7387525b16/038d75ec-9421-4d79-9865-0dbebd5271d9 >Thread-15912386::DEBUG::2013-11-20 02:31:49,222::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15912386::DEBUG::2013-11-20 02:31:49,223::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15912386::DEBUG::2013-11-20 02:31:49,889::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912386::DEBUG::2013-11-20 02:31:50,262::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15912386::DEBUG::2013-11-20 02:31:50,263::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=15 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15912386::DEBUG::2013-11-20 02:31:50,589::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000412951 s, 1.2 MB/s\n'; <rc> = 0 >Thread-15912386::DEBUG::2013-11-20 02:31:50,590::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000412951 s, 1.2 MB/s'], size: 512 >Thread-15912386::DEBUG::2013-11-20 02:31:50,591::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=15 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15912386::DEBUG::2013-11-20 02:31:51,004::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000301411 s, 1.7 MB/s\n'; <rc> = 0 >Thread-15912386::DEBUG::2013-11-20 02:31:51,005::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000301411 s, 1.7 MB/s'], size: 512 >Thread-15912386::DEBUG::2013-11-20 02:31:51,007::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.038d75ec-9421-4d79-9865-0dbebd5271d9`ReqID=`b0a176bb-cd08-4284-b53c-f5f72cee42a9`::Request was made in '/usr/share/vdsm/storage/volume.py' line '533' at 'prepare' >Thread-15912386::DEBUG::2013-11-20 02:31:51,007::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.038d75ec-9421-4d79-9865-0dbebd5271d9' for lock type 'exclusive' >Thread-15912386::DEBUG::2013-11-20 02:31:51,008::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.038d75ec-9421-4d79-9865-0dbebd5271d9' is free. Now locking as 'exclusive' (1 active user) >Thread-15912386::DEBUG::2013-11-20 02:31:51,009::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.038d75ec-9421-4d79-9865-0dbebd5271d9`ReqID=`b0a176bb-cd08-4284-b53c-f5f72cee42a9`::Granted request >Thread-15912386::DEBUG::2013-11-20 02:31:51,011::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=15 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15912386::DEBUG::2013-11-20 02:31:51,407::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000200469 s, 2.6 MB/s\n'; <rc> = 0 >Thread-15912386::DEBUG::2013-11-20 02:31:51,408::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000200469 s, 2.6 MB/s'], size: 512 >Thread-15912386::DEBUG::2013-11-20 02:31:51,410::task::495::TaskManager.Task::(_debug) Task c01207ea-4d77-41d5-859c-65b0b57669a2: finished: None >Thread-15912386::DEBUG::2013-11-20 02:31:51,411::task::495::TaskManager.Task::(_debug) Task c01207ea-4d77-41d5-859c-65b0b57669a2: moving from state preparing -> state finished >Thread-15912386::DEBUG::2013-11-20 02:31:51,411::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912386::DEBUG::2013-11-20 02:31:51,412::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912386::DEBUG::2013-11-20 02:31:51,413::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912386::DEBUG::2013-11-20 02:31:51,414::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912386::DEBUG::2013-11-20 02:31:51,414::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912386::DEBUG::2013-11-20 02:31:51,415::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912386::DEBUG::2013-11-20 02:31:51,415::task::495::TaskManager.Task::(_debug) Task c01207ea-4d77-41d5-859c-65b0b57669a2: ref 0 aborting False >Thread-15912386::INFO::2013-11-20 02:31:51,416::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15912386::INFO::2013-11-20 02:31:51,417::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912386::DEBUG::2013-11-20 02:31:51,418::task::495::TaskManager.Task::(_debug) Task 7a69231e-d457-476f-a56c-492479e16b94: moving from state init -> state preparing >Thread-15912386::DEBUG::2013-11-20 02:31:51,419::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d4cd1488-9ec8-4d77-b278-e5c67ae84502`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1666' at 'public_getVolumePath' >Thread-15912386::DEBUG::2013-11-20 02:31:51,420::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912386::DEBUG::2013-11-20 02:31:51,420::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912386::DEBUG::2013-11-20 02:31:51,421::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d4cd1488-9ec8-4d77-b278-e5c67ae84502`::Granted request >Thread-15912386::DEBUG::2013-11-20 02:31:51,422::task::495::TaskManager.Task::(_debug) Task 7a69231e-d457-476f-a56c-492479e16b94: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912386::DEBUG::2013-11-20 02:31:51,422::task::495::TaskManager.Task::(_debug) Task 7a69231e-d457-476f-a56c-492479e16b94: ref 1 aborting False >Thread-15912386::DEBUG::2013-11-20 02:31:51,423::task::495::TaskManager.Task::(_debug) Task 7a69231e-d457-476f-a56c-492479e16b94: finished: {'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/d8be6e76-2f7f-4b56-9ada-c3fd543a9794/038d75ec-9421-4d79-9865-0dbebd5271d9'} >Thread-15912386::DEBUG::2013-11-20 02:31:51,424::task::495::TaskManager.Task::(_debug) Task 7a69231e-d457-476f-a56c-492479e16b94: moving from state preparing -> state finished >Thread-15912386::DEBUG::2013-11-20 02:31:51,425::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912386::DEBUG::2013-11-20 02:31:51,425::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912386::DEBUG::2013-11-20 02:31:51,426::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912386::DEBUG::2013-11-20 02:31:51,427::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912386::DEBUG::2013-11-20 02:31:51,427::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912386::DEBUG::2013-11-20 02:31:51,428::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912386::DEBUG::2013-11-20 02:31:51,428::task::495::TaskManager.Task::(_debug) Task 7a69231e-d457-476f-a56c-492479e16b94: ref 0 aborting False >Thread-15912386::INFO::2013-11-20 02:31:51,429::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, Return response: {'status': {'message': 'OK', 'code': 0}, 'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/d8be6e76-2f7f-4b56-9ada-c3fd543a9794/038d75ec-9421-4d79-9865-0dbebd5271d9'} >Thread-15912386::INFO::2013-11-20 02:31:51,430::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912386::DEBUG::2013-11-20 02:31:51,430::task::495::TaskManager.Task::(_debug) Task f168676e-b6df-466b-9158-c3b5104cf873: moving from state init -> state preparing >Thread-15912386::DEBUG::2013-11-20 02:31:51,431::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`23b99c0f-72b1-4949-ac5e-cb830c2aa9d6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912386::DEBUG::2013-11-20 02:31:51,432::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912386::DEBUG::2013-11-20 02:31:51,433::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912386::DEBUG::2013-11-20 02:31:51,433::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`23b99c0f-72b1-4949-ac5e-cb830c2aa9d6`::Granted request >Thread-15912386::DEBUG::2013-11-20 02:31:51,434::task::495::TaskManager.Task::(_debug) Task f168676e-b6df-466b-9158-c3b5104cf873: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912386::DEBUG::2013-11-20 02:31:51,435::task::495::TaskManager.Task::(_debug) Task f168676e-b6df-466b-9158-c3b5104cf873: ref 1 aborting False >Thread-15912386::DEBUG::2013-11-20 02:31:51,435::task::495::TaskManager.Task::(_debug) Task f168676e-b6df-466b-9158-c3b5104cf873: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912386::DEBUG::2013-11-20 02:31:51,436::task::495::TaskManager.Task::(_debug) Task f168676e-b6df-466b-9158-c3b5104cf873: moving from state preparing -> state finished >Thread-15912386::DEBUG::2013-11-20 02:31:51,436::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912386::DEBUG::2013-11-20 02:31:51,437::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912386::DEBUG::2013-11-20 02:31:51,438::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912386::DEBUG::2013-11-20 02:31:51,438::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912386::DEBUG::2013-11-20 02:31:51,439::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912386::DEBUG::2013-11-20 02:31:51,439::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912386::DEBUG::2013-11-20 02:31:51,440::task::495::TaskManager.Task::(_debug) Task f168676e-b6df-466b-9158-c3b5104cf873: ref 0 aborting False >Thread-15912386::INFO::2013-11-20 02:31:51,440::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912386::INFO::2013-11-20 02:31:51,441::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16) >Thread-15912386::DEBUG::2013-11-20 02:31:51,442::task::495::TaskManager.Task::(_debug) Task 395eff4f-2812-47ae-8751-927442be49d6: moving from state init -> state preparing >Thread-15912386::INFO::2013-11-20 02:31:51,442::blockSD::674::Storage.StorageDomain::(validate) sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 >Thread-15912386::DEBUG::2013-11-20 02:31:51,443::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15912386::DEBUG::2013-11-20 02:31:51,444::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15912386::DEBUG::2013-11-20 02:31:51,444::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15912386::DEBUG::2013-11-20 02:31:51,446::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15912386::DEBUG::2013-11-20 02:31:51,446::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15912386::DEBUG::2013-11-20 02:31:51,448::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15912390::INFO::2013-11-20 02:31:51,660::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912390::DEBUG::2013-11-20 02:31:51,662::task::495::TaskManager.Task::(_debug) Task 9de89b0b-3080-4d8f-a540-9de3237eb831: moving from state init -> state preparing >Thread-15912390::DEBUG::2013-11-20 02:31:51,663::task::495::TaskManager.Task::(_debug) Task 9de89b0b-3080-4d8f-a540-9de3237eb831: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000427007675171', 'lastCheck': 1384914711.2783451, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148391723633', 'lastCheck': 1384914702.7506361, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00133585929871', 'lastCheck': 1384914702.752203, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000707864761353', 'lastCheck': 1384914708.9284389, 'valid': True, 'code': 0}} >Thread-15912390::DEBUG::2013-11-20 02:31:51,663::task::495::TaskManager.Task::(_debug) Task 9de89b0b-3080-4d8f-a540-9de3237eb831: moving from state preparing -> state finished >Thread-15912390::DEBUG::2013-11-20 02:31:51,664::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912390::DEBUG::2013-11-20 02:31:51,665::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912390::DEBUG::2013-11-20 02:31:51,665::task::495::TaskManager.Task::(_debug) Task 9de89b0b-3080-4d8f-a540-9de3237eb831: ref 0 aborting False >Thread-15912390::INFO::2013-11-20 02:31:51,666::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000427007675171', 'lastCheck': 1384914711.2783451, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148391723633', 'lastCheck': 1384914702.7506361, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00133585929871', 'lastCheck': 1384914702.752203, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000707864761353', 'lastCheck': 1384914708.9284389, 'valid': True, 'code': 0}} >Thread-15912386::DEBUG::2013-11-20 02:31:52,085::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912386::DEBUG::2013-11-20 02:31:52,089::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15912386::DEBUG::2013-11-20 02:31:52,090::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_ID=1', u'POOL_SPM_LVER=286', u'_SHA_CKSUM=09323c06acf399057c7d77c795a76a028bc8ee27'] >Thread-15912386::DEBUG::2013-11-20 02:31:52,097::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`5ada4500-006f-4070-9787-d02938cfad60`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1297' at 'public_getStorageDomainInfo' >Thread-15912386::DEBUG::2013-11-20 02:31:52,098::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912386::DEBUG::2013-11-20 02:31:52,099::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912386::DEBUG::2013-11-20 02:31:52,100::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`5ada4500-006f-4070-9787-d02938cfad60`::Granted request >Thread-15912386::DEBUG::2013-11-20 02:31:52,101::task::495::TaskManager.Task::(_debug) Task 395eff4f-2812-47ae-8751-927442be49d6: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912386::DEBUG::2013-11-20 02:31:52,102::task::495::TaskManager.Task::(_debug) Task 395eff4f-2812-47ae-8751-927442be49d6: ref 1 aborting False >Thread-15912386::WARNING::2013-11-20 02:31:52,104::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912386::DEBUG::2013-11-20 02:31:52,105::task::495::TaskManager.Task::(_debug) Task 395eff4f-2812-47ae-8751-927442be49d6: finished: {'info': {'uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'vguuid': 'aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', 'lver': 286, 'state': 'OK', 'version': '2', 'role': 'Master', 'master_ver': 1, 'spm_id': 1, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'OS'}} >Thread-15912386::DEBUG::2013-11-20 02:31:52,106::task::495::TaskManager.Task::(_debug) Task 395eff4f-2812-47ae-8751-927442be49d6: moving from state preparing -> state finished >Thread-15912386::DEBUG::2013-11-20 02:31:52,106::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912386::DEBUG::2013-11-20 02:31:52,107::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912386::DEBUG::2013-11-20 02:31:52,108::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912386::DEBUG::2013-11-20 02:31:52,108::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912386::DEBUG::2013-11-20 02:31:52,109::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912386::DEBUG::2013-11-20 02:31:52,109::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912386::DEBUG::2013-11-20 02:31:52,110::task::495::TaskManager.Task::(_debug) Task 395eff4f-2812-47ae-8751-927442be49d6: ref 0 aborting False >Thread-15912386::INFO::2013-11-20 02:31:52,111::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'vguuid': 'aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', 'lver': 286, 'state': 'OK', 'version': '2', 'role': 'Master', 'master_ver': 1, 'spm_id': 1, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'OS'}} >Thread-15912386::DEBUG::2013-11-20 02:31:52,125::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/libexec/vdsm/hooks/before_vm_start/50_vhostmd' (cwd None) >Thread-15912386::DEBUG::2013-11-20 02:31:52,598::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912386::INFO::2013-11-20 02:31:52,599::hooks::51::root::(_runHooksDir) >Thread-15912386::DEBUG::2013-11-20 02:31:52,600::libvirtvm::1077::vm.Vm::(_run) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::<?xml version="1.0" encoding="utf-8"?> ><domain type="kvm"> > <name>KLEE-DC01</name> > <uuid>98ec88b5-2c08-4ff2-b607-c3f4e281f6af</uuid> > <memory>2097152</memory> > <currentMemory>2097152</currentMemory> > <vcpu>4</vcpu> > <devices> > <disk device="disk" type="block"> > <source dev="/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/d8be6e76-2f7f-4b56-9ada-c3fd543a9794/038d75ec-9421-4d79-9865-0dbebd5271d9"/> > <target bus="virtio" dev="vda"/> > <serial>56-9ada-c3fd543a9794</serial> > <driver cache="none" error_policy="stop" io="native" name="qemu" type="raw"/> > </disk> > <disk device="cdrom" type="file"> > <target bus="ide" dev="hdc"/> > </disk> > <controller index="0" ports="16" type="virtio-serial"/> > <channel type="unix"> > <target name="com.redhat.rhevm.vdsm" type="virtio"/> > <source mode="bind" path="/var/lib/libvirt/qemu/channels/KLEE-DC01.com.redhat.rhevm.vdsm"/> > </channel> > <memballoon model="none"/> > <interface type="bridge"> > <mac address="00:1a:4a:a8:64:06"/> > <model type="virtio"/> > <source bridge="rhevm"/> > </interface> > <input bus="ps2" type="mouse"/> > <video> > <model heads="1" type="qxl" vram="65536"/> > </video> > <channel type="spicevmc"> > <target name="com.redhat.spice.0" type="virtio"/> > </channel> > <graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice"> > <channel mode="secure" name="main"/> > <channel mode="secure" name="inputs"/> > </graphics> > </devices> > <os> > <type arch="x86_64" machine="rhel6.2.0">hvm</type> > <boot dev="hd"/> > <smbios mode="sysinfo"/> > </os> > <sysinfo type="smbios"> > <system> > <entry name="manufacturer">Red Hat</entry> > <entry name="product">RHEV Hypervisor</entry> > <entry name="version">6.3-20121012.0.el6_3</entry> > <entry name="serial">30343536-3138-5A43-4A32-333630373442_AC:16:2D:76:EB:F0</entry> > <entry name="uuid">98ec88b5-2c08-4ff2-b607-c3f4e281f6af</entry> > </system> > </sysinfo> > <clock adjustment="3600" offset="variable"> > <timer name="rtc" tickpolicy="catchup"/> > </clock> > <features> > <acpi/> > </features> > <cpu match="exact"> > <model>Westmere</model> > <topology cores="2" sockets="2" threads="1"/> > </cpu> ></domain> > >Thread-1024767::ERROR::2013-11-20 02:31:53,344::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >libvirtEventLoop::DEBUG::2013-11-20 02:31:53,606::libvirtvm::1401::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::event Started detail 0 opaque None >Thread-15912386::DEBUG::2013-11-20 02:31:53,653::utils::307::vm.Vm::(start) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Start statistics collection >Thread-15912392::DEBUG::2013-11-20 02:31:53,654::utils::336::vm.Vm::(run) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Stats thread started >Thread-15912393::DEBUG::2013-11-20 02:31:53,656::guestIF::47::root::(_connect) Attempting connection to /var/lib/libvirt/qemu/channels/KLEE-DC01.com.redhat.rhevm.vdsm >Thread-15912392::INFO::2013-11-20 02:31:53,658::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912393::DEBUG::2013-11-20 02:31:53,659::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /usr/share/vdsm/prepare-vmchannel /var/lib/libvirt/qemu/channels/KLEE-DC01.com.redhat.rhevm.vdsm' (cwd None) >Thread-15912392::DEBUG::2013-11-20 02:31:53,660::task::495::TaskManager.Task::(_debug) Task 6ec0436c-c099-4b94-aebe-f8a3e780adc1: moving from state init -> state preparing >Thread-15912386::WARNING::2013-11-20 02:31:53,661::libvirtvm::1124::vm.Vm::(_readPauseCode) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::_readPauseCode unsupported by libvirt vm >Thread-15912392::DEBUG::2013-11-20 02:31:53,883::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`5ffdd12f-f610-4c1c-bc65-3f7982027deb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912386::DEBUG::2013-11-20 02:31:53,891::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /usr/share/vdsm/get-vm-pid KLEE-DC01' (cwd None) >Thread-15912392::DEBUG::2013-11-20 02:31:53,896::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:31:54,157::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:31:54,159::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`5ffdd12f-f610-4c1c-bc65-3f7982027deb`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:31:54,160::task::495::TaskManager.Task::(_debug) Task 6ec0436c-c099-4b94-aebe-f8a3e780adc1: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:31:54,161::task::495::TaskManager.Task::(_debug) Task 6ec0436c-c099-4b94-aebe-f8a3e780adc1: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:31:54,162::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15912392::DEBUG::2013-11-20 02:31:54,162::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15912393::DEBUG::2013-11-20 02:31:54,549::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912386::DEBUG::2013-11-20 02:31:54,552::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912393::DEBUG::2013-11-20 02:31:54,555::guestIF::53::root::(_connect) connected to /var/lib/libvirt/qemu/channels/KLEE-DC01.com.redhat.rhevm.vdsm >Thread-15912386::DEBUG::2013-11-20 02:31:54,566::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /sbin/service ksmtuned retune' (cwd None) >Thread-15912386::DEBUG::2013-11-20 02:31:55,092::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912386::DEBUG::2013-11-20 02:31:55,095::vm::358::vm.Vm::(_startUnderlyingVm) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::_ongoingCreations released >Thread-15912392::DEBUG::2013-11-20 02:31:55,167::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912392::DEBUG::2013-11-20 02:31:55,408::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15912392::DEBUG::2013-11-20 02:31:55,409::task::495::TaskManager.Task::(_debug) Task 6ec0436c-c099-4b94-aebe-f8a3e780adc1: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:31:55,409::task::495::TaskManager.Task::(_debug) Task 6ec0436c-c099-4b94-aebe-f8a3e780adc1: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:31:55,410::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:31:55,410::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:31:55,410::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:31:55,411::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:31:55,411::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:31:55,411::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:31:55,412::task::495::TaskManager.Task::(_debug) Task 6ec0436c-c099-4b94-aebe-f8a3e780adc1: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:31:55,412::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912396::DEBUG::2013-11-20 02:31:57,913::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912396::DEBUG::2013-11-20 02:31:57,914::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::CPU stats not available >Thread-15912396::DEBUG::2013-11-20 02:31:57,914::libvirtvm::202::vm.Vm::(_getNetworkStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Network stats not available >Thread-15912396::DEBUG::2013-11-20 02:31:57,921::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk stats not available >Thread-15912396::DEBUG::2013-11-20 02:31:57,922::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk latency not available >Thread-15912396::DEBUG::2013-11-20 02:31:57,923::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': u'5901', 'timeOffset': '3600', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet0': {'state': 'unknown', 'macAddr': u'00:1a:4a:a8:64:06', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '0', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '0', 'truesize': '64424509440'}}, 'boot': 'c', 'statsAge': '0.47', 'cpuIdle': '100.00', 'elapsedTime': '13', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15912398::DEBUG::2013-11-20 02:32:01,014::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912398::DEBUG::2013-11-20 02:32:01,015::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::CPU stats not available >Thread-15912398::DEBUG::2013-11-20 02:32:01,015::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk stats not available >Thread-15912398::DEBUG::2013-11-20 02:32:01,016::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk latency not available >Thread-15912398::DEBUG::2013-11-20 02:32:01,016::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': u'5901', 'timeOffset': '3600', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '0', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '0', 'truesize': '64424509440'}}, 'boot': 'c', 'statsAge': '0.57', 'cpuIdle': '100.00', 'elapsedTime': '16', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15912399::DEBUG::2013-11-20 02:32:02,089::clientIF::54::vds::(wrapper) [192.168.100.135]::call setVmTicket with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'kcPKiiNPOkVa', '120') {} >Thread-15912399::DEBUG::2013-11-20 02:32:02,124::clientIF::59::vds::(wrapper) return setVmTicket with {'status': {'message': 'Done', 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:32:03,531::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912401::DEBUG::2013-11-20 02:32:04,058::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912401::DEBUG::2013-11-20 02:32:04,059::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::CPU stats not available >Thread-15912401::DEBUG::2013-11-20 02:32:04,060::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk stats not available >Thread-15912401::DEBUG::2013-11-20 02:32:04,061::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk latency not available >Thread-15912401::DEBUG::2013-11-20 02:32:04,061::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': u'5901', 'timeOffset': '3600', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '0', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '0', 'truesize': '64424509440'}}, 'boot': 'c', 'statsAge': '0.62', 'cpuIdle': '100.00', 'elapsedTime': '19', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >libvirtEventLoop::DEBUG::2013-11-20 02:32:05,922::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::graphics event phase 0 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >libvirtEventLoop::DEBUG::2013-11-20 02:32:05,923::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::graphics event phase 1 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >libvirtEventLoop::DEBUG::2013-11-20 02:32:05,924::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::graphics event phase 0 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >libvirtEventLoop::DEBUG::2013-11-20 02:32:05,925::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::graphics event phase 1 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >libvirtEventLoop::DEBUG::2013-11-20 02:32:06,007::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::graphics event phase 0 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >libvirtEventLoop::DEBUG::2013-11-20 02:32:06,008::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::graphics event phase 1 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >libvirtEventLoop::DEBUG::2013-11-20 02:32:06,072::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::graphics event phase 0 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >libvirtEventLoop::DEBUG::2013-11-20 02:32:06,073::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::graphics event phase 1 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >libvirtEventLoop::DEBUG::2013-11-20 02:32:06,154::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::graphics event phase 0 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >libvirtEventLoop::DEBUG::2013-11-20 02:32:06,155::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::graphics event phase 1 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >Thread-15912402::INFO::2013-11-20 02:32:07,126::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912402::DEBUG::2013-11-20 02:32:07,127::task::495::TaskManager.Task::(_debug) Task e8d78412-afae-46d9-ac69-0a1dd0ad5094: moving from state init -> state preparing >Thread-15912402::DEBUG::2013-11-20 02:32:07,127::task::495::TaskManager.Task::(_debug) Task e8d78412-afae-46d9-ac69-0a1dd0ad5094: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000308036804199', 'lastCheck': 1384914721.5232639, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00181698799133', 'lastCheck': 1384914723.5588739, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00177907943726', 'lastCheck': 1384914723.5344419, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00046706199646', 'lastCheck': 1384914719.1628461, 'valid': True, 'code': 0}} >Thread-15912402::DEBUG::2013-11-20 02:32:07,128::task::495::TaskManager.Task::(_debug) Task e8d78412-afae-46d9-ac69-0a1dd0ad5094: moving from state preparing -> state finished >Thread-15912402::DEBUG::2013-11-20 02:32:07,129::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912402::DEBUG::2013-11-20 02:32:07,130::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912402::DEBUG::2013-11-20 02:32:07,131::task::495::TaskManager.Task::(_debug) Task e8d78412-afae-46d9-ac69-0a1dd0ad5094: ref 0 aborting False >Thread-15912402::INFO::2013-11-20 02:32:07,131::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000308036804199', 'lastCheck': 1384914721.5232639, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00181698799133', 'lastCheck': 1384914723.5588739, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00177907943726', 'lastCheck': 1384914723.5344419, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00046706199646', 'lastCheck': 1384914719.1628461, 'valid': True, 'code': 0}} >Thread-15912403::DEBUG::2013-11-20 02:32:07,146::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::CPU stats not available >Thread-15912403::DEBUG::2013-11-20 02:32:07,147::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk stats not available >Thread-15912403::DEBUG::2013-11-20 02:32:07,148::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk latency not available >Thread-15912405::DEBUG::2013-11-20 02:32:10,406::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912405::DEBUG::2013-11-20 02:32:10,407::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::CPU stats not available >Thread-15912405::DEBUG::2013-11-20 02:32:10,407::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk stats not available >Thread-15912405::DEBUG::2013-11-20 02:32:10,408::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk latency not available >Thread-15912405::DEBUG::2013-11-20 02:32:10,409::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': u'5901', 'timeOffset': '3600', 'pauseCode': 'NOERR', 'clientIp': '192.168.50.51', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '0', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '0', 'truesize': '64424509440'}}, 'boot': 'c', 'statsAge': '0.96', 'cpuIdle': '100.00', 'elapsedTime': '25', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15912407::DEBUG::2013-11-20 02:32:13,524::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912407::DEBUG::2013-11-20 02:32:13,525::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk stats not available >Thread-15912407::DEBUG::2013-11-20 02:32:13,528::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk latency not available >Thread-15912407::DEBUG::2013-11-20 02:32:13,529::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': u'5901', 'timeOffset': '3600', 'pauseCode': 'NOERR', 'clientIp': '192.168.50.51', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'monitorResponse': '0', 'cpuUser': '79.82', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '0', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '0', 'truesize': '64424509440'}}, 'boot': 'c', 'statsAge': '0.08', 'cpuIdle': '20.18', 'elapsedTime': '28', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-1024767::ERROR::2013-11-20 02:32:13,549::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912409::DEBUG::2013-11-20 02:32:16,571::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912409::DEBUG::2013-11-20 02:32:16,572::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk stats not available >Thread-15912409::DEBUG::2013-11-20 02:32:16,573::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk latency not available >Thread-15912409::DEBUG::2013-11-20 02:32:16,574::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': u'5901', 'timeOffset': '3600', 'pauseCode': 'NOERR', 'clientIp': '192.168.50.51', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'monitorResponse': '0', 'cpuUser': '79.82', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '0', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '0', 'truesize': '64424509440'}}, 'boot': 'c', 'statsAge': '1.12', 'cpuIdle': '20.18', 'elapsedTime': '31', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15912411::DEBUG::2013-11-20 02:32:19,658::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912411::DEBUG::2013-11-20 02:32:19,659::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk stats not available >Thread-15912411::DEBUG::2013-11-20 02:32:19,660::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk latency not available >Thread-15912411::DEBUG::2013-11-20 02:32:19,660::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': u'5901', 'timeOffset': '3600', 'pauseCode': 'NOERR', 'clientIp': '192.168.50.51', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'monitorResponse': '0', 'cpuUser': '79.82', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '0', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '0', 'truesize': '64424509440'}}, 'boot': 'c', 'statsAge': '0.21', 'cpuIdle': '20.18', 'elapsedTime': '34', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15912412::INFO::2013-11-20 02:32:22,735::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912412::DEBUG::2013-11-20 02:32:22,736::task::495::TaskManager.Task::(_debug) Task 94604849-0316-4b70-a7e5-106dcb5ae511: moving from state init -> state preparing >Thread-15912412::DEBUG::2013-11-20 02:32:22,737::task::495::TaskManager.Task::(_debug) Task 94604849-0316-4b70-a7e5-106dcb5ae511: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00124311447144', 'lastCheck': 1384914741.5506051, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00160789489746', 'lastCheck': 1384914733.574085, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.001296043396', 'lastCheck': 1384914733.55141, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000448942184448', 'lastCheck': 1384914739.184212, 'valid': True, 'code': 0}} >Thread-15912412::DEBUG::2013-11-20 02:32:22,738::task::495::TaskManager.Task::(_debug) Task 94604849-0316-4b70-a7e5-106dcb5ae511: moving from state preparing -> state finished >Thread-15912412::DEBUG::2013-11-20 02:32:22,738::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912412::DEBUG::2013-11-20 02:32:22,739::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912412::DEBUG::2013-11-20 02:32:22,739::task::495::TaskManager.Task::(_debug) Task 94604849-0316-4b70-a7e5-106dcb5ae511: ref 0 aborting False >Thread-15912412::INFO::2013-11-20 02:32:22,740::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00124311447144', 'lastCheck': 1384914741.5506051, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00160789489746', 'lastCheck': 1384914733.574085, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.001296043396', 'lastCheck': 1384914733.55141, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000448942184448', 'lastCheck': 1384914739.184212, 'valid': True, 'code': 0}} >Thread-15912413::DEBUG::2013-11-20 02:32:22,762::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk stats not available >Thread-15912413::DEBUG::2013-11-20 02:32:22,763::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk latency not available >Thread-1024767::ERROR::2013-11-20 02:32:23,565::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912415::DEBUG::2013-11-20 02:32:25,952::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912415::DEBUG::2013-11-20 02:32:25,953::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk stats not available >Thread-15912415::DEBUG::2013-11-20 02:32:25,954::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk latency not available >Thread-15912415::DEBUG::2013-11-20 02:32:25,955::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': u'5901', 'timeOffset': '3600', 'pauseCode': 'NOERR', 'clientIp': '192.168.50.51', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'monitorResponse': '0', 'cpuUser': '79.95', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '0', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '0', 'truesize': '64424509440'}}, 'boot': 'c', 'statsAge': '0.50', 'cpuIdle': '20.05', 'elapsedTime': '41', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15912417::DEBUG::2013-11-20 02:32:29,043::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912417::DEBUG::2013-11-20 02:32:29,044::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk stats not available >Thread-15912417::DEBUG::2013-11-20 02:32:29,044::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk latency not available >Thread-15912417::DEBUG::2013-11-20 02:32:29,045::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': u'5901', 'timeOffset': '3600', 'pauseCode': 'NOERR', 'clientIp': '192.168.50.51', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'monitorResponse': '0', 'cpuUser': '79.95', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '0', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '0', 'truesize': '64424509440'}}, 'boot': 'c', 'statsAge': '1.58', 'cpuIdle': '20.05', 'elapsedTime': '44', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15912419::DEBUG::2013-11-20 02:32:32,114::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912419::DEBUG::2013-11-20 02:32:32,115::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk stats not available >Thread-15912419::DEBUG::2013-11-20 02:32:32,115::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk latency not available >Thread-15912419::DEBUG::2013-11-20 02:32:32,116::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': u'5901', 'timeOffset': '3600', 'pauseCode': 'NOERR', 'clientIp': '192.168.50.51', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'monitorResponse': '0', 'cpuUser': '79.95', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '0', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '0', 'truesize': '64424509440'}}, 'boot': 'c', 'statsAge': '0.65', 'cpuIdle': '20.05', 'elapsedTime': '47', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-1024767::ERROR::2013-11-20 02:32:33,582::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912421::DEBUG::2013-11-20 02:32:35,203::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912421::DEBUG::2013-11-20 02:32:35,204::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk stats not available >Thread-15912421::DEBUG::2013-11-20 02:32:35,205::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk latency not available >Thread-15912421::DEBUG::2013-11-20 02:32:35,205::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': u'5901', 'timeOffset': '3600', 'pauseCode': 'NOERR', 'clientIp': '192.168.50.51', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'monitorResponse': '0', 'cpuUser': '79.95', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '0', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '0', 'truesize': '64424509440'}}, 'boot': 'c', 'statsAge': '1.74', 'cpuIdle': '20.05', 'elapsedTime': '50', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15912422::INFO::2013-11-20 02:32:38,271::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912422::DEBUG::2013-11-20 02:32:38,272::task::495::TaskManager.Task::(_debug) Task 57fa1083-97d6-4e0b-97a5-b0ef9b5658d3: moving from state init -> state preparing >Thread-15912422::DEBUG::2013-11-20 02:32:38,273::task::495::TaskManager.Task::(_debug) Task 57fa1083-97d6-4e0b-97a5-b0ef9b5658d3: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000263929367065', 'lastCheck': 1384914751.562943, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00131702423096', 'lastCheck': 1384914753.6043501, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00151205062866', 'lastCheck': 1384914753.5854781, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000293016433716', 'lastCheck': 1384914749.194684, 'valid': True, 'code': 0}} >Thread-15912422::DEBUG::2013-11-20 02:32:38,274::task::495::TaskManager.Task::(_debug) Task 57fa1083-97d6-4e0b-97a5-b0ef9b5658d3: moving from state preparing -> state finished >Thread-15912422::DEBUG::2013-11-20 02:32:38,274::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912422::DEBUG::2013-11-20 02:32:38,275::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912422::DEBUG::2013-11-20 02:32:38,275::task::495::TaskManager.Task::(_debug) Task 57fa1083-97d6-4e0b-97a5-b0ef9b5658d3: ref 0 aborting False >Thread-15912422::INFO::2013-11-20 02:32:38,276::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000263929367065', 'lastCheck': 1384914751.562943, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00131702423096', 'lastCheck': 1384914753.6043501, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00151205062866', 'lastCheck': 1384914753.5854781, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000293016433716', 'lastCheck': 1384914749.194684, 'valid': True, 'code': 0}} >Thread-15912423::DEBUG::2013-11-20 02:32:38,310::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk stats not available >Thread-15912423::DEBUG::2013-11-20 02:32:38,311::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk latency not available >Thread-15912425::DEBUG::2013-11-20 02:32:41,486::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912425::DEBUG::2013-11-20 02:32:41,487::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk stats not available >Thread-15912425::DEBUG::2013-11-20 02:32:41,487::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk latency not available >Thread-15912425::DEBUG::2013-11-20 02:32:41,488::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': u'5901', 'timeOffset': '3600', 'pauseCode': 'NOERR', 'clientIp': '192.168.50.51', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'monitorResponse': '0', 'cpuUser': '79.94', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '0', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '0', 'truesize': '64424509440'}}, 'boot': 'c', 'statsAge': '0.02', 'cpuIdle': '20.06', 'elapsedTime': '56', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-1024767::ERROR::2013-11-20 02:32:43,599::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912427::DEBUG::2013-11-20 02:32:44,544::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912427::DEBUG::2013-11-20 02:32:44,545::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk stats not available >Thread-15912427::DEBUG::2013-11-20 02:32:44,546::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk latency not available >Thread-15912427::DEBUG::2013-11-20 02:32:44,546::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': u'5901', 'timeOffset': '3600', 'pauseCode': 'NOERR', 'clientIp': '192.168.50.51', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'monitorResponse': '0', 'cpuUser': '79.94', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '0', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '0', 'truesize': '64424509440'}}, 'boot': 'c', 'statsAge': '1.07', 'cpuIdle': '20.06', 'elapsedTime': '59', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15912429::DEBUG::2013-11-20 02:32:47,623::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912429::DEBUG::2013-11-20 02:32:47,624::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk stats not available >Thread-15912429::DEBUG::2013-11-20 02:32:47,625::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk latency not available >Thread-15912429::DEBUG::2013-11-20 02:32:47,625::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayIp': '0', 'displayPort': u'5900', 'session': 'Unknown', 'displaySecurePort': u'5901', 'timeOffset': '3600', 'pauseCode': 'NOERR', 'clientIp': '192.168.50.51', 'kvmEnable': 'true', 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'monitorResponse': '0', 'cpuUser': '79.94', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '0', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '0', 'truesize': '64424509440'}}, 'boot': 'c', 'statsAge': '0.15', 'cpuIdle': '20.06', 'elapsedTime': '62', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-1024767::ERROR::2013-11-20 02:32:53,616::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912431::INFO::2013-11-20 02:32:53,745::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912431::DEBUG::2013-11-20 02:32:53,746::task::495::TaskManager.Task::(_debug) Task 839f2a56-ae75-45e9-9300-c08989030ea6: moving from state init -> state preparing >Thread-15912431::DEBUG::2013-11-20 02:32:53,746::task::495::TaskManager.Task::(_debug) Task 839f2a56-ae75-45e9-9300-c08989030ea6: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000406980514526', 'lastCheck': 1384914771.5869241, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00141286849976', 'lastCheck': 1384914773.6329861, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00130796432495', 'lastCheck': 1384914773.618912, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000362157821655', 'lastCheck': 1384914769.2168159, 'valid': True, 'code': 0}} >Thread-15912431::DEBUG::2013-11-20 02:32:53,747::task::495::TaskManager.Task::(_debug) Task 839f2a56-ae75-45e9-9300-c08989030ea6: moving from state preparing -> state finished >Thread-15912431::DEBUG::2013-11-20 02:32:53,748::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912431::DEBUG::2013-11-20 02:32:53,748::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912431::DEBUG::2013-11-20 02:32:53,749::task::495::TaskManager.Task::(_debug) Task 839f2a56-ae75-45e9-9300-c08989030ea6: ref 0 aborting False >Thread-15912431::INFO::2013-11-20 02:32:53,749::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000406980514526', 'lastCheck': 1384914771.5869241, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00141286849976', 'lastCheck': 1384914773.6329861, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00130796432495', 'lastCheck': 1384914773.618912, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000362157821655', 'lastCheck': 1384914769.2168159, 'valid': True, 'code': 0}} >Thread-15912432::DEBUG::2013-11-20 02:32:53,764::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk stats not available >Thread-15912432::DEBUG::2013-11-20 02:32:53,764::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Disk latency not available >Thread-15912392::INFO::2013-11-20 02:32:55,477::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:32:55,478::task::495::TaskManager.Task::(_debug) Task 803f2594-927a-4b41-b978-8656e4599ce5: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:32:55,485::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`1270859c-8d70-489e-8636-3bee97269223`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:32:55,485::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:32:55,486::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:32:55,487::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`1270859c-8d70-489e-8636-3bee97269223`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:32:55,487::task::495::TaskManager.Task::(_debug) Task 803f2594-927a-4b41-b978-8656e4599ce5: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:32:55,488::task::495::TaskManager.Task::(_debug) Task 803f2594-927a-4b41-b978-8656e4599ce5: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:32:55,489::task::495::TaskManager.Task::(_debug) Task 803f2594-927a-4b41-b978-8656e4599ce5: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:32:55,489::task::495::TaskManager.Task::(_debug) Task 803f2594-927a-4b41-b978-8656e4599ce5: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:32:55,490::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:32:55,491::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:32:55,491::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:32:55,492::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:32:55,492::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:32:55,493::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:32:55,493::task::495::TaskManager.Task::(_debug) Task 803f2594-927a-4b41-b978-8656e4599ce5: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:32:55,494::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-1024767::ERROR::2013-11-20 02:33:03,633::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912437::INFO::2013-11-20 02:33:08,946::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912437::DEBUG::2013-11-20 02:33:08,947::task::495::TaskManager.Task::(_debug) Task ac85250e-89fa-43fa-a348-3fdf093a6bf4: moving from state init -> state preparing >Thread-15912437::DEBUG::2013-11-20 02:33:08,947::task::495::TaskManager.Task::(_debug) Task ac85250e-89fa-43fa-a348-3fdf093a6bf4: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000337839126587', 'lastCheck': 1384914781.5997419, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00168299674988', 'lastCheck': 1384914783.645865, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00189614295959', 'lastCheck': 1384914783.636493, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000407934188843', 'lastCheck': 1384914779.227442, 'valid': True, 'code': 0}} >Thread-15912437::DEBUG::2013-11-20 02:33:08,948::task::495::TaskManager.Task::(_debug) Task ac85250e-89fa-43fa-a348-3fdf093a6bf4: moving from state preparing -> state finished >Thread-15912437::DEBUG::2013-11-20 02:33:08,949::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912437::DEBUG::2013-11-20 02:33:08,949::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912437::DEBUG::2013-11-20 02:33:08,950::task::495::TaskManager.Task::(_debug) Task ac85250e-89fa-43fa-a348-3fdf093a6bf4: ref 0 aborting False >Thread-15912437::INFO::2013-11-20 02:33:08,950::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000337839126587', 'lastCheck': 1384914781.5997419, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00168299674988', 'lastCheck': 1384914783.645865, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00189614295959', 'lastCheck': 1384914783.636493, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000407934188843', 'lastCheck': 1384914779.227442, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:33:13,649::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-1024767::ERROR::2013-11-20 02:33:23,666::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912443::INFO::2013-11-20 02:33:24,197::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912443::DEBUG::2013-11-20 02:33:24,198::task::495::TaskManager.Task::(_debug) Task 521f764b-780b-4729-bd13-1413a0ee506c: moving from state init -> state preparing >Thread-15912443::DEBUG::2013-11-20 02:33:24,198::task::495::TaskManager.Task::(_debug) Task 521f764b-780b-4729-bd13-1413a0ee506c: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000343084335327', 'lastCheck': 1384914801.6248159, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00121307373047', 'lastCheck': 1384914803.6749339, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00188398361206', 'lastCheck': 1384914803.669872, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000363111495972', 'lastCheck': 1384914799.250772, 'valid': True, 'code': 0}} >Thread-15912443::DEBUG::2013-11-20 02:33:24,199::task::495::TaskManager.Task::(_debug) Task 521f764b-780b-4729-bd13-1413a0ee506c: moving from state preparing -> state finished >Thread-15912443::DEBUG::2013-11-20 02:33:24,200::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912443::DEBUG::2013-11-20 02:33:24,200::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912443::DEBUG::2013-11-20 02:33:24,201::task::495::TaskManager.Task::(_debug) Task 521f764b-780b-4729-bd13-1413a0ee506c: ref 0 aborting False >Thread-15912443::INFO::2013-11-20 02:33:24,201::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000343084335327', 'lastCheck': 1384914801.6248159, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00121307373047', 'lastCheck': 1384914803.6749339, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00188398361206', 'lastCheck': 1384914803.669872, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000363111495972', 'lastCheck': 1384914799.250772, 'valid': True, 'code': 0}} >Thread-15912393::DEBUG::2013-11-20 02:33:28,992::guestIF::97::vm.Vm::(_handleMessage) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::u'None' >Thread-42840::DEBUG::2013-11-20 02:33:29,262::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 02:33:30,013::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 02:33:33,683::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912449::INFO::2013-11-20 02:33:39,458::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912449::DEBUG::2013-11-20 02:33:39,459::task::495::TaskManager.Task::(_debug) Task 266d9c5b-5aa3-4ba2-9ef8-417c7f70594c: moving from state init -> state preparing >Thread-15912449::DEBUG::2013-11-20 02:33:39,460::task::495::TaskManager.Task::(_debug) Task 266d9c5b-5aa3-4ba2-9ef8-417c7f70594c: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000313997268677', 'lastCheck': 1384914811.6371779, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142097473145', 'lastCheck': 1384914813.6899159, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00150394439697', 'lastCheck': 1384914813.686748, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000325202941895', 'lastCheck': 1384914810.015805, 'valid': True, 'code': 0}} >Thread-15912449::DEBUG::2013-11-20 02:33:39,461::task::495::TaskManager.Task::(_debug) Task 266d9c5b-5aa3-4ba2-9ef8-417c7f70594c: moving from state preparing -> state finished >Thread-15912449::DEBUG::2013-11-20 02:33:39,461::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912449::DEBUG::2013-11-20 02:33:39,462::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912449::DEBUG::2013-11-20 02:33:39,462::task::495::TaskManager.Task::(_debug) Task 266d9c5b-5aa3-4ba2-9ef8-417c7f70594c: ref 0 aborting False >Thread-15912449::INFO::2013-11-20 02:33:39,463::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000313997268677', 'lastCheck': 1384914811.6371779, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142097473145', 'lastCheck': 1384914813.6899159, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00150394439697', 'lastCheck': 1384914813.686748, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000325202941895', 'lastCheck': 1384914810.015805, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:33:43,702::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-1024767::ERROR::2013-11-20 02:33:53,722::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912455::INFO::2013-11-20 02:33:54,846::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912455::DEBUG::2013-11-20 02:33:54,847::task::495::TaskManager.Task::(_debug) Task df8171a5-2c4e-4ee9-b779-2fc8e979e8ce: moving from state init -> state preparing >Thread-15912455::DEBUG::2013-11-20 02:33:54,848::task::495::TaskManager.Task::(_debug) Task df8171a5-2c4e-4ee9-b779-2fc8e979e8ce: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000281095504761', 'lastCheck': 1384914831.660831, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00275802612305', 'lastCheck': 1384914833.7244861, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00133013725281', 'lastCheck': 1384914833.7249589, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000338792800903', 'lastCheck': 1384914830.038132, 'valid': True, 'code': 0}} >Thread-15912455::DEBUG::2013-11-20 02:33:54,849::task::495::TaskManager.Task::(_debug) Task df8171a5-2c4e-4ee9-b779-2fc8e979e8ce: moving from state preparing -> state finished >Thread-15912455::DEBUG::2013-11-20 02:33:54,849::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912455::DEBUG::2013-11-20 02:33:54,850::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912455::DEBUG::2013-11-20 02:33:54,850::task::495::TaskManager.Task::(_debug) Task df8171a5-2c4e-4ee9-b779-2fc8e979e8ce: ref 0 aborting False >Thread-15912455::INFO::2013-11-20 02:33:54,851::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000281095504761', 'lastCheck': 1384914831.660831, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00275802612305', 'lastCheck': 1384914833.7244861, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00133013725281', 'lastCheck': 1384914833.7249589, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000338792800903', 'lastCheck': 1384914830.038132, 'valid': True, 'code': 0}} >Thread-15912392::INFO::2013-11-20 02:33:55,552::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:33:55,553::task::495::TaskManager.Task::(_debug) Task 3a72fc7b-8c83-4122-8cfa-77e3af4a3bd6: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:33:55,555::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0b14751a-61af-4086-81f8-3af496e15141`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:33:55,555::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:33:55,556::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:33:55,556::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0b14751a-61af-4086-81f8-3af496e15141`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:33:55,557::task::495::TaskManager.Task::(_debug) Task 3a72fc7b-8c83-4122-8cfa-77e3af4a3bd6: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:33:55,558::task::495::TaskManager.Task::(_debug) Task 3a72fc7b-8c83-4122-8cfa-77e3af4a3bd6: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:33:55,559::task::495::TaskManager.Task::(_debug) Task 3a72fc7b-8c83-4122-8cfa-77e3af4a3bd6: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:33:55,559::task::495::TaskManager.Task::(_debug) Task 3a72fc7b-8c83-4122-8cfa-77e3af4a3bd6: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:33:55,560::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:33:55,560::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:33:55,561::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:33:55,562::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:33:55,562::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:33:55,563::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:33:55,563::task::495::TaskManager.Task::(_debug) Task 3a72fc7b-8c83-4122-8cfa-77e3af4a3bd6: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:33:55,564::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-1024767::ERROR::2013-11-20 02:34:03,741::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912461::INFO::2013-11-20 02:34:10,086::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912461::DEBUG::2013-11-20 02:34:10,087::task::495::TaskManager.Task::(_debug) Task 3256a0b4-1cce-42ef-96d0-5dae021708f9: moving from state init -> state preparing >Thread-15912461::DEBUG::2013-11-20 02:34:10,088::task::495::TaskManager.Task::(_debug) Task 3256a0b4-1cce-42ef-96d0-5dae021708f9: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000285148620605', 'lastCheck': 1384914841.6738191, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00238108634949', 'lastCheck': 1384914843.74283, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00187206268311', 'lastCheck': 1384914843.7442269, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000447988510132', 'lastCheck': 1384914850.060374, 'valid': True, 'code': 0}} >Thread-15912461::DEBUG::2013-11-20 02:34:10,089::task::495::TaskManager.Task::(_debug) Task 3256a0b4-1cce-42ef-96d0-5dae021708f9: moving from state preparing -> state finished >Thread-15912461::DEBUG::2013-11-20 02:34:10,089::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912461::DEBUG::2013-11-20 02:34:10,090::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912461::DEBUG::2013-11-20 02:34:10,090::task::495::TaskManager.Task::(_debug) Task 3256a0b4-1cce-42ef-96d0-5dae021708f9: ref 0 aborting False >Thread-15912461::INFO::2013-11-20 02:34:10,091::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000285148620605', 'lastCheck': 1384914841.6738191, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00238108634949', 'lastCheck': 1384914843.74283, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00187206268311', 'lastCheck': 1384914843.7442269, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000447988510132', 'lastCheck': 1384914850.060374, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:34:13,759::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-1024767::ERROR::2013-11-20 02:34:23,775::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912467::INFO::2013-11-20 02:34:25,354::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912467::DEBUG::2013-11-20 02:34:25,355::task::495::TaskManager.Task::(_debug) Task b0f19f19-3fb7-4945-b938-f01441268f58: moving from state init -> state preparing >Thread-15912467::DEBUG::2013-11-20 02:34:25,355::task::495::TaskManager.Task::(_debug) Task b0f19f19-3fb7-4945-b938-f01441268f58: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000262975692749', 'lastCheck': 1384914861.6985321, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00192618370056', 'lastCheck': 1384914863.7757189, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00140309333801', 'lastCheck': 1384914863.777935, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000370979309082', 'lastCheck': 1384914860.0713151, 'valid': True, 'code': 0}} >Thread-15912467::DEBUG::2013-11-20 02:34:25,356::task::495::TaskManager.Task::(_debug) Task b0f19f19-3fb7-4945-b938-f01441268f58: moving from state preparing -> state finished >Thread-15912467::DEBUG::2013-11-20 02:34:25,356::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912467::DEBUG::2013-11-20 02:34:25,357::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912467::DEBUG::2013-11-20 02:34:25,358::task::495::TaskManager.Task::(_debug) Task b0f19f19-3fb7-4945-b938-f01441268f58: ref 0 aborting False >Thread-15912467::INFO::2013-11-20 02:34:25,358::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000262975692749', 'lastCheck': 1384914861.6985321, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00192618370056', 'lastCheck': 1384914863.7757189, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00140309333801', 'lastCheck': 1384914863.777935, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000370979309082', 'lastCheck': 1384914860.0713151, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:34:33,792::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912473::INFO::2013-11-20 02:34:40,646::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912473::DEBUG::2013-11-20 02:34:40,646::task::495::TaskManager.Task::(_debug) Task 7e0d2356-615a-4752-8139-6a0851e2be65: moving from state init -> state preparing >Thread-15912473::DEBUG::2013-11-20 02:34:40,647::task::495::TaskManager.Task::(_debug) Task 7e0d2356-615a-4752-8139-6a0851e2be65: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000269889831543', 'lastCheck': 1384914871.7096601, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00178003311157', 'lastCheck': 1384914873.790519, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00152397155762', 'lastCheck': 1384914873.794816, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000419855117798', 'lastCheck': 1384914880.0925889, 'valid': True, 'code': 0}} >Thread-15912473::DEBUG::2013-11-20 02:34:40,648::task::495::TaskManager.Task::(_debug) Task 7e0d2356-615a-4752-8139-6a0851e2be65: moving from state preparing -> state finished >Thread-15912473::DEBUG::2013-11-20 02:34:40,648::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912473::DEBUG::2013-11-20 02:34:40,649::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912473::DEBUG::2013-11-20 02:34:40,649::task::495::TaskManager.Task::(_debug) Task 7e0d2356-615a-4752-8139-6a0851e2be65: ref 0 aborting False >Thread-15912473::INFO::2013-11-20 02:34:40,650::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000269889831543', 'lastCheck': 1384914871.7096601, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00178003311157', 'lastCheck': 1384914873.790519, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00152397155762', 'lastCheck': 1384914873.794816, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000419855117798', 'lastCheck': 1384914880.0925889, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:34:43,808::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-1024767::ERROR::2013-11-20 02:34:53,823::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912392::INFO::2013-11-20 02:34:55,633::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:34:55,634::task::495::TaskManager.Task::(_debug) Task 0301642b-ca01-41a6-80f1-548b430aec07: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:34:55,646::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`2da0ccb8-8479-495c-8297-8de06ace0cdd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:34:55,646::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:34:55,647::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:34:55,647::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`2da0ccb8-8479-495c-8297-8de06ace0cdd`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:34:55,648::task::495::TaskManager.Task::(_debug) Task 0301642b-ca01-41a6-80f1-548b430aec07: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:34:55,649::task::495::TaskManager.Task::(_debug) Task 0301642b-ca01-41a6-80f1-548b430aec07: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:34:55,649::task::495::TaskManager.Task::(_debug) Task 0301642b-ca01-41a6-80f1-548b430aec07: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:34:55,650::task::495::TaskManager.Task::(_debug) Task 0301642b-ca01-41a6-80f1-548b430aec07: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:34:55,650::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:34:55,650::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:34:55,651::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:34:55,651::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:34:55,652::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:34:55,652::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:34:55,653::task::495::TaskManager.Task::(_debug) Task 0301642b-ca01-41a6-80f1-548b430aec07: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:34:55,653::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912479::INFO::2013-11-20 02:34:55,902::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912479::DEBUG::2013-11-20 02:34:55,903::task::495::TaskManager.Task::(_debug) Task 72abc95c-79c5-4bff-85e1-6107a228d688: moving from state init -> state preparing >Thread-15912479::DEBUG::2013-11-20 02:34:55,904::task::495::TaskManager.Task::(_debug) Task 72abc95c-79c5-4bff-85e1-6107a228d688: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000383853912354', 'lastCheck': 1384914891.735106, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00287294387817', 'lastCheck': 1384914893.8232291, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00151920318604', 'lastCheck': 1384914893.8261559, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000380992889404', 'lastCheck': 1384914890.1039081, 'valid': True, 'code': 0}} >Thread-15912479::DEBUG::2013-11-20 02:34:55,905::task::495::TaskManager.Task::(_debug) Task 72abc95c-79c5-4bff-85e1-6107a228d688: moving from state preparing -> state finished >Thread-15912479::DEBUG::2013-11-20 02:34:55,905::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912479::DEBUG::2013-11-20 02:34:55,906::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912479::DEBUG::2013-11-20 02:34:55,906::task::495::TaskManager.Task::(_debug) Task 72abc95c-79c5-4bff-85e1-6107a228d688: ref 0 aborting False >Thread-15912479::INFO::2013-11-20 02:34:55,907::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000383853912354', 'lastCheck': 1384914891.735106, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00287294387817', 'lastCheck': 1384914893.8232291, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00151920318604', 'lastCheck': 1384914893.8261559, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000380992889404', 'lastCheck': 1384914890.1039081, 'valid': True, 'code': 0}} >Thread-15912482::DEBUG::2013-11-20 02:35:00,525::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912482::INFO::2013-11-20 02:35:00,526::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: validateStorageServerConnection, args: (domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{'connection': 'kairo.brd-klee.dk:/export', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36', 'port': ''}]) >Thread-15912482::DEBUG::2013-11-20 02:35:00,527::task::495::TaskManager.Task::(_debug) Task ca8dacce-45eb-4fc8-8240-1211413468f1: moving from state init -> state preparing >Thread-15912482::INFO::2013-11-20 02:35:00,527::storage_connection::127::Storage.ServerConnection::(validate) Request to validate NFS storage server >Thread-15912482::INFO::2013-11-20 02:35:00,528::storage_connection::53::Storage.ServerConnection::(__validateConnectionParams) conList=[{'connection': 'kairo.brd-klee.dk:/export', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36', 'port': ''}] >Thread-15912482::DEBUG::2013-11-20 02:35:00,532::fileUtils::83::Storage.Misc.excCmd::(mount) '/usr/bin/sudo -n /bin/mount -o soft,timeo=600,retrans=6,nosharecache,vers=3 -t nfs kairo.brd-klee.dk:/export /tmp/tmpXQYbRu' (cwd None) >Thread-15912482::DEBUG::2013-11-20 02:35:00,970::fileUtils::83::Storage.Misc.excCmd::(mount) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912482::DEBUG::2013-11-20 02:35:00,982::fileUtils::112::Storage.Misc.excCmd::(umount) '/usr/bin/sudo -n /bin/umount -t nfs -f /tmp/tmpXQYbRu' (cwd None) >Thread-15912482::DEBUG::2013-11-20 02:35:01,409::fileUtils::112::Storage.Misc.excCmd::(umount) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912482::DEBUG::2013-11-20 02:35:01,412::task::495::TaskManager.Task::(_debug) Task ca8dacce-45eb-4fc8-8240-1211413468f1: finished: {'statuslist': [{'status': 0, 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36'}]} >Thread-15912482::DEBUG::2013-11-20 02:35:01,413::task::495::TaskManager.Task::(_debug) Task ca8dacce-45eb-4fc8-8240-1211413468f1: moving from state preparing -> state finished >Thread-15912482::DEBUG::2013-11-20 02:35:01,413::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912482::DEBUG::2013-11-20 02:35:01,414::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912482::DEBUG::2013-11-20 02:35:01,414::task::495::TaskManager.Task::(_debug) Task ca8dacce-45eb-4fc8-8240-1211413468f1: ref 0 aborting False >Thread-15912482::INFO::2013-11-20 02:35:01,415::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: validateStorageServerConnection, Return response: {'status': {'message': 'OK', 'code': 0}, 'statuslist': [{'status': 0, 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36'}]} >Thread-15912483::DEBUG::2013-11-20 02:35:01,432::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912483::INFO::2013-11-20 02:35:01,432::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: connectStorageServer, args: (domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{'connection': 'kairo.brd-klee.dk:/export', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36', 'port': ''}]) >Thread-15912483::DEBUG::2013-11-20 02:35:01,433::task::495::TaskManager.Task::(_debug) Task 2202f945-2d24-4cbe-aade-94f1d0d17fa5: moving from state init -> state preparing >Thread-15912483::INFO::2013-11-20 02:35:01,433::storage_connection::95::Storage.ServerConnection::(connect) Request to connect NFS storage server >Thread-15912483::INFO::2013-11-20 02:35:01,433::storage_connection::53::Storage.ServerConnection::(__validateConnectionParams) conList=[{'connection': 'kairo.brd-klee.dk:/export', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36', 'port': ''}] >Thread-15912483::DEBUG::2013-11-20 02:35:01,445::lvm::547::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15912483::DEBUG::2013-11-20 02:35:01,445::lvm::549::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15912483::DEBUG::2013-11-20 02:35:01,446::lvm::559::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15912483::DEBUG::2013-11-20 02:35:01,447::lvm::561::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15912483::DEBUG::2013-11-20 02:35:01,447::lvm::580::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15912483::DEBUG::2013-11-20 02:35:01,448::lvm::582::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15912483::DEBUG::2013-11-20 02:35:01,449::task::495::TaskManager.Task::(_debug) Task 2202f945-2d24-4cbe-aade-94f1d0d17fa5: finished: {'statuslist': [{'status': 0, 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36'}]} >Thread-15912483::DEBUG::2013-11-20 02:35:01,449::task::495::TaskManager.Task::(_debug) Task 2202f945-2d24-4cbe-aade-94f1d0d17fa5: moving from state preparing -> state finished >Thread-15912483::DEBUG::2013-11-20 02:35:01,450::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912483::DEBUG::2013-11-20 02:35:01,450::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912483::DEBUG::2013-11-20 02:35:01,451::task::495::TaskManager.Task::(_debug) Task 2202f945-2d24-4cbe-aade-94f1d0d17fa5: ref 0 aborting False >Thread-15912483::INFO::2013-11-20 02:35:01,452::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: connectStorageServer, Return response: {'status': {'message': 'OK', 'code': 0}, 'statuslist': [{'status': 0, 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36'}]} >Thread-42838::DEBUG::2013-11-20 02:35:01,744::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-42838::DEBUG::2013-11-20 02:35:01,749::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 02:35:02,443::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-42838::DEBUG::2013-11-20 02:35:02,454::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-1024767::ERROR::2013-11-20 02:35:03,994::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-42840::DEBUG::2013-11-20 02:35:10,125::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-42840::DEBUG::2013-11-20 02:35:10,126::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 02:35:10,730::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-42840::DEBUG::2013-11-20 02:35:10,733::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15912487::INFO::2013-11-20 02:35:11,106::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912487::DEBUG::2013-11-20 02:35:11,107::task::495::TaskManager.Task::(_debug) Task bf50aa77-85fc-45f8-8056-16d1ade45536: moving from state init -> state preparing >Thread-15912487::DEBUG::2013-11-20 02:35:11,108::task::495::TaskManager.Task::(_debug) Task bf50aa77-85fc-45f8-8056-16d1ade45536: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000394821166992', 'lastCheck': 1384914902.457823, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0022439956665', 'lastCheck': 1384914903.9991341, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00251817703247', 'lastCheck': 1384914903.9989159, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000380039215088', 'lastCheck': 1384914910.735188, 'valid': True, 'code': 0}} >Thread-15912487::DEBUG::2013-11-20 02:35:11,109::task::495::TaskManager.Task::(_debug) Task bf50aa77-85fc-45f8-8056-16d1ade45536: moving from state preparing -> state finished >Thread-15912487::DEBUG::2013-11-20 02:35:11,109::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912487::DEBUG::2013-11-20 02:35:11,110::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912487::DEBUG::2013-11-20 02:35:11,111::task::495::TaskManager.Task::(_debug) Task bf50aa77-85fc-45f8-8056-16d1ade45536: ref 0 aborting False >Thread-15912487::INFO::2013-11-20 02:35:11,112::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000394821166992', 'lastCheck': 1384914902.457823, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0022439956665', 'lastCheck': 1384914903.9991341, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00251817703247', 'lastCheck': 1384914903.9989159, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000380039215088', 'lastCheck': 1384914910.735188, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:35:14,015::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912393::DEBUG::2013-11-20 02:35:19,191::guestIF::112::vm.Vm::(_handleMessage) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::Guest system shuts down. >Thread-1024767::ERROR::2013-11-20 02:35:24,031::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >libvirtEventLoop::DEBUG::2013-11-20 02:35:25,062::vm::498::vm.Vm::(onReboot) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::reboot event >Thread-15912493::INFO::2013-11-20 02:35:26,341::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912493::DEBUG::2013-11-20 02:35:26,342::task::495::TaskManager.Task::(_debug) Task f14bfc78-1ad6-464a-9da5-d92d14f7549d: moving from state init -> state preparing >Thread-15912493::DEBUG::2013-11-20 02:35:26,343::task::495::TaskManager.Task::(_debug) Task f14bfc78-1ad6-464a-9da5-d92d14f7549d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000282049179077', 'lastCheck': 1384914922.482832, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00250315666199', 'lastCheck': 1384914924.0334179, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00144815444946', 'lastCheck': 1384914924.034229, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000388860702515', 'lastCheck': 1384914920.7463801, 'valid': True, 'code': 0}} >Thread-15912493::DEBUG::2013-11-20 02:35:26,343::task::495::TaskManager.Task::(_debug) Task f14bfc78-1ad6-464a-9da5-d92d14f7549d: moving from state preparing -> state finished >Thread-15912493::DEBUG::2013-11-20 02:35:26,344::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912493::DEBUG::2013-11-20 02:35:26,345::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912493::DEBUG::2013-11-20 02:35:26,345::task::495::TaskManager.Task::(_debug) Task f14bfc78-1ad6-464a-9da5-d92d14f7549d: ref 0 aborting False >Thread-15912493::INFO::2013-11-20 02:35:26,346::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000282049179077', 'lastCheck': 1384914922.482832, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00250315666199', 'lastCheck': 1384914924.0334179, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00144815444946', 'lastCheck': 1384914924.034229, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000388860702515', 'lastCheck': 1384914920.7463801, 'valid': True, 'code': 0}} >Thread-15912496::DEBUG::2013-11-20 02:35:29,531::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912496::DEBUG::2013-11-20 02:35:29,532::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': 'Unknown', 'memUsage': '35', 'acpiEnable': 'true', 'pid': '19721', 'displayPort': u'5900', 'session': 'Unknown', 'netIfaces': [{u'inet6': [u'fe80::7d2d:e139:619b:7edc'], u'hw': u'00-1a-4a-a8-64-06', u'inet': [u'192.168.100.168'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': '3600', 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '22318374912', u'fs': u'NTFS'}], 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'displayType': 'qxl', 'guestName': u'KLEE-DC01', 'boot': 'c', 'elapsedTime': '4', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Zabbix Agent 2.0.2', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'RHEV-Spice-Agent64', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10'], 'guestOs': u'Unknown', 'nice': '', 'status': 'RebootInProgress', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'displayIp': '0', 'lastLogout': 1384914925.0776689, 'lastUser': u'None', 'clientIp': '192.168.50.51', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '5085719', 'apparentsize': '64424509440', 'writeLatency': '480185', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '240100', 'readRate': '3254049.10', 'truesize': '64424509440', 'writeRate': '1605522.44'}}, 'monitorResponse': '0', 'statsAge': '1.84', 'cpuUser': '99.93', 'cpuIdle': '0.07', 'lastLogin': 1384914808.992449, 'guestIPs': u'192.168.100.168', 'displaySecurePort': u'5901'}]} >Thread-15912498::DEBUG::2013-11-20 02:35:32,624::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912498::DEBUG::2013-11-20 02:35:32,625::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': 'Unknown', 'memUsage': '35', 'acpiEnable': 'true', 'pid': '19721', 'displayPort': u'5900', 'session': 'Unknown', 'netIfaces': [{u'inet6': [u'fe80::7d2d:e139:619b:7edc'], u'hw': u'00-1a-4a-a8-64-06', u'inet': [u'192.168.100.168'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': '3600', 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '22318374912', u'fs': u'NTFS'}], 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'displayType': 'qxl', 'guestName': u'KLEE-DC01', 'boot': 'c', 'elapsedTime': '7', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Zabbix Agent 2.0.2', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'RHEV-Spice-Agent64', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10'], 'guestOs': u'Unknown', 'nice': '', 'status': 'RebootInProgress', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'displayIp': '0', 'lastLogout': 1384914925.0776689, 'lastUser': u'None', 'clientIp': '192.168.50.51', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '5085719', 'apparentsize': '64424509440', 'writeLatency': '480185', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '240100', 'readRate': '3254049.10', 'truesize': '64424509440', 'writeRate': '1605522.44'}}, 'monitorResponse': '0', 'statsAge': '0.93', 'cpuUser': '99.93', 'cpuIdle': '0.07', 'lastLogin': 1384914808.992449, 'guestIPs': u'192.168.100.168', 'displaySecurePort': u'5901'}]} >Thread-1024767::ERROR::2013-11-20 02:35:34,050::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >libvirtEventLoop::DEBUG::2013-11-20 02:35:34,315::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::graphics event phase 2 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >libvirtEventLoop::DEBUG::2013-11-20 02:35:34,316::guestIF::154::vm.Vm::(desktopLock) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::desktopLock called >Thread-15912500::DEBUG::2013-11-20 02:35:35,716::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912500::DEBUG::2013-11-20 02:35:35,717::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': 'Unknown', 'memUsage': '35', 'acpiEnable': 'true', 'pid': '19721', 'displayPort': u'5900', 'session': 'Unknown', 'netIfaces': [{u'inet6': [u'fe80::7d2d:e139:619b:7edc'], u'hw': u'00-1a-4a-a8-64-06', u'inet': [u'192.168.100.168'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': '3600', 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '22318374912', u'fs': u'NTFS'}], 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'displayType': 'qxl', 'guestName': u'KLEE-DC01', 'boot': 'c', 'elapsedTime': '10', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Zabbix Agent 2.0.2', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'RHEV-Spice-Agent64', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10'], 'guestOs': u'Unknown', 'nice': '', 'status': 'RebootInProgress', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'displayIp': '0', 'lastLogout': 1384914925.0776689, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '5085719', 'apparentsize': '64424509440', 'writeLatency': '480185', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '240100', 'readRate': '3254049.10', 'truesize': '64424509440', 'writeRate': '1605522.44'}}, 'monitorResponse': '0', 'statsAge': '0.02', 'cpuUser': '99.93', 'cpuIdle': '0.07', 'lastLogin': 1384914808.992449, 'guestIPs': u'192.168.100.168', 'displaySecurePort': u'5901'}]} >Thread-15912502::DEBUG::2013-11-20 02:35:38,900::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912502::DEBUG::2013-11-20 02:35:38,902::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': 'Unknown', 'memUsage': '35', 'acpiEnable': 'true', 'pid': '19721', 'displayPort': u'5900', 'session': 'Unknown', 'netIfaces': [{u'inet6': [u'fe80::7d2d:e139:619b:7edc'], u'hw': u'00-1a-4a-a8-64-06', u'inet': [u'192.168.100.168'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': '3600', 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '22318374912', u'fs': u'NTFS'}], 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'displayType': 'qxl', 'guestName': u'KLEE-DC01', 'boot': 'c', 'elapsedTime': '13', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Zabbix Agent 2.0.2', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'RHEV-Spice-Agent64', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10'], 'guestOs': u'Unknown', 'nice': '', 'status': 'RebootInProgress', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'displayIp': '0', 'lastLogout': 1384914925.0776689, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '5085719', 'apparentsize': '64424509440', 'writeLatency': '480185', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '240100', 'readRate': '3254049.10', 'truesize': '64424509440', 'writeRate': '1605522.44'}}, 'monitorResponse': '0', 'statsAge': '1.20', 'cpuUser': '99.93', 'cpuIdle': '0.07', 'lastLogin': 1384914808.992449, 'guestIPs': u'192.168.100.168', 'displaySecurePort': u'5901'}]} >Thread-15912503::INFO::2013-11-20 02:35:42,060::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912503::DEBUG::2013-11-20 02:35:42,061::task::495::TaskManager.Task::(_debug) Task 0339dc05-04fc-466a-a39d-5575018f2fa5: moving from state init -> state preparing >Thread-15912503::DEBUG::2013-11-20 02:35:42,061::task::495::TaskManager.Task::(_debug) Task 0339dc05-04fc-466a-a39d-5575018f2fa5: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000271081924438', 'lastCheck': 1384914932.4941659, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00178790092468', 'lastCheck': 1384914934.052454, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00137805938721', 'lastCheck': 1384914934.0533841, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000355005264282', 'lastCheck': 1384914940.7691929, 'valid': True, 'code': 0}} >Thread-15912503::DEBUG::2013-11-20 02:35:42,062::task::495::TaskManager.Task::(_debug) Task 0339dc05-04fc-466a-a39d-5575018f2fa5: moving from state preparing -> state finished >Thread-15912503::DEBUG::2013-11-20 02:35:42,063::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912503::DEBUG::2013-11-20 02:35:42,063::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912503::DEBUG::2013-11-20 02:35:42,064::task::495::TaskManager.Task::(_debug) Task 0339dc05-04fc-466a-a39d-5575018f2fa5: ref 0 aborting False >Thread-15912503::INFO::2013-11-20 02:35:42,065::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000271081924438', 'lastCheck': 1384914932.4941659, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00178790092468', 'lastCheck': 1384914934.052454, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00137805938721', 'lastCheck': 1384914934.0533841, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000355005264282', 'lastCheck': 1384914940.7691929, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:35:44,066::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912506::DEBUG::2013-11-20 02:35:45,993::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912506::DEBUG::2013-11-20 02:35:45,995::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': 'Unknown', 'memUsage': '35', 'acpiEnable': 'true', 'pid': '19721', 'displayPort': u'5900', 'session': 'Unknown', 'netIfaces': [{u'inet6': [u'fe80::7d2d:e139:619b:7edc'], u'hw': u'00-1a-4a-a8-64-06', u'inet': [u'192.168.100.168'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': '3600', 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '22318374912', u'fs': u'NTFS'}], 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'displayType': 'qxl', 'guestName': u'KLEE-DC01', 'boot': 'c', 'elapsedTime': '20', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Zabbix Agent 2.0.2', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'RHEV-Spice-Agent64', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10'], 'guestOs': u'Unknown', 'nice': '', 'status': 'RebootInProgress', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'displayIp': '0', 'lastLogout': 1384914925.0776689, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '5085719', 'apparentsize': '64424509440', 'writeLatency': '480185', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '240100', 'readRate': '3254049.10', 'truesize': '64424509440', 'writeRate': '1605522.44'}}, 'monitorResponse': '0', 'statsAge': '0.29', 'cpuUser': '66.62', 'cpuIdle': '33.38', 'lastLogin': 1384914808.992449, 'guestIPs': u'192.168.100.168', 'displaySecurePort': u'5901'}]} >Thread-15912508::DEBUG::2013-11-20 02:35:49,302::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912508::DEBUG::2013-11-20 02:35:49,303::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayPort': u'5900', 'session': 'Unknown', 'netIfaces': [{u'inet6': [u'fe80::7d2d:e139:619b:7edc'], u'hw': u'00-1a-4a-a8-64-06', u'inet': [u'192.168.100.168'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': '3600', 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '22318374912', u'fs': u'NTFS'}], 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'displayType': 'qxl', 'guestName': u'KLEE-DC01', 'boot': 'c', 'elapsedTime': '24', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Zabbix Agent 2.0.2', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'RHEV-Spice-Agent64', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10'], 'guestOs': u'Unknown', 'nice': '', 'status': 'RebootInProgress', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'displayIp': '0', 'lastLogout': 1384914925.0776689, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '5085719', 'apparentsize': '64424509440', 'writeLatency': '480185', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '240100', 'readRate': '3254049.10', 'truesize': '64424509440', 'writeRate': '1605522.44'}}, 'monitorResponse': '0', 'statsAge': '1.60', 'cpuUser': '66.62', 'cpuIdle': '33.38', 'lastLogin': 1384914808.992449, 'guestIPs': u'192.168.100.168', 'displaySecurePort': u'5901'}]} >Thread-15912510::DEBUG::2013-11-20 02:35:52,790::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912510::DEBUG::2013-11-20 02:35:52,791::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayPort': u'5900', 'session': 'Unknown', 'netIfaces': [{u'inet6': [u'fe80::7d2d:e139:619b:7edc'], u'hw': u'00-1a-4a-a8-64-06', u'inet': [u'192.168.100.168'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': '3600', 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '22318374912', u'fs': u'NTFS'}], 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'displayType': 'qxl', 'guestName': u'KLEE-DC01', 'boot': 'c', 'elapsedTime': '27', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Zabbix Agent 2.0.2', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'RHEV-Spice-Agent64', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10'], 'guestOs': u'Unknown', 'nice': '', 'status': 'RebootInProgress', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'displayIp': '0', 'lastLogout': 1384914925.0776689, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '5085719', 'apparentsize': '64424509440', 'writeLatency': '480185', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '240100', 'readRate': '3254049.10', 'truesize': '64424509440', 'writeRate': '1605522.44'}}, 'monitorResponse': '0', 'statsAge': '1.08', 'cpuUser': '66.62', 'cpuIdle': '33.38', 'lastLogin': 1384914808.992449, 'guestIPs': u'192.168.100.168', 'displaySecurePort': u'5901'}]} >Thread-1024767::ERROR::2013-11-20 02:35:54,084::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912392::INFO::2013-11-20 02:35:55,708::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:35:55,709::task::495::TaskManager.Task::(_debug) Task 0f74a0d8-17c5-4bfc-ad1a-d803f5306ea3: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:35:55,711::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ba945ca1-0e8e-47fc-be64-20f909b0d432`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:35:55,711::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:35:55,712::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:35:55,713::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ba945ca1-0e8e-47fc-be64-20f909b0d432`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:35:55,714::task::495::TaskManager.Task::(_debug) Task 0f74a0d8-17c5-4bfc-ad1a-d803f5306ea3: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:35:55,715::task::495::TaskManager.Task::(_debug) Task 0f74a0d8-17c5-4bfc-ad1a-d803f5306ea3: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:35:55,716::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15912392::DEBUG::2013-11-20 02:35:55,717::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15912512::DEBUG::2013-11-20 02:35:56,128::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912512::DEBUG::2013-11-20 02:35:56,130::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayPort': u'5900', 'session': 'Unknown', 'netIfaces': [{u'inet6': [u'fe80::7d2d:e139:619b:7edc'], u'hw': u'00-1a-4a-a8-64-06', u'inet': [u'192.168.100.168'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': '3600', 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '22318374912', u'fs': u'NTFS'}], 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'displayType': 'qxl', 'guestName': u'KLEE-DC01', 'boot': 'c', 'elapsedTime': '31', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Zabbix Agent 2.0.2', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'RHEV-Spice-Agent64', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10'], 'guestOs': u'Unknown', 'nice': '', 'status': 'RebootInProgress', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'displayIp': '0', 'lastLogout': 1384914925.0776689, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '5085719', 'apparentsize': '64424509440', 'writeLatency': '480185', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '240100', 'readRate': '3254049.10', 'truesize': '64424509440', 'writeRate': '1605522.44'}}, 'monitorResponse': '0', 'statsAge': '0.42', 'cpuUser': '66.62', 'cpuIdle': '33.38', 'lastLogin': 1384914808.992449, 'guestIPs': u'192.168.100.168', 'displaySecurePort': u'5901'}]} >Thread-15912392::DEBUG::2013-11-20 02:35:56,637::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912392::DEBUG::2013-11-20 02:35:56,879::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15912392::DEBUG::2013-11-20 02:35:56,880::task::495::TaskManager.Task::(_debug) Task 0f74a0d8-17c5-4bfc-ad1a-d803f5306ea3: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:35:56,880::task::495::TaskManager.Task::(_debug) Task 0f74a0d8-17c5-4bfc-ad1a-d803f5306ea3: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:35:56,881::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:35:56,881::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:35:56,882::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:35:56,882::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:35:56,882::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:35:56,883::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:35:56,883::task::495::TaskManager.Task::(_debug) Task 0f74a0d8-17c5-4bfc-ad1a-d803f5306ea3: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:35:56,883::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912513::INFO::2013-11-20 02:36:00,981::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912513::DEBUG::2013-11-20 02:36:00,982::task::495::TaskManager.Task::(_debug) Task e78f88dd-4492-4e07-b729-9f48d5c7bc22: moving from state init -> state preparing >Thread-15912513::DEBUG::2013-11-20 02:36:00,983::task::495::TaskManager.Task::(_debug) Task e78f88dd-4492-4e07-b729-9f48d5c7bc22: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000350952148438', 'lastCheck': 1384914952.517581, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00125908851624', 'lastCheck': 1384914954.0859089, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00121402740479', 'lastCheck': 1384914954.0868239, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00431394577026', 'lastCheck': 1384914960.939889, 'valid': True, 'code': 0}} >Thread-15912513::DEBUG::2013-11-20 02:36:00,983::task::495::TaskManager.Task::(_debug) Task e78f88dd-4492-4e07-b729-9f48d5c7bc22: moving from state preparing -> state finished >Thread-15912513::DEBUG::2013-11-20 02:36:00,984::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912513::DEBUG::2013-11-20 02:36:00,984::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912513::DEBUG::2013-11-20 02:36:00,985::task::495::TaskManager.Task::(_debug) Task e78f88dd-4492-4e07-b729-9f48d5c7bc22: ref 0 aborting False >Thread-15912513::INFO::2013-11-20 02:36:00,986::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000350952148438', 'lastCheck': 1384914952.517581, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00125908851624', 'lastCheck': 1384914954.0859089, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00121402740479', 'lastCheck': 1384914954.0868239, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00431394577026', 'lastCheck': 1384914960.939889, 'valid': True, 'code': 0}} >Thread-42838::DEBUG::2013-11-20 02:36:02,528::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 02:36:03,158::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 02:36:04,103::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912516::DEBUG::2013-11-20 02:36:04,722::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912516::DEBUG::2013-11-20 02:36:04,723::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayPort': u'5900', 'session': 'Unknown', 'netIfaces': [{u'inet6': [u'fe80::7d2d:e139:619b:7edc'], u'hw': u'00-1a-4a-a8-64-06', u'inet': [u'192.168.100.168'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': '3600', 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '22318374912', u'fs': u'NTFS'}], 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'displayType': 'qxl', 'guestName': u'KLEE-DC01', 'boot': 'c', 'elapsedTime': '39', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Zabbix Agent 2.0.2', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'RHEV-Spice-Agent64', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10'], 'guestOs': u'Unknown', 'nice': '', 'status': 'RebootInProgress', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'displayIp': '0', 'lastLogout': 1384914925.0776689, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '2358866', 'apparentsize': '64424509440', 'writeLatency': '1259847', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '238791', 'readRate': '9215864.07', 'truesize': '64424509440', 'writeRate': '1143350.56'}}, 'monitorResponse': '0', 'statsAge': '1.82', 'cpuUser': '67.96', 'cpuIdle': '32.04', 'lastLogin': 1384914808.992449, 'guestIPs': u'192.168.100.168', 'displaySecurePort': u'5901'}]} >Thread-15912518::DEBUG::2013-11-20 02:36:08,606::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912518::DEBUG::2013-11-20 02:36:08,607::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayPort': u'5900', 'session': 'Unknown', 'netIfaces': [{u'inet6': [u'fe80::7d2d:e139:619b:7edc'], u'hw': u'00-1a-4a-a8-64-06', u'inet': [u'192.168.100.168'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': '3600', 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '22318374912', u'fs': u'NTFS'}], 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'displayType': 'qxl', 'guestName': u'KLEE-DC01', 'boot': 'c', 'elapsedTime': '43', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Zabbix Agent 2.0.2', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'RHEV-Spice-Agent64', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10'], 'guestOs': u'Unknown', 'nice': '', 'status': 'RebootInProgress', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'displayIp': '0', 'lastLogout': 1384914925.0776689, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '2358866', 'apparentsize': '64424509440', 'writeLatency': '1259847', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '238791', 'readRate': '9215864.07', 'truesize': '64424509440', 'writeRate': '1143350.56'}}, 'monitorResponse': '0', 'statsAge': '1.70', 'cpuUser': '67.96', 'cpuIdle': '32.04', 'lastLogin': 1384914808.992449, 'guestIPs': u'192.168.100.168', 'displaySecurePort': u'5901'}]} >Thread-15912520::DEBUG::2013-11-20 02:36:12,273::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912520::DEBUG::2013-11-20 02:36:12,273::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayPort': u'5900', 'session': 'Unknown', 'netIfaces': [{u'inet6': [u'fe80::7d2d:e139:619b:7edc'], u'hw': u'00-1a-4a-a8-64-06', u'inet': [u'192.168.100.168'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': '3600', 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '22318374912', u'fs': u'NTFS'}], 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'displayType': 'qxl', 'guestName': u'KLEE-DC01', 'boot': 'c', 'elapsedTime': '47', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Zabbix Agent 2.0.2', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'RHEV-Spice-Agent64', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10'], 'guestOs': u'Unknown', 'nice': '', 'status': 'RebootInProgress', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'displayIp': '0', 'lastLogout': 1384914925.0776689, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '2358866', 'apparentsize': '64424509440', 'writeLatency': '1259847', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '238791', 'readRate': '9215864.07', 'truesize': '64424509440', 'writeRate': '1143350.56'}}, 'monitorResponse': '0', 'statsAge': '0.36', 'cpuUser': '99.83', 'cpuIdle': '0.17', 'lastLogin': 1384914808.992449, 'guestIPs': u'192.168.100.168', 'displaySecurePort': u'5901'}]} >Thread-1024767::ERROR::2013-11-20 02:36:14,262::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912522::DEBUG::2013-11-20 02:36:15,595::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912522::DEBUG::2013-11-20 02:36:15,597::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayPort': u'5900', 'session': 'Unknown', 'netIfaces': [{u'inet6': [u'fe80::7d2d:e139:619b:7edc'], u'hw': u'00-1a-4a-a8-64-06', u'inet': [u'192.168.100.168'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': '3600', 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '22318374912', u'fs': u'NTFS'}], 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'displayType': 'qxl', 'guestName': u'KLEE-DC01', 'boot': 'c', 'elapsedTime': '50', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Zabbix Agent 2.0.2', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'RHEV-Spice-Agent64', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10'], 'guestOs': u'Unknown', 'nice': '', 'status': 'RebootInProgress', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'displayIp': '0', 'lastLogout': 1384914925.0776689, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '2358866', 'apparentsize': '64424509440', 'writeLatency': '1259847', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '238791', 'readRate': '9215864.07', 'truesize': '64424509440', 'writeRate': '1143350.56'}}, 'monitorResponse': '0', 'statsAge': '0.68', 'cpuUser': '99.83', 'cpuIdle': '0.17', 'lastLogin': 1384914808.992449, 'guestIPs': u'192.168.100.168', 'displaySecurePort': u'5901'}]} >Thread-15912523::INFO::2013-11-20 02:36:19,096::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912523::DEBUG::2013-11-20 02:36:19,097::task::495::TaskManager.Task::(_debug) Task 3c31ce24-b2b8-4ea2-9c81-9ad70d17a724: moving from state init -> state preparing >Thread-15912523::DEBUG::2013-11-20 02:36:19,098::task::495::TaskManager.Task::(_debug) Task 3c31ce24-b2b8-4ea2-9c81-9ad70d17a724: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000200033187866', 'lastCheck': 1384914973.1737111, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00108599662781', 'lastCheck': 1384914974.243679, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000864028930664', 'lastCheck': 1384914974.2636421, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00048303604126', 'lastCheck': 1384914970.9516399, 'valid': True, 'code': 0}} >Thread-15912523::DEBUG::2013-11-20 02:36:19,098::task::495::TaskManager.Task::(_debug) Task 3c31ce24-b2b8-4ea2-9c81-9ad70d17a724: moving from state preparing -> state finished >Thread-15912523::DEBUG::2013-11-20 02:36:19,099::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912523::DEBUG::2013-11-20 02:36:19,100::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912523::DEBUG::2013-11-20 02:36:19,100::task::495::TaskManager.Task::(_debug) Task 3c31ce24-b2b8-4ea2-9c81-9ad70d17a724: ref 0 aborting False >Thread-15912523::INFO::2013-11-20 02:36:19,101::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000200033187866', 'lastCheck': 1384914973.1737111, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00108599662781', 'lastCheck': 1384914974.243679, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000864028930664', 'lastCheck': 1384914974.2636421, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00048303604126', 'lastCheck': 1384914970.9516399, 'valid': True, 'code': 0}} >Thread-15912526::DEBUG::2013-11-20 02:36:22,747::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912526::DEBUG::2013-11-20 02:36:22,749::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayPort': u'5900', 'session': 'Unknown', 'netIfaces': [{u'inet6': [u'fe80::7d2d:e139:619b:7edc'], u'hw': u'00-1a-4a-a8-64-06', u'inet': [u'192.168.100.168'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': '3600', 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '22318374912', u'fs': u'NTFS'}], 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'displayType': 'qxl', 'guestName': u'KLEE-DC01', 'boot': 'c', 'elapsedTime': '57', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Zabbix Agent 2.0.2', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'RHEV-Spice-Agent64', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10'], 'guestOs': u'Unknown', 'nice': '', 'status': 'RebootInProgress', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'displayIp': '0', 'lastLogout': 1384914925.0776689, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '2358866', 'apparentsize': '64424509440', 'writeLatency': '1259847', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '238791', 'readRate': '9215864.07', 'truesize': '64424509440', 'writeRate': '1143350.56'}}, 'monitorResponse': '0', 'statsAge': '0.83', 'cpuUser': '99.83', 'cpuIdle': '0.17', 'lastLogin': 1384914808.992449, 'guestIPs': u'192.168.100.168', 'displaySecurePort': u'5901'}]} >Thread-1024767::ERROR::2013-11-20 02:36:24,276::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912528::DEBUG::2013-11-20 02:36:25,932::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912528::DEBUG::2013-11-20 02:36:25,933::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayPort': u'5900', 'session': 'Unknown', 'netIfaces': [{u'inet6': [u'fe80::7d2d:e139:619b:7edc'], u'hw': u'00-1a-4a-a8-64-06', u'inet': [u'192.168.100.168'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': '3600', 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '22318374912', u'fs': u'NTFS'}], 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'displayType': 'qxl', 'guestName': u'KLEE-DC01', 'boot': 'c', 'elapsedTime': '60', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Zabbix Agent 2.0.2', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'RHEV-Spice-Agent64', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10'], 'guestOs': u'Unknown', 'nice': '', 'status': 'RebootInProgress', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'displayIp': '0', 'lastLogout': 1384914925.0776689, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '2358866', 'apparentsize': '64424509440', 'writeLatency': '1259847', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '238791', 'readRate': '9215864.07', 'truesize': '64424509440', 'writeRate': '1143350.56'}}, 'monitorResponse': '0', 'statsAge': '1.01', 'cpuUser': '99.83', 'cpuIdle': '0.17', 'lastLogin': 1384914808.992449, 'guestIPs': u'192.168.100.168', 'displaySecurePort': u'5901'}]} >Thread-15912530::DEBUG::2013-11-20 02:36:29,787::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912530::DEBUG::2013-11-20 02:36:29,788::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayPort': u'5900', 'session': 'Unknown', 'netIfaces': [{u'inet6': [u'fe80::7d2d:e139:619b:7edc'], u'hw': u'00-1a-4a-a8-64-06', u'inet': [u'192.168.100.168'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': '3600', 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '22318374912', u'fs': u'NTFS'}], 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'displayType': 'qxl', 'guestName': u'KLEE-DC01', 'boot': 'c', 'elapsedTime': '64', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Zabbix Agent 2.0.2', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'RHEV-Spice-Agent64', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10'], 'guestOs': u'Unknown', 'nice': '', 'status': 'RebootInProgress', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'displayIp': '0', 'lastLogout': 1384914925.0776689, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '2358866', 'apparentsize': '64424509440', 'writeLatency': '1259847', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '238791', 'readRate': '9215864.07', 'truesize': '64424509440', 'writeRate': '1143350.56'}}, 'monitorResponse': '0', 'statsAge': '0.86', 'cpuUser': '153.23', 'cpuIdle': '0.00', 'lastLogin': 1384914808.992449, 'guestIPs': u'192.168.100.168', 'displaySecurePort': u'5901'}]} >Thread-15912532::DEBUG::2013-11-20 02:36:33,897::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912532::DEBUG::2013-11-20 02:36:33,898::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayPort': u'5900', 'session': 'Unknown', 'netIfaces': [{u'inet6': [u'fe80::7d2d:e139:619b:7edc'], u'hw': u'00-1a-4a-a8-64-06', u'inet': [u'192.168.100.168'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': '3600', 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '22318374912', u'fs': u'NTFS'}], 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'displayType': 'qxl', 'guestName': u'KLEE-DC01', 'boot': 'c', 'elapsedTime': '68', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Zabbix Agent 2.0.2', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'RHEV-Spice-Agent64', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10'], 'guestOs': u'Unknown', 'nice': '', 'status': 'RebootInProgress', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'displayIp': '0', 'lastLogout': 1384914925.0776689, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '2358866', 'apparentsize': '64424509440', 'writeLatency': '1259847', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '238791', 'readRate': '9215864.07', 'truesize': '64424509440', 'writeRate': '1143350.56'}}, 'monitorResponse': '0', 'statsAge': '0.96', 'cpuUser': '153.23', 'cpuIdle': '0.00', 'lastLogin': 1384914808.992449, 'guestIPs': u'192.168.100.168', 'displaySecurePort': u'5901'}]} >Thread-1024767::ERROR::2013-11-20 02:36:34,291::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912533::INFO::2013-11-20 02:36:36,978::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912533::DEBUG::2013-11-20 02:36:36,978::task::495::TaskManager.Task::(_debug) Task 0cc5872d-984d-46e9-8b4f-74480c882146: moving from state init -> state preparing >Thread-15912533::DEBUG::2013-11-20 02:36:36,979::task::495::TaskManager.Task::(_debug) Task 0cc5872d-984d-46e9-8b4f-74480c882146: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000270128250122', 'lastCheck': 1384914993.1972289, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00135087966919', 'lastCheck': 1384914994.2721529, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00153589248657', 'lastCheck': 1384914994.294167, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000453948974609', 'lastCheck': 1384914990.9750359, 'valid': True, 'code': 0}} >Thread-15912533::DEBUG::2013-11-20 02:36:36,980::task::495::TaskManager.Task::(_debug) Task 0cc5872d-984d-46e9-8b4f-74480c882146: moving from state preparing -> state finished >Thread-15912533::DEBUG::2013-11-20 02:36:36,980::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912533::DEBUG::2013-11-20 02:36:36,981::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912533::DEBUG::2013-11-20 02:36:36,981::task::495::TaskManager.Task::(_debug) Task 0cc5872d-984d-46e9-8b4f-74480c882146: ref 0 aborting False >Thread-15912533::INFO::2013-11-20 02:36:36,982::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000270128250122', 'lastCheck': 1384914993.1972289, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00135087966919', 'lastCheck': 1384914994.2721529, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00153589248657', 'lastCheck': 1384914994.294167, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000453948974609', 'lastCheck': 1384914990.9750359, 'valid': True, 'code': 0}} >Thread-15912536::DEBUG::2013-11-20 02:36:40,548::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912536::DEBUG::2013-11-20 02:36:40,549::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayPort': u'5900', 'session': 'Unknown', 'netIfaces': [{u'inet6': [u'fe80::7d2d:e139:619b:7edc'], u'hw': u'00-1a-4a-a8-64-06', u'inet': [u'192.168.100.168'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': '3600', 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '22318374912', u'fs': u'NTFS'}], 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'displayType': 'qxl', 'guestName': u'KLEE-DC01', 'boot': 'c', 'elapsedTime': '75', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Zabbix Agent 2.0.2', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'RHEV-Spice-Agent64', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10'], 'guestOs': u'Unknown', 'nice': '', 'status': 'RebootInProgress', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'displayIp': '0', 'lastLogout': 1384914925.0776689, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '2358866', 'apparentsize': '64424509440', 'writeLatency': '1259847', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '238791', 'readRate': '9215864.07', 'truesize': '64424509440', 'writeRate': '1143350.56'}}, 'monitorResponse': '0', 'statsAge': '1.61', 'cpuUser': '153.23', 'cpuIdle': '0.00', 'lastLogin': 1384914808.992449, 'guestIPs': u'192.168.100.168', 'displaySecurePort': u'5901'}]} >Thread-1024767::ERROR::2013-11-20 02:36:44,307::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912538::DEBUG::2013-11-20 02:36:45,248::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912538::DEBUG::2013-11-20 02:36:45,249::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayPort': u'5900', 'session': 'Unknown', 'netIfaces': [{u'inet6': [u'fe80::7d2d:e139:619b:7edc'], u'hw': u'00-1a-4a-a8-64-06', u'inet': [u'192.168.100.168'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': '3600', 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '22318374912', u'fs': u'NTFS'}], 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'displayType': 'qxl', 'guestName': u'KLEE-DC01', 'boot': 'c', 'elapsedTime': '80', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Zabbix Agent 2.0.2', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'RHEV-Spice-Agent64', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10'], 'guestOs': u'Unknown', 'nice': '', 'status': 'RebootInProgress', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'displayIp': '0', 'lastLogout': 1384914925.0776689, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '2358866', 'apparentsize': '64424509440', 'writeLatency': '1259847', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '238791', 'readRate': '9215864.07', 'truesize': '64424509440', 'writeRate': '1143350.56'}}, 'monitorResponse': '0', 'statsAge': '0.31', 'cpuUser': '186.46', 'cpuIdle': '0.00', 'lastLogin': 1384914808.992449, 'guestIPs': u'192.168.100.168', 'displaySecurePort': u'5901'}]} >Thread-15912540::DEBUG::2013-11-20 02:36:50,644::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912540::DEBUG::2013-11-20 02:36:50,645::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayPort': u'5900', 'session': 'Unknown', 'netIfaces': [{u'inet6': [u'fe80::7d2d:e139:619b:7edc'], u'hw': u'00-1a-4a-a8-64-06', u'inet': [u'192.168.100.168'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': '3600', 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '22318374912', u'fs': u'NTFS'}], 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'displayType': 'qxl', 'guestName': u'KLEE-DC01', 'boot': 'c', 'elapsedTime': '85', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Zabbix Agent 2.0.2', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'RHEV-Spice-Agent64', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10'], 'guestOs': u'Unknown', 'nice': '', 'status': 'RebootInProgress', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'displayIp': '0', 'lastLogout': 1384914925.0776689, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '2358866', 'apparentsize': '64424509440', 'writeLatency': '1259847', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '238791', 'readRate': '9215864.07', 'truesize': '64424509440', 'writeRate': '1143350.56'}}, 'monitorResponse': '0', 'statsAge': '1.70', 'cpuUser': '186.46', 'cpuIdle': '0.00', 'lastLogin': 1384914808.992449, 'guestIPs': u'192.168.100.168', 'displaySecurePort': u'5901'}]} >Thread-1024767::ERROR::2013-11-20 02:36:54,322::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912542::DEBUG::2013-11-20 02:36:54,394::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912542::DEBUG::2013-11-20 02:36:54,395::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '19721', 'displayPort': u'5900', 'session': 'Unknown', 'netIfaces': [{u'inet6': [u'fe80::7d2d:e139:619b:7edc'], u'hw': u'00-1a-4a-a8-64-06', u'inet': [u'192.168.100.168'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': '3600', 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '22318374912', u'fs': u'NTFS'}], 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'displayType': 'qxl', 'guestName': u'KLEE-DC01', 'boot': 'c', 'elapsedTime': '89', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Zabbix Agent 2.0.2', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'RHEV-Spice-Agent64', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10'], 'guestOs': u'Unknown', 'nice': '', 'status': 'RebootInProgress', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'displayIp': '0', 'lastLogout': 1384914925.0776689, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '2358866', 'apparentsize': '64424509440', 'writeLatency': '1259847', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '238791', 'readRate': '9215864.07', 'truesize': '64424509440', 'writeRate': '1143350.56'}}, 'monitorResponse': '0', 'statsAge': '1.45', 'cpuUser': '186.46', 'cpuIdle': '0.00', 'lastLogin': 1384914808.992449, 'guestIPs': u'192.168.100.168', 'displaySecurePort': u'5901'}]} >Thread-15912393::DEBUG::2013-11-20 02:36:54,482::guestIF::97::vm.Vm::(_handleMessage) vmId=`98ec88b5-2c08-4ff2-b607-c3f4e281f6af`::u'None' >Thread-15912392::INFO::2013-11-20 02:36:56,946::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:36:56,946::task::495::TaskManager.Task::(_debug) Task cd74541d-3305-4916-898a-cd1262583a30: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:36:56,962::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d100d61e-d980-44dc-95c5-abfa941657fd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:36:56,962::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:36:56,963::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:36:56,963::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d100d61e-d980-44dc-95c5-abfa941657fd`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:36:56,964::task::495::TaskManager.Task::(_debug) Task cd74541d-3305-4916-898a-cd1262583a30: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:36:56,964::task::495::TaskManager.Task::(_debug) Task cd74541d-3305-4916-898a-cd1262583a30: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:36:56,965::task::495::TaskManager.Task::(_debug) Task cd74541d-3305-4916-898a-cd1262583a30: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:36:56,965::task::495::TaskManager.Task::(_debug) Task cd74541d-3305-4916-898a-cd1262583a30: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:36:56,965::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:36:56,966::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:36:56,966::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:36:56,967::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:36:56,967::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:36:56,968::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:36:56,968::task::495::TaskManager.Task::(_debug) Task cd74541d-3305-4916-898a-cd1262583a30: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:36:56,969::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912543::INFO::2013-11-20 02:36:57,902::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912543::DEBUG::2013-11-20 02:36:57,903::task::495::TaskManager.Task::(_debug) Task c084be9e-03a3-4e94-8315-15e9ec9b4faf: moving from state init -> state preparing >Thread-15912543::DEBUG::2013-11-20 02:36:57,904::task::495::TaskManager.Task::(_debug) Task c084be9e-03a3-4e94-8315-15e9ec9b4faf: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000298023223877', 'lastCheck': 1384915013.221884, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00126886367798', 'lastCheck': 1384915014.303246, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00125193595886', 'lastCheck': 1384915014.324827, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000454902648926', 'lastCheck': 1384915010.99804, 'valid': True, 'code': 0}} >Thread-15912543::DEBUG::2013-11-20 02:36:57,904::task::495::TaskManager.Task::(_debug) Task c084be9e-03a3-4e94-8315-15e9ec9b4faf: moving from state preparing -> state finished >Thread-15912543::DEBUG::2013-11-20 02:36:57,905::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912543::DEBUG::2013-11-20 02:36:57,905::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912543::DEBUG::2013-11-20 02:36:57,906::task::495::TaskManager.Task::(_debug) Task c084be9e-03a3-4e94-8315-15e9ec9b4faf: ref 0 aborting False >Thread-15912543::INFO::2013-11-20 02:36:57,907::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000298023223877', 'lastCheck': 1384915013.221884, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00126886367798', 'lastCheck': 1384915014.303246, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00125193595886', 'lastCheck': 1384915014.324827, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000454902648926', 'lastCheck': 1384915010.99804, 'valid': True, 'code': 0}} >Thread-15912546::DEBUG::2013-11-20 02:37:03,157::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('98ec88b5-2c08-4ff2-b607-c3f4e281f6af',) {} >Thread-15912546::DEBUG::2013-11-20 02:37:03,158::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'username': u'None', 'memUsage': '38', 'acpiEnable': 'true', 'pid': '19721', 'displayPort': u'5900', 'session': 'Unknown', 'netIfaces': [{u'inet6': [u'fe80::7d2d:e139:619b:7edc'], u'hw': u'00-1a-4a-a8-64-06', u'inet': [u'192.168.100.168'], u'name': u'Red Hat VirtIO Ethernet Adapter'}], 'timeOffset': '3600', 'pauseCode': 'NOERR', 'disksUsage': [{u'path': u'c:\\', u'total': '64055406592', u'used': '22318374912', u'fs': u'NTFS'}], 'network': {u'vnet0': {'macAddr': u'00:1a:4a:a8:64:06', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet0'}}, 'displayType': 'qxl', 'guestName': u'KLEE-DC01', 'boot': 'c', 'elapsedTime': '98', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [u'Zabbix Agent 2.0.2', u'RHEV-Tools 3.0.39', u'RHEV-Network64 3.0.7', u'RHEV-Block64 3.0.9', u'RHEV-Spice-Agent64', u'Acronis Backup & Recovery 11.5 Agent Core', u'Acronis Backup & Recovery 11.5\xa0Agent for Windows', u'RHEV-Serial64 3.0.6', u'RHEV-Agent64 3.0.10'], 'guestOs': u'Unknown', 'nice': '', 'status': 'Running', 'vmId': '98ec88b5-2c08-4ff2-b607-c3f4e281f6af', 'displayIp': '0', 'lastLogout': 1384914925.0776689, 'lastUser': u'None', 'clientIp': '', 'kvmEnable': 'true', 'disks': {u'vda': {'readLatency': '2793268', 'apparentsize': '64424509440', 'writeLatency': '1563827', 'imageID': 'd8be6e76-2f7f-4b56-9ada-c3fd543a9794', 'flushLatency': '226682', 'readRate': '7366975.14', 'truesize': '64424509440', 'writeRate': '1650446.35'}}, 'monitorResponse': '0', 'statsAge': '0.17', 'cpuUser': '153.00', 'cpuIdle': '0.00', 'lastLogin': 1384915014.482013, 'guestIPs': u'192.168.100.168', 'displaySecurePort': u'5901'}]} >Thread-1024767::ERROR::2013-11-20 02:37:04,337::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-1024767::ERROR::2013-11-20 02:37:14,352::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912550::INFO::2013-11-20 02:37:16,103::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912550::DEBUG::2013-11-20 02:37:16,104::task::495::TaskManager.Task::(_debug) Task d7611c87-e4e9-461b-89a6-1c16eb4e85d7: moving from state init -> state preparing >Thread-15912550::DEBUG::2013-11-20 02:37:16,105::task::495::TaskManager.Task::(_debug) Task d7611c87-e4e9-461b-89a6-1c16eb4e85d7: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000288009643555', 'lastCheck': 1384915033.2466469, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00121998786926', 'lastCheck': 1384915034.3350379, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00104999542236', 'lastCheck': 1384915034.3543899, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000444889068604', 'lastCheck': 1384915031.019691, 'valid': True, 'code': 0}} >Thread-15912550::DEBUG::2013-11-20 02:37:16,106::task::495::TaskManager.Task::(_debug) Task d7611c87-e4e9-461b-89a6-1c16eb4e85d7: moving from state preparing -> state finished >Thread-15912550::DEBUG::2013-11-20 02:37:16,106::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912550::DEBUG::2013-11-20 02:37:16,107::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912550::DEBUG::2013-11-20 02:37:16,107::task::495::TaskManager.Task::(_debug) Task d7611c87-e4e9-461b-89a6-1c16eb4e85d7: ref 0 aborting False >Thread-15912550::INFO::2013-11-20 02:37:16,108::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000288009643555', 'lastCheck': 1384915033.2466469, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00121998786926', 'lastCheck': 1384915034.3350379, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00104999542236', 'lastCheck': 1384915034.3543899, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000444889068604', 'lastCheck': 1384915031.019691, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:37:24,366::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912556::INFO::2013-11-20 02:37:31,761::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912556::DEBUG::2013-11-20 02:37:31,762::task::495::TaskManager.Task::(_debug) Task 128ea5c2-1618-4707-8728-324dd246f50b: moving from state init -> state preparing >Thread-15912556::DEBUG::2013-11-20 02:37:31,763::task::495::TaskManager.Task::(_debug) Task 128ea5c2-1618-4707-8728-324dd246f50b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000309944152832', 'lastCheck': 1384915043.2583799, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00108313560486', 'lastCheck': 1384915044.3484759, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00147294998169', 'lastCheck': 1384915044.369036, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00042986869812', 'lastCheck': 1384915051.040086, 'valid': True, 'code': 0}} >Thread-15912556::DEBUG::2013-11-20 02:37:31,763::task::495::TaskManager.Task::(_debug) Task 128ea5c2-1618-4707-8728-324dd246f50b: moving from state preparing -> state finished >Thread-15912556::DEBUG::2013-11-20 02:37:31,764::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912556::DEBUG::2013-11-20 02:37:31,764::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912556::DEBUG::2013-11-20 02:37:31,765::task::495::TaskManager.Task::(_debug) Task 128ea5c2-1618-4707-8728-324dd246f50b: ref 0 aborting False >Thread-15912556::INFO::2013-11-20 02:37:31,766::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000309944152832', 'lastCheck': 1384915043.2583799, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00108313560486', 'lastCheck': 1384915044.3484759, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00147294998169', 'lastCheck': 1384915044.369036, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00042986869812', 'lastCheck': 1384915051.040086, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:37:34,382::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-1024767::ERROR::2013-11-20 02:37:44,397::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912562::INFO::2013-11-20 02:37:48,882::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912562::DEBUG::2013-11-20 02:37:48,883::task::495::TaskManager.Task::(_debug) Task 13de12d3-742a-4933-b86d-06a9b19130e9: moving from state init -> state preparing >Thread-15912562::DEBUG::2013-11-20 02:37:48,884::task::495::TaskManager.Task::(_debug) Task 13de12d3-742a-4933-b86d-06a9b19130e9: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000377178192139', 'lastCheck': 1384915063.283426, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000998020172119', 'lastCheck': 1384915064.3790269, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000948190689087', 'lastCheck': 1384915064.399308, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000685930252075', 'lastCheck': 1384915061.0519381, 'valid': True, 'code': 0}} >Thread-15912562::DEBUG::2013-11-20 02:37:48,885::task::495::TaskManager.Task::(_debug) Task 13de12d3-742a-4933-b86d-06a9b19130e9: moving from state preparing -> state finished >Thread-15912562::DEBUG::2013-11-20 02:37:48,885::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912562::DEBUG::2013-11-20 02:37:48,886::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912562::DEBUG::2013-11-20 02:37:48,886::task::495::TaskManager.Task::(_debug) Task 13de12d3-742a-4933-b86d-06a9b19130e9: ref 0 aborting False >Thread-15912562::INFO::2013-11-20 02:37:48,887::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000377178192139', 'lastCheck': 1384915063.283426, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000998020172119', 'lastCheck': 1384915064.3790269, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000948190689087', 'lastCheck': 1384915064.399308, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000685930252075', 'lastCheck': 1384915061.0519381, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:37:54,412::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912392::INFO::2013-11-20 02:37:57,029::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:37:57,030::task::495::TaskManager.Task::(_debug) Task 53fb9720-289a-4733-b246-4cda3c27a894: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:37:57,031::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`7863bc82-0a9a-4ab9-805b-76170e39a471`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:37:57,032::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:37:57,033::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:37:57,033::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`7863bc82-0a9a-4ab9-805b-76170e39a471`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:37:57,034::task::495::TaskManager.Task::(_debug) Task 53fb9720-289a-4733-b246-4cda3c27a894: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:37:57,035::task::495::TaskManager.Task::(_debug) Task 53fb9720-289a-4733-b246-4cda3c27a894: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:37:57,035::task::495::TaskManager.Task::(_debug) Task 53fb9720-289a-4733-b246-4cda3c27a894: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:37:57,036::task::495::TaskManager.Task::(_debug) Task 53fb9720-289a-4733-b246-4cda3c27a894: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:37:57,036::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:37:57,037::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:37:57,037::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:37:57,038::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:37:57,038::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:37:57,039::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:37:57,040::task::495::TaskManager.Task::(_debug) Task 53fb9720-289a-4733-b246-4cda3c27a894: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:37:57,040::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-1024767::ERROR::2013-11-20 02:38:04,427::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912568::INFO::2013-11-20 02:38:04,497::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912568::DEBUG::2013-11-20 02:38:04,497::task::495::TaskManager.Task::(_debug) Task 5d869cba-1d01-49cb-b2a5-2ab56285eef6: moving from state init -> state preparing >Thread-15912568::DEBUG::2013-11-20 02:38:04,498::task::495::TaskManager.Task::(_debug) Task 5d869cba-1d01-49cb-b2a5-2ab56285eef6: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000263929367065', 'lastCheck': 1384915083.3068709, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00174117088318', 'lastCheck': 1384915084.409812, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00137686729431', 'lastCheck': 1384915084.4300411, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00045919418335', 'lastCheck': 1384915081.0736721, 'valid': True, 'code': 0}} >Thread-15912568::DEBUG::2013-11-20 02:38:04,499::task::495::TaskManager.Task::(_debug) Task 5d869cba-1d01-49cb-b2a5-2ab56285eef6: moving from state preparing -> state finished >Thread-15912568::DEBUG::2013-11-20 02:38:04,499::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912568::DEBUG::2013-11-20 02:38:04,500::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912568::DEBUG::2013-11-20 02:38:04,500::task::495::TaskManager.Task::(_debug) Task 5d869cba-1d01-49cb-b2a5-2ab56285eef6: ref 0 aborting False >Thread-15912568::INFO::2013-11-20 02:38:04,501::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000263929367065', 'lastCheck': 1384915083.3068709, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00174117088318', 'lastCheck': 1384915084.409812, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00137686729431', 'lastCheck': 1384915084.4300411, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00045919418335', 'lastCheck': 1384915081.0736721, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:38:14,442::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912574::INFO::2013-11-20 02:38:21,730::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912574::DEBUG::2013-11-20 02:38:21,730::task::495::TaskManager.Task::(_debug) Task 47edaa8b-150c-4078-a071-bc9ac46cbc95: moving from state init -> state preparing >Thread-15912574::DEBUG::2013-11-20 02:38:21,731::task::495::TaskManager.Task::(_debug) Task 47edaa8b-150c-4078-a071-bc9ac46cbc95: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000293970108032', 'lastCheck': 1384915093.3193631, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00116300582886', 'lastCheck': 1384915094.4245601, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000926971435547', 'lastCheck': 1384915094.444376, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000432014465332', 'lastCheck': 1384915101.0949349, 'valid': True, 'code': 0}} >Thread-15912574::DEBUG::2013-11-20 02:38:21,732::task::495::TaskManager.Task::(_debug) Task 47edaa8b-150c-4078-a071-bc9ac46cbc95: moving from state preparing -> state finished >Thread-15912574::DEBUG::2013-11-20 02:38:21,732::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912574::DEBUG::2013-11-20 02:38:21,733::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912574::DEBUG::2013-11-20 02:38:21,733::task::495::TaskManager.Task::(_debug) Task 47edaa8b-150c-4078-a071-bc9ac46cbc95: ref 0 aborting False >Thread-15912574::INFO::2013-11-20 02:38:21,734::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000293970108032', 'lastCheck': 1384915093.3193631, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00116300582886', 'lastCheck': 1384915094.4245601, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000926971435547', 'lastCheck': 1384915094.444376, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000432014465332', 'lastCheck': 1384915101.0949349, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:38:24,456::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-42840::DEBUG::2013-11-20 02:38:31,105::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 02:38:31,746::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-1024767::ERROR::2013-11-20 02:38:34,472::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912580::INFO::2013-11-20 02:38:38,089::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912580::DEBUG::2013-11-20 02:38:38,091::task::495::TaskManager.Task::(_debug) Task 687f51b5-2bd8-46e6-b56c-da9defb6a1fc: moving from state init -> state preparing >Thread-15912580::DEBUG::2013-11-20 02:38:38,093::task::495::TaskManager.Task::(_debug) Task 687f51b5-2bd8-46e6-b56c-da9defb6a1fc: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000397920608521', 'lastCheck': 1384915113.3434701, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00105786323547', 'lastCheck': 1384915114.454313, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00091814994812', 'lastCheck': 1384915114.473804, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000346899032593', 'lastCheck': 1384915111.7482569, 'valid': True, 'code': 0}} >Thread-15912580::DEBUG::2013-11-20 02:38:38,095::task::495::TaskManager.Task::(_debug) Task 687f51b5-2bd8-46e6-b56c-da9defb6a1fc: moving from state preparing -> state finished >Thread-15912580::DEBUG::2013-11-20 02:38:38,096::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912580::DEBUG::2013-11-20 02:38:38,097::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912580::DEBUG::2013-11-20 02:38:38,098::task::495::TaskManager.Task::(_debug) Task 687f51b5-2bd8-46e6-b56c-da9defb6a1fc: ref 0 aborting False >Thread-15912580::INFO::2013-11-20 02:38:38,099::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000397920608521', 'lastCheck': 1384915113.3434701, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00105786323547', 'lastCheck': 1384915114.454313, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00091814994812', 'lastCheck': 1384915114.473804, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000346899032593', 'lastCheck': 1384915111.7482569, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:38:44,488::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912586::INFO::2013-11-20 02:38:54,383::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912586::DEBUG::2013-11-20 02:38:54,384::task::495::TaskManager.Task::(_debug) Task c055f16f-d89d-4799-b4bc-012f9b87bd4b: moving from state init -> state preparing >Thread-15912586::DEBUG::2013-11-20 02:38:54,384::task::495::TaskManager.Task::(_debug) Task c055f16f-d89d-4799-b4bc-012f9b87bd4b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000313997268677', 'lastCheck': 1384915133.3674231, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00156497955322', 'lastCheck': 1384915124.4701829, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00134205818176', 'lastCheck': 1384915124.490989, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00035285949707', 'lastCheck': 1384915131.7678101, 'valid': True, 'code': 0}} >Thread-15912586::DEBUG::2013-11-20 02:38:54,385::task::495::TaskManager.Task::(_debug) Task c055f16f-d89d-4799-b4bc-012f9b87bd4b: moving from state preparing -> state finished >Thread-15912586::DEBUG::2013-11-20 02:38:54,385::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912586::DEBUG::2013-11-20 02:38:54,386::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912586::DEBUG::2013-11-20 02:38:54,387::task::495::TaskManager.Task::(_debug) Task c055f16f-d89d-4799-b4bc-012f9b87bd4b: ref 0 aborting False >Thread-15912586::INFO::2013-11-20 02:38:54,387::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000313997268677', 'lastCheck': 1384915133.3674231, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00156497955322', 'lastCheck': 1384915124.4701829, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00134205818176', 'lastCheck': 1384915124.490989, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00035285949707', 'lastCheck': 1384915131.7678101, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:38:54,505::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912392::INFO::2013-11-20 02:38:57,125::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:38:57,126::task::495::TaskManager.Task::(_debug) Task fb5789f1-c06d-4ea7-9c00-d62640fcfa29: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:38:57,127::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b9369e88-1b33-498e-b151-44f7cf84d387`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:38:57,128::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:38:57,128::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:38:57,129::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b9369e88-1b33-498e-b151-44f7cf84d387`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:38:57,130::task::495::TaskManager.Task::(_debug) Task fb5789f1-c06d-4ea7-9c00-d62640fcfa29: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:38:57,130::task::495::TaskManager.Task::(_debug) Task fb5789f1-c06d-4ea7-9c00-d62640fcfa29: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:38:57,131::task::495::TaskManager.Task::(_debug) Task fb5789f1-c06d-4ea7-9c00-d62640fcfa29: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:38:57,132::task::495::TaskManager.Task::(_debug) Task fb5789f1-c06d-4ea7-9c00-d62640fcfa29: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:38:57,132::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:38:57,133::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:38:57,133::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:38:57,134::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:38:57,134::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:38:57,135::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:38:57,135::task::495::TaskManager.Task::(_debug) Task fb5789f1-c06d-4ea7-9c00-d62640fcfa29: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:38:57,136::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-1024767::ERROR::2013-11-20 02:39:04,520::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912592::INFO::2013-11-20 02:39:10,435::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912592::DEBUG::2013-11-20 02:39:10,436::task::495::TaskManager.Task::(_debug) Task c875f357-a99f-47f6-8bc4-21b2fd3894b2: moving from state init -> state preparing >Thread-15912592::DEBUG::2013-11-20 02:39:10,436::task::495::TaskManager.Task::(_debug) Task c875f357-a99f-47f6-8bc4-21b2fd3894b2: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000282049179077', 'lastCheck': 1384915143.3783841, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.131078004837', 'lastCheck': 1384915144.631851, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.397634029388', 'lastCheck': 1384915144.918833, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000341892242432', 'lastCheck': 1384915141.7792151, 'valid': True, 'code': 0}} >Thread-15912592::DEBUG::2013-11-20 02:39:10,437::task::495::TaskManager.Task::(_debug) Task c875f357-a99f-47f6-8bc4-21b2fd3894b2: moving from state preparing -> state finished >Thread-15912592::DEBUG::2013-11-20 02:39:10,438::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912592::DEBUG::2013-11-20 02:39:10,438::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912592::DEBUG::2013-11-20 02:39:10,439::task::495::TaskManager.Task::(_debug) Task c875f357-a99f-47f6-8bc4-21b2fd3894b2: ref 0 aborting False >Thread-15912592::INFO::2013-11-20 02:39:10,439::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000282049179077', 'lastCheck': 1384915143.3783841, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.131078004837', 'lastCheck': 1384915144.631851, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.397634029388', 'lastCheck': 1384915144.918833, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000341892242432', 'lastCheck': 1384915141.7792151, 'valid': True, 'code': 0}} >Thread-1024767::ERROR::2013-11-20 02:39:14,930::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-1024767::ERROR::2013-11-20 02:39:24,944::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912598::INFO::2013-11-20 02:39:26,537::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912598::DEBUG::2013-11-20 02:39:26,537::task::495::TaskManager.Task::(_debug) Task 27fbe1ad-bb0d-4e2a-af3d-e63330d5e293: moving from state init -> state preparing >Thread-15912598::DEBUG::2013-11-20 02:39:26,538::task::495::TaskManager.Task::(_debug) Task 27fbe1ad-bb0d-4e2a-af3d-e63330d5e293: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000373840332031', 'lastCheck': 1384915163.4015219, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142788887024', 'lastCheck': 1384915164.660593, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00155401229858', 'lastCheck': 1384915164.947165, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000473022460938', 'lastCheck': 1384915161.8012991, 'valid': True, 'code': 0}} >Thread-15912598::DEBUG::2013-11-20 02:39:26,538::task::495::TaskManager.Task::(_debug) Task 27fbe1ad-bb0d-4e2a-af3d-e63330d5e293: moving from state preparing -> state finished >Thread-15912598::DEBUG::2013-11-20 02:39:26,538::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912598::DEBUG::2013-11-20 02:39:26,539::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912598::DEBUG::2013-11-20 02:39:26,539::task::495::TaskManager.Task::(_debug) Task 27fbe1ad-bb0d-4e2a-af3d-e63330d5e293: ref 0 aborting False >Thread-15912598::INFO::2013-11-20 02:39:26,539::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000373840332031', 'lastCheck': 1384915163.4015219, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142788887024', 'lastCheck': 1384915164.660593, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00155401229858', 'lastCheck': 1384915164.947165, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000473022460938', 'lastCheck': 1384915161.8012991, 'valid': True, 'code': 0}} >Thread-15912602::DEBUG::2013-11-20 02:39:34,417::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912602::INFO::2013-11-20 02:39:34,418::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912602::DEBUG::2013-11-20 02:39:34,418::task::495::TaskManager.Task::(_debug) Task de602967-a974-4eae-aa53-bbafddde71fa: moving from state init -> state preparing >Thread-15912602::DEBUG::2013-11-20 02:39:34,419::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=Free spmLver=286 spmId=1 >Thread-15912602::DEBUG::2013-11-20 02:39:34,419::task::495::TaskManager.Task::(_debug) Task de602967-a974-4eae-aa53-bbafddde71fa: finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 286}} >Thread-15912602::DEBUG::2013-11-20 02:39:34,419::task::495::TaskManager.Task::(_debug) Task de602967-a974-4eae-aa53-bbafddde71fa: moving from state preparing -> state finished >Thread-15912602::DEBUG::2013-11-20 02:39:34,420::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912602::DEBUG::2013-11-20 02:39:34,420::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912602::DEBUG::2013-11-20 02:39:34,420::task::495::TaskManager.Task::(_debug) Task de602967-a974-4eae-aa53-bbafddde71fa: ref 0 aborting False >Thread-15912602::INFO::2013-11-20 02:39:34,421::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 286}} >Thread-1024767::ERROR::2013-11-20 02:39:34,960::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912603::DEBUG::2013-11-20 02:39:35,280::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912603::INFO::2013-11-20 02:39:35,281::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: spmStart, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 prevID=1 prevLVER=286 recoveryMode=0 scsiFencing=true maxHostID=250 domVersion=2) >Thread-15912603::DEBUG::2013-11-20 02:39:35,281::task::495::TaskManager.Task::(_debug) Task 59c70706-854b-4ae3-91cc-4a2cfb6f46d4: moving from state init -> state preparing >Thread-15912603::DEBUG::2013-11-20 02:39:35,283::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f0cb8553-bebe-44fd-9325-4c4c1d7bf6f6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '330' at 'public_spmStart' >Thread-15912603::DEBUG::2013-11-20 02:39:35,283::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >Thread-15912603::DEBUG::2013-11-20 02:39:35,284::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >Thread-15912603::DEBUG::2013-11-20 02:39:35,284::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f0cb8553-bebe-44fd-9325-4c4c1d7bf6f6`::Granted request >Thread-15912603::DEBUG::2013-11-20 02:39:35,285::task::495::TaskManager.Task::(_debug) Task 59c70706-854b-4ae3-91cc-4a2cfb6f46d4: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (exclusive) >Thread-15912603::DEBUG::2013-11-20 02:39:35,286::task::495::TaskManager.Task::(_debug) Task 59c70706-854b-4ae3-91cc-4a2cfb6f46d4: ref 1 aborting False >Thread-15912603::DEBUG::2013-11-20 02:39:35,286::task::495::TaskManager.Task::(_debug) Task 59c70706-854b-4ae3-91cc-4a2cfb6f46d4: Prepare: 1 jobs exist, move to acquiring >Thread-15912603::DEBUG::2013-11-20 02:39:35,287::task::495::TaskManager.Task::(_debug) Task 59c70706-854b-4ae3-91cc-4a2cfb6f46d4: moving from state preparing -> state acquiring >Thread-15912603::DEBUG::2013-11-20 02:39:35,287::task::495::TaskManager.Task::(_debug) Task 59c70706-854b-4ae3-91cc-4a2cfb6f46d4: moving from state acquiring -> state queued >Thread-15912603::DEBUG::2013-11-20 02:39:35,288::taskManager::37::TaskManager::(_queueTask) queueing task: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912603::DEBUG::2013-11-20 02:39:35,288::taskManager::43::TaskManager::(_queueTask) task queued: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >3c078476-ceba-49ba-8c4f-cfc47f187488::DEBUG::2013-11-20 02:39:35,289::threadPool::47::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1 >Thread-15912603::DEBUG::2013-11-20 02:39:35,289::task::1183::TaskManager.Task::(prepare) returning >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:35,290::threadPool::192::Misc.ThreadPool.WorkerThread::(run) Task: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 running: <bound method Task.commit of <storage.task.Task instance at 0x7f4870055290>> with: None >Thread-15912603::DEBUG::2013-11-20 02:39:35,291::task::495::TaskManager.Task::(_debug) Task 59c70706-854b-4ae3-91cc-4a2cfb6f46d4: ref 0 aborting False >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:35,291::task::1195::TaskManager.Task::(commit) Commiting task: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912603::INFO::2013-11-20 02:39:35,292::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: spmStart, Return response: {'status': {'message': 'OK', 'code': 0}, 'uuid': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:35,293::task::495::TaskManager.Task::(_debug) Task 59c70706-854b-4ae3-91cc-4a2cfb6f46d4: moving from state queued -> state running >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:35,294::task::907::TaskManager.Task::(_runJobs) Task.run: running job 0: spmStart: <bound method SPM.start of <storage.spm.SPM instance at 0x1bf9f38>> (args: ('c4e5889a-348d-11e2-83fe-9f72369eab30', 1, '286', 0, 'true', 250, 2) kwargs: {}) >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:35,295::task::300::TaskManager.Task::(run) Job.run: running spmStart: <bound method SPM.start of <storage.spm.SPM instance at 0x1bf9f38>> (args: ('c4e5889a-348d-11e2-83fe-9f72369eab30', 1, '286', 0, 'true', 250, 2) kwargs: {}) callback None >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:35,295::spm::433::Storage.SPM::(start) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:35,295::misc::1010::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:35,296::misc::1012::SamplingMethod::(__call__) Got in to sampling method >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:35,297::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:35,298::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:35,298::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:35,299::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:35,300::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:35,301::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:35,994::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:35,997::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:35,998::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=286', u'_SHA_CKSUM=d45d960da2c767fa0cdea03c6c2e2fb94c077f58', u'POOL_SPM_ID=-1'] >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,000::misc::1020::SamplingMethod::(__call__) Returning last result >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,000::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,001::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,002::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,002::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,003::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,004::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15912604::DEBUG::2013-11-20 02:39:36,390::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912604::INFO::2013-11-20 02:39:36,392::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912604::DEBUG::2013-11-20 02:39:36,392::task::495::TaskManager.Task::(_debug) Task a61e3400-2125-44ad-b442-9dab4b0f9eb5: moving from state init -> state preparing >Thread-15912604::DEBUG::2013-11-20 02:39:36,393::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912604::DEBUG::2013-11-20 02:39:36,393::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912604::DEBUG::2013-11-20 02:39:36,394::task::495::TaskManager.Task::(_debug) Task a61e3400-2125-44ad-b442-9dab4b0f9eb5: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912604::DEBUG::2013-11-20 02:39:36,395::task::495::TaskManager.Task::(_debug) Task a61e3400-2125-44ad-b442-9dab4b0f9eb5: moving from state preparing -> state finished >Thread-15912604::DEBUG::2013-11-20 02:39:36,395::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912604::DEBUG::2013-11-20 02:39:36,396::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912604::DEBUG::2013-11-20 02:39:36,396::task::495::TaskManager.Task::(_debug) Task a61e3400-2125-44ad-b442-9dab4b0f9eb5: ref 0 aborting False >Thread-15912604::INFO::2013-11-20 02:39:36,397::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,528::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,531::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,532::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=286', u'_SHA_CKSUM=d45d960da2c767fa0cdea03c6c2e2fb94c077f58', u'POOL_SPM_ID=-1'] >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::INFO::2013-11-20 02:39:36,533::spm::457::Storage.SPM::(start) expected previd:-1 lver:286 got request for previd:1 lver:286 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,535::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,535::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,536::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,536::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,537::persistentDict::163::Storage.PersistentDict::(__init__) Created a persistant dict with LvMetadataRW backend >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,538::blockSD::164::Storage.Misc.excCmd::(readlines) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,894::blockSD::164::Storage.Misc.excCmd::(readlines) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000227041 s, 9.0 MB/s\n'; <rc> = 0 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,895::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000227041 s, 9.0 MB/s'], size: 2048 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,895::persistentDict::204::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[] >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::WARNING::2013-11-20 02:39:36,895::persistentDict::226::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,896::persistentDict::163::Storage.PersistentDict::(__init__) Created a persistant dict with VGTagMetadataRW backend >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,896::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,897::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,897::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,898::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,898::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:36,899::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:37,420::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:37,423::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:37,424::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=286', u'_SHA_CKSUM=d45d960da2c767fa0cdea03c6c2e2fb94c077f58', u'POOL_SPM_ID=-1'] >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:37,426::safelease::52::ClusterLock::(acquire) Acquiring cluster lock for domain 6b892067-ac7e-4039-a7be-5b7387525b16 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:37,427::safelease::61::Storage.Misc.excCmd::(acquire) '/usr/bin/sudo -n /usr/bin/setsid /usr/bin/ionice -c1 -n0 /bin/su vdsm -s /bin/sh -c "/usr/libexec/vdsm/spmprotect.sh start 6b892067-ac7e-4039-a7be-5b7387525b16 2 5 /dev/6b892067-ac7e-4039-a7be-5b7387525b16/leases 60000 10000 3"' (cwd /usr/libexec/vdsm/) >Thread-15912606::DEBUG::2013-11-20 02:39:37,641::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912606::INFO::2013-11-20 02:39:37,644::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912606::DEBUG::2013-11-20 02:39:37,645::task::495::TaskManager.Task::(_debug) Task 53894cc8-834f-42fd-bf22-07f48aa62fec: moving from state init -> state preparing >Thread-15912606::DEBUG::2013-11-20 02:39:37,645::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912606::DEBUG::2013-11-20 02:39:37,646::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912606::DEBUG::2013-11-20 02:39:37,646::task::495::TaskManager.Task::(_debug) Task 53894cc8-834f-42fd-bf22-07f48aa62fec: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912606::DEBUG::2013-11-20 02:39:37,647::task::495::TaskManager.Task::(_debug) Task 53894cc8-834f-42fd-bf22-07f48aa62fec: moving from state preparing -> state finished >Thread-15912606::DEBUG::2013-11-20 02:39:37,648::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912606::DEBUG::2013-11-20 02:39:37,648::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912606::DEBUG::2013-11-20 02:39:37,649::task::495::TaskManager.Task::(_debug) Task 53894cc8-834f-42fd-bf22-07f48aa62fec: ref 0 aborting False >Thread-15912606::INFO::2013-11-20 02:39:37,649::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912607::DEBUG::2013-11-20 02:39:38,659::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912607::INFO::2013-11-20 02:39:38,660::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912607::DEBUG::2013-11-20 02:39:38,661::task::495::TaskManager.Task::(_debug) Task 142e3c84-91d4-420b-ac86-49fead251ac6: moving from state init -> state preparing >Thread-15912607::DEBUG::2013-11-20 02:39:38,661::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912607::DEBUG::2013-11-20 02:39:38,662::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912607::DEBUG::2013-11-20 02:39:38,662::task::495::TaskManager.Task::(_debug) Task 142e3c84-91d4-420b-ac86-49fead251ac6: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912607::DEBUG::2013-11-20 02:39:38,663::task::495::TaskManager.Task::(_debug) Task 142e3c84-91d4-420b-ac86-49fead251ac6: moving from state preparing -> state finished >Thread-15912607::DEBUG::2013-11-20 02:39:38,663::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912607::DEBUG::2013-11-20 02:39:38,664::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912607::DEBUG::2013-11-20 02:39:38,665::task::495::TaskManager.Task::(_debug) Task 142e3c84-91d4-420b-ac86-49fead251ac6: ref 0 aborting False >Thread-15912607::INFO::2013-11-20 02:39:38,665::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912608::DEBUG::2013-11-20 02:39:39,679::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912608::INFO::2013-11-20 02:39:39,680::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912608::DEBUG::2013-11-20 02:39:39,681::task::495::TaskManager.Task::(_debug) Task 5789907b-890e-4fd1-9787-cef71af4b513: moving from state init -> state preparing >Thread-15912608::DEBUG::2013-11-20 02:39:39,681::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912608::DEBUG::2013-11-20 02:39:39,682::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912608::DEBUG::2013-11-20 02:39:39,682::task::495::TaskManager.Task::(_debug) Task 5789907b-890e-4fd1-9787-cef71af4b513: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912608::DEBUG::2013-11-20 02:39:39,683::task::495::TaskManager.Task::(_debug) Task 5789907b-890e-4fd1-9787-cef71af4b513: moving from state preparing -> state finished >Thread-15912608::DEBUG::2013-11-20 02:39:39,684::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912608::DEBUG::2013-11-20 02:39:39,684::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912608::DEBUG::2013-11-20 02:39:39,685::task::495::TaskManager.Task::(_debug) Task 5789907b-890e-4fd1-9787-cef71af4b513: ref 0 aborting False >Thread-15912608::INFO::2013-11-20 02:39:39,685::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912610::DEBUG::2013-11-20 02:39:42,225::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912610::INFO::2013-11-20 02:39:42,226::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912610::DEBUG::2013-11-20 02:39:42,226::task::495::TaskManager.Task::(_debug) Task a7ca7cb0-f868-45e7-8bbb-0398f815e0ad: moving from state init -> state preparing >Thread-15912610::DEBUG::2013-11-20 02:39:42,227::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912610::DEBUG::2013-11-20 02:39:42,227::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912610::DEBUG::2013-11-20 02:39:42,228::task::495::TaskManager.Task::(_debug) Task a7ca7cb0-f868-45e7-8bbb-0398f815e0ad: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912610::DEBUG::2013-11-20 02:39:42,229::task::495::TaskManager.Task::(_debug) Task a7ca7cb0-f868-45e7-8bbb-0398f815e0ad: moving from state preparing -> state finished >Thread-15912610::DEBUG::2013-11-20 02:39:42,229::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912610::DEBUG::2013-11-20 02:39:42,230::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912610::DEBUG::2013-11-20 02:39:42,230::task::495::TaskManager.Task::(_debug) Task a7ca7cb0-f868-45e7-8bbb-0398f815e0ad: ref 0 aborting False >Thread-15912610::INFO::2013-11-20 02:39:42,231::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912611::INFO::2013-11-20 02:39:42,806::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912611::DEBUG::2013-11-20 02:39:42,807::task::495::TaskManager.Task::(_debug) Task 8e19232c-40c5-4222-86bc-dcb8452728d9: moving from state init -> state preparing >Thread-15912611::DEBUG::2013-11-20 02:39:42,830::task::495::TaskManager.Task::(_debug) Task 8e19232c-40c5-4222-86bc-dcb8452728d9: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000296115875244', 'lastCheck': 1384915173.4126129, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00160193443298', 'lastCheck': 1384915174.6759131, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00136685371399', 'lastCheck': 1384915174.963186, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000437021255493', 'lastCheck': 1384915181.8235841, 'valid': True, 'code': 0}} >Thread-15912611::DEBUG::2013-11-20 02:39:42,830::task::495::TaskManager.Task::(_debug) Task 8e19232c-40c5-4222-86bc-dcb8452728d9: moving from state preparing -> state finished >Thread-15912611::DEBUG::2013-11-20 02:39:42,831::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912611::DEBUG::2013-11-20 02:39:42,831::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912611::DEBUG::2013-11-20 02:39:42,832::task::495::TaskManager.Task::(_debug) Task 8e19232c-40c5-4222-86bc-dcb8452728d9: ref 0 aborting False >Thread-15912611::INFO::2013-11-20 02:39:42,833::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000296115875244', 'lastCheck': 1384915173.4126129, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00160193443298', 'lastCheck': 1384915174.6759131, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00136685371399', 'lastCheck': 1384915174.963186, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000437021255493', 'lastCheck': 1384915181.8235841, 'valid': True, 'code': 0}} >Thread-15912613::DEBUG::2013-11-20 02:39:43,240::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912613::INFO::2013-11-20 02:39:43,241::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912613::DEBUG::2013-11-20 02:39:43,241::task::495::TaskManager.Task::(_debug) Task 95aba5e7-b920-4002-8035-4bba0bed81e7: moving from state init -> state preparing >Thread-15912613::DEBUG::2013-11-20 02:39:43,242::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912613::DEBUG::2013-11-20 02:39:43,242::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912613::DEBUG::2013-11-20 02:39:43,243::task::495::TaskManager.Task::(_debug) Task 95aba5e7-b920-4002-8035-4bba0bed81e7: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912613::DEBUG::2013-11-20 02:39:43,243::task::495::TaskManager.Task::(_debug) Task 95aba5e7-b920-4002-8035-4bba0bed81e7: moving from state preparing -> state finished >Thread-15912613::DEBUG::2013-11-20 02:39:43,244::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912613::DEBUG::2013-11-20 02:39:43,244::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912613::DEBUG::2013-11-20 02:39:43,245::task::495::TaskManager.Task::(_debug) Task 95aba5e7-b920-4002-8035-4bba0bed81e7: ref 0 aborting False >Thread-15912613::INFO::2013-11-20 02:39:43,245::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912614::DEBUG::2013-11-20 02:39:44,259::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912614::INFO::2013-11-20 02:39:44,260::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912614::DEBUG::2013-11-20 02:39:44,261::task::495::TaskManager.Task::(_debug) Task 4655205c-51ab-4d13-9b72-5cbe9726e343: moving from state init -> state preparing >Thread-15912614::DEBUG::2013-11-20 02:39:44,261::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912614::DEBUG::2013-11-20 02:39:44,262::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912614::DEBUG::2013-11-20 02:39:44,262::task::495::TaskManager.Task::(_debug) Task 4655205c-51ab-4d13-9b72-5cbe9726e343: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912614::DEBUG::2013-11-20 02:39:44,263::task::495::TaskManager.Task::(_debug) Task 4655205c-51ab-4d13-9b72-5cbe9726e343: moving from state preparing -> state finished >Thread-15912614::DEBUG::2013-11-20 02:39:44,263::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912614::DEBUG::2013-11-20 02:39:44,264::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912614::DEBUG::2013-11-20 02:39:44,264::task::495::TaskManager.Task::(_debug) Task 4655205c-51ab-4d13-9b72-5cbe9726e343: ref 0 aborting False >Thread-15912614::INFO::2013-11-20 02:39:44,265::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-1024767::ERROR::2013-11-20 02:39:45,090::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912615::DEBUG::2013-11-20 02:39:45,274::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912615::INFO::2013-11-20 02:39:45,275::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912615::DEBUG::2013-11-20 02:39:45,276::task::495::TaskManager.Task::(_debug) Task 1dcf3e44-4f98-4e53-be84-b439574c56d0: moving from state init -> state preparing >Thread-15912615::DEBUG::2013-11-20 02:39:45,276::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912615::DEBUG::2013-11-20 02:39:45,277::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912615::DEBUG::2013-11-20 02:39:45,277::task::495::TaskManager.Task::(_debug) Task 1dcf3e44-4f98-4e53-be84-b439574c56d0: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912615::DEBUG::2013-11-20 02:39:45,278::task::495::TaskManager.Task::(_debug) Task 1dcf3e44-4f98-4e53-be84-b439574c56d0: moving from state preparing -> state finished >Thread-15912615::DEBUG::2013-11-20 02:39:45,278::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912615::DEBUG::2013-11-20 02:39:45,279::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912615::DEBUG::2013-11-20 02:39:45,279::task::495::TaskManager.Task::(_debug) Task 1dcf3e44-4f98-4e53-be84-b439574c56d0: ref 0 aborting False >Thread-15912615::INFO::2013-11-20 02:39:45,280::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912616::DEBUG::2013-11-20 02:39:46,289::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912616::INFO::2013-11-20 02:39:46,291::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912616::DEBUG::2013-11-20 02:39:46,291::task::495::TaskManager.Task::(_debug) Task c14a2c5d-be78-4cf5-a17f-e26a5d98b616: moving from state init -> state preparing >Thread-15912616::DEBUG::2013-11-20 02:39:46,292::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912616::DEBUG::2013-11-20 02:39:46,292::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912616::DEBUG::2013-11-20 02:39:46,293::task::495::TaskManager.Task::(_debug) Task c14a2c5d-be78-4cf5-a17f-e26a5d98b616: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912616::DEBUG::2013-11-20 02:39:46,293::task::495::TaskManager.Task::(_debug) Task c14a2c5d-be78-4cf5-a17f-e26a5d98b616: moving from state preparing -> state finished >Thread-15912616::DEBUG::2013-11-20 02:39:46,294::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912616::DEBUG::2013-11-20 02:39:46,294::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912616::DEBUG::2013-11-20 02:39:46,295::task::495::TaskManager.Task::(_debug) Task c14a2c5d-be78-4cf5-a17f-e26a5d98b616: ref 0 aborting False >Thread-15912616::INFO::2013-11-20 02:39:46,295::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912618::DEBUG::2013-11-20 02:39:47,306::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912618::INFO::2013-11-20 02:39:47,308::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912618::DEBUG::2013-11-20 02:39:47,308::task::495::TaskManager.Task::(_debug) Task 8dce0a05-777f-4595-9c98-09b25f7f8d15: moving from state init -> state preparing >Thread-15912618::DEBUG::2013-11-20 02:39:47,309::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912618::DEBUG::2013-11-20 02:39:47,309::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912618::DEBUG::2013-11-20 02:39:47,310::task::495::TaskManager.Task::(_debug) Task 8dce0a05-777f-4595-9c98-09b25f7f8d15: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912618::DEBUG::2013-11-20 02:39:47,310::task::495::TaskManager.Task::(_debug) Task 8dce0a05-777f-4595-9c98-09b25f7f8d15: moving from state preparing -> state finished >Thread-15912618::DEBUG::2013-11-20 02:39:47,311::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912618::DEBUG::2013-11-20 02:39:47,311::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912618::DEBUG::2013-11-20 02:39:47,312::task::495::TaskManager.Task::(_debug) Task 8dce0a05-777f-4595-9c98-09b25f7f8d15: ref 0 aborting False >Thread-15912618::INFO::2013-11-20 02:39:47,312::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912619::DEBUG::2013-11-20 02:39:48,322::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912619::INFO::2013-11-20 02:39:48,323::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912619::DEBUG::2013-11-20 02:39:48,324::task::495::TaskManager.Task::(_debug) Task d03618af-c27a-4f8e-82a0-484dcf3f3419: moving from state init -> state preparing >Thread-15912619::DEBUG::2013-11-20 02:39:48,325::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912619::DEBUG::2013-11-20 02:39:48,325::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912619::DEBUG::2013-11-20 02:39:48,326::task::495::TaskManager.Task::(_debug) Task d03618af-c27a-4f8e-82a0-484dcf3f3419: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912619::DEBUG::2013-11-20 02:39:48,326::task::495::TaskManager.Task::(_debug) Task d03618af-c27a-4f8e-82a0-484dcf3f3419: moving from state preparing -> state finished >Thread-15912619::DEBUG::2013-11-20 02:39:48,327::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912619::DEBUG::2013-11-20 02:39:48,327::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912619::DEBUG::2013-11-20 02:39:48,328::task::495::TaskManager.Task::(_debug) Task d03618af-c27a-4f8e-82a0-484dcf3f3419: ref 0 aborting False >Thread-15912619::INFO::2013-11-20 02:39:48,328::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912620::DEBUG::2013-11-20 02:39:49,339::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912620::INFO::2013-11-20 02:39:49,341::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912620::DEBUG::2013-11-20 02:39:49,342::task::495::TaskManager.Task::(_debug) Task 96cd8d6a-2eb7-4232-a523-4bc4fba46fd6: moving from state init -> state preparing >Thread-15912620::DEBUG::2013-11-20 02:39:49,342::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912620::DEBUG::2013-11-20 02:39:49,343::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912620::DEBUG::2013-11-20 02:39:49,344::task::495::TaskManager.Task::(_debug) Task 96cd8d6a-2eb7-4232-a523-4bc4fba46fd6: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912620::DEBUG::2013-11-20 02:39:49,345::task::495::TaskManager.Task::(_debug) Task 96cd8d6a-2eb7-4232-a523-4bc4fba46fd6: moving from state preparing -> state finished >Thread-15912620::DEBUG::2013-11-20 02:39:49,345::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912620::DEBUG::2013-11-20 02:39:49,346::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912620::DEBUG::2013-11-20 02:39:49,347::task::495::TaskManager.Task::(_debug) Task 96cd8d6a-2eb7-4232-a523-4bc4fba46fd6: ref 0 aborting False >Thread-15912620::INFO::2013-11-20 02:39:49,348::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912622::DEBUG::2013-11-20 02:39:50,490::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912622::INFO::2013-11-20 02:39:50,492::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912622::DEBUG::2013-11-20 02:39:50,493::task::495::TaskManager.Task::(_debug) Task 07da0950-bb80-4ca1-b18d-25538c2205ca: moving from state init -> state preparing >Thread-15912622::DEBUG::2013-11-20 02:39:50,494::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912622::DEBUG::2013-11-20 02:39:50,495::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912622::DEBUG::2013-11-20 02:39:50,496::task::495::TaskManager.Task::(_debug) Task 07da0950-bb80-4ca1-b18d-25538c2205ca: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912622::DEBUG::2013-11-20 02:39:50,497::task::495::TaskManager.Task::(_debug) Task 07da0950-bb80-4ca1-b18d-25538c2205ca: moving from state preparing -> state finished >Thread-15912622::DEBUG::2013-11-20 02:39:50,497::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912622::DEBUG::2013-11-20 02:39:50,498::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912622::DEBUG::2013-11-20 02:39:50,499::task::495::TaskManager.Task::(_debug) Task 07da0950-bb80-4ca1-b18d-25538c2205ca: ref 0 aborting False >Thread-15912622::INFO::2013-11-20 02:39:50,500::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912623::DEBUG::2013-11-20 02:39:51,511::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912623::INFO::2013-11-20 02:39:51,512::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912623::DEBUG::2013-11-20 02:39:51,513::task::495::TaskManager.Task::(_debug) Task fa4abc5d-8f60-4dcb-bfcd-9f3be8cf234b: moving from state init -> state preparing >Thread-15912623::DEBUG::2013-11-20 02:39:51,513::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912623::DEBUG::2013-11-20 02:39:51,514::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912623::DEBUG::2013-11-20 02:39:51,514::task::495::TaskManager.Task::(_debug) Task fa4abc5d-8f60-4dcb-bfcd-9f3be8cf234b: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912623::DEBUG::2013-11-20 02:39:51,515::task::495::TaskManager.Task::(_debug) Task fa4abc5d-8f60-4dcb-bfcd-9f3be8cf234b: moving from state preparing -> state finished >Thread-15912623::DEBUG::2013-11-20 02:39:51,515::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912623::DEBUG::2013-11-20 02:39:51,516::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912623::DEBUG::2013-11-20 02:39:51,516::task::495::TaskManager.Task::(_debug) Task fa4abc5d-8f60-4dcb-bfcd-9f3be8cf234b: ref 0 aborting False >Thread-15912623::INFO::2013-11-20 02:39:51,517::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912624::DEBUG::2013-11-20 02:39:52,528::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912624::INFO::2013-11-20 02:39:52,530::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912624::DEBUG::2013-11-20 02:39:52,531::task::495::TaskManager.Task::(_debug) Task 568c0a63-a955-4623-a205-eeae068cc5e6: moving from state init -> state preparing >Thread-15912624::DEBUG::2013-11-20 02:39:52,531::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912624::DEBUG::2013-11-20 02:39:52,532::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912624::DEBUG::2013-11-20 02:39:52,533::task::495::TaskManager.Task::(_debug) Task 568c0a63-a955-4623-a205-eeae068cc5e6: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912624::DEBUG::2013-11-20 02:39:52,533::task::495::TaskManager.Task::(_debug) Task 568c0a63-a955-4623-a205-eeae068cc5e6: moving from state preparing -> state finished >Thread-15912624::DEBUG::2013-11-20 02:39:52,534::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912624::DEBUG::2013-11-20 02:39:52,534::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912624::DEBUG::2013-11-20 02:39:52,535::task::495::TaskManager.Task::(_debug) Task 568c0a63-a955-4623-a205-eeae068cc5e6: ref 0 aborting False >Thread-15912624::INFO::2013-11-20 02:39:52,535::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912626::DEBUG::2013-11-20 02:39:53,546::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912626::INFO::2013-11-20 02:39:53,547::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912626::DEBUG::2013-11-20 02:39:53,547::task::495::TaskManager.Task::(_debug) Task c24771bb-9f92-414f-9405-dd013dca9d06: moving from state init -> state preparing >Thread-15912626::DEBUG::2013-11-20 02:39:53,548::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912626::DEBUG::2013-11-20 02:39:53,548::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912626::DEBUG::2013-11-20 02:39:53,549::task::495::TaskManager.Task::(_debug) Task c24771bb-9f92-414f-9405-dd013dca9d06: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912626::DEBUG::2013-11-20 02:39:53,550::task::495::TaskManager.Task::(_debug) Task c24771bb-9f92-414f-9405-dd013dca9d06: moving from state preparing -> state finished >Thread-15912626::DEBUG::2013-11-20 02:39:53,550::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912626::DEBUG::2013-11-20 02:39:53,551::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912626::DEBUG::2013-11-20 02:39:53,551::task::495::TaskManager.Task::(_debug) Task c24771bb-9f92-414f-9405-dd013dca9d06: ref 0 aborting False >Thread-15912626::INFO::2013-11-20 02:39:53,552::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912627::DEBUG::2013-11-20 02:39:54,561::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912627::INFO::2013-11-20 02:39:54,562::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912627::DEBUG::2013-11-20 02:39:54,563::task::495::TaskManager.Task::(_debug) Task 1bb32cdb-ca93-4f62-be78-dbac6519f049: moving from state init -> state preparing >Thread-15912627::DEBUG::2013-11-20 02:39:54,563::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912627::DEBUG::2013-11-20 02:39:54,564::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912627::DEBUG::2013-11-20 02:39:54,564::task::495::TaskManager.Task::(_debug) Task 1bb32cdb-ca93-4f62-be78-dbac6519f049: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912627::DEBUG::2013-11-20 02:39:54,565::task::495::TaskManager.Task::(_debug) Task 1bb32cdb-ca93-4f62-be78-dbac6519f049: moving from state preparing -> state finished >Thread-15912627::DEBUG::2013-11-20 02:39:54,565::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912627::DEBUG::2013-11-20 02:39:54,566::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912627::DEBUG::2013-11-20 02:39:54,566::task::495::TaskManager.Task::(_debug) Task 1bb32cdb-ca93-4f62-be78-dbac6519f049: ref 0 aborting False >Thread-15912627::INFO::2013-11-20 02:39:54,567::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-1024767::ERROR::2013-11-20 02:39:55,105::sp::103::Storage.StatsThread::(run) Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 98, in run > File "/usr/share/vdsm/storage/sp.py", line 1314, in _repostats > File "/usr/share/vdsm/storage/sd.py", line 437, in validateMaster > File "/usr/share/vdsm/storage/sd.py", line 660, in isMaster > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'ROLE' >Thread-15912628::DEBUG::2013-11-20 02:39:55,577::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912628::INFO::2013-11-20 02:39:55,578::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912628::DEBUG::2013-11-20 02:39:55,579::task::495::TaskManager.Task::(_debug) Task 995ecd5f-eeb9-401a-b7a0-0bfc25fc3033: moving from state init -> state preparing >Thread-15912628::DEBUG::2013-11-20 02:39:55,579::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912628::DEBUG::2013-11-20 02:39:55,580::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912628::DEBUG::2013-11-20 02:39:55,580::task::495::TaskManager.Task::(_debug) Task 995ecd5f-eeb9-401a-b7a0-0bfc25fc3033: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912628::DEBUG::2013-11-20 02:39:55,581::task::495::TaskManager.Task::(_debug) Task 995ecd5f-eeb9-401a-b7a0-0bfc25fc3033: moving from state preparing -> state finished >Thread-15912628::DEBUG::2013-11-20 02:39:55,581::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912628::DEBUG::2013-11-20 02:39:55,582::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912628::DEBUG::2013-11-20 02:39:55,582::task::495::TaskManager.Task::(_debug) Task 995ecd5f-eeb9-401a-b7a0-0bfc25fc3033: ref 0 aborting False >Thread-15912628::INFO::2013-11-20 02:39:55,583::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912630::DEBUG::2013-11-20 02:39:56,593::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912630::INFO::2013-11-20 02:39:56,594::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912630::DEBUG::2013-11-20 02:39:56,594::task::495::TaskManager.Task::(_debug) Task dcfc9085-d39e-4af9-9db9-eab11c342223: moving from state init -> state preparing >Thread-15912630::DEBUG::2013-11-20 02:39:56,595::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912630::DEBUG::2013-11-20 02:39:56,595::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912630::DEBUG::2013-11-20 02:39:56,596::task::495::TaskManager.Task::(_debug) Task dcfc9085-d39e-4af9-9db9-eab11c342223: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912630::DEBUG::2013-11-20 02:39:56,597::task::495::TaskManager.Task::(_debug) Task dcfc9085-d39e-4af9-9db9-eab11c342223: moving from state preparing -> state finished >Thread-15912630::DEBUG::2013-11-20 02:39:56,597::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912630::DEBUG::2013-11-20 02:39:56,598::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912630::DEBUG::2013-11-20 02:39:56,598::task::495::TaskManager.Task::(_debug) Task dcfc9085-d39e-4af9-9db9-eab11c342223: ref 0 aborting False >Thread-15912630::INFO::2013-11-20 02:39:56,599::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912392::INFO::2013-11-20 02:39:57,201::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:39:57,202::task::495::TaskManager.Task::(_debug) Task f384f876-1b1f-4256-9f64-d8e923528cb7: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:39:57,204::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`22653d9c-c200-4331-850a-3e998a2d5a62`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:39:57,204::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:39:57,205::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:39:57,206::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`22653d9c-c200-4331-850a-3e998a2d5a62`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:39:57,207::task::495::TaskManager.Task::(_debug) Task f384f876-1b1f-4256-9f64-d8e923528cb7: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:39:57,207::task::495::TaskManager.Task::(_debug) Task f384f876-1b1f-4256-9f64-d8e923528cb7: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:39:57,208::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15912392::DEBUG::2013-11-20 02:39:57,209::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15912631::DEBUG::2013-11-20 02:39:57,608::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912631::INFO::2013-11-20 02:39:57,609::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912631::DEBUG::2013-11-20 02:39:57,610::task::495::TaskManager.Task::(_debug) Task 3f9ac69d-e188-4618-926a-5022c9955b40: moving from state init -> state preparing >Thread-15912631::DEBUG::2013-11-20 02:39:57,610::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912631::DEBUG::2013-11-20 02:39:57,610::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912631::DEBUG::2013-11-20 02:39:57,611::task::495::TaskManager.Task::(_debug) Task 3f9ac69d-e188-4618-926a-5022c9955b40: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912631::DEBUG::2013-11-20 02:39:57,611::task::495::TaskManager.Task::(_debug) Task 3f9ac69d-e188-4618-926a-5022c9955b40: moving from state preparing -> state finished >Thread-15912631::DEBUG::2013-11-20 02:39:57,612::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912631::DEBUG::2013-11-20 02:39:57,612::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912631::DEBUG::2013-11-20 02:39:57,612::task::495::TaskManager.Task::(_debug) Task 3f9ac69d-e188-4618-926a-5022c9955b40: ref 0 aborting False >Thread-15912631::INFO::2013-11-20 02:39:57,613::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:57,869::safelease::61::Storage.Misc.excCmd::(acquire) SUCCESS: <err> = ''; <rc> = 0 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:57,872::safelease::65::ClusterLock::(acquire) Clustered lock acquired successfully >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:57,874::spm::468::Storage.SPM::(start) spm lock acquired successfully >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:57,876::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting... >Thread-15912392::DEBUG::2013-11-20 02:39:57,883::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912392::DEBUG::2013-11-20 02:39:58,129::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15912392::DEBUG::2013-11-20 02:39:58,130::task::495::TaskManager.Task::(_debug) Task f384f876-1b1f-4256-9f64-d8e923528cb7: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:58,131::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15912392::DEBUG::2013-11-20 02:39:58,131::task::495::TaskManager.Task::(_debug) Task f384f876-1b1f-4256-9f64-d8e923528cb7: moving from state preparing -> state finished >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:58,133::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15912392::DEBUG::2013-11-20 02:39:58,133::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:58,134::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15912392::DEBUG::2013-11-20 02:39:58,135::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:58,136::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15912392::DEBUG::2013-11-20 02:39:58,137::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:58,138::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15912392::DEBUG::2013-11-20 02:39:58,139::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:58,141::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15912392::DEBUG::2013-11-20 02:39:58,141::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:39:58,142::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:39:58,379::task::495::TaskManager.Task::(_debug) Task f384f876-1b1f-4256-9f64-d8e923528cb7: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:39:58,381::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912632::DEBUG::2013-11-20 02:39:58,627::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912632::INFO::2013-11-20 02:39:58,629::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912632::DEBUG::2013-11-20 02:39:58,629::task::495::TaskManager.Task::(_debug) Task 9ba0234a-e9a5-4bfb-ba9e-1ae9b92fde4b: moving from state init -> state preparing >Thread-15912632::DEBUG::2013-11-20 02:39:58,630::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912632::DEBUG::2013-11-20 02:39:58,631::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912632::DEBUG::2013-11-20 02:39:58,631::task::495::TaskManager.Task::(_debug) Task 9ba0234a-e9a5-4bfb-ba9e-1ae9b92fde4b: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912632::DEBUG::2013-11-20 02:39:58,632::task::495::TaskManager.Task::(_debug) Task 9ba0234a-e9a5-4bfb-ba9e-1ae9b92fde4b: moving from state preparing -> state finished >Thread-15912632::DEBUG::2013-11-20 02:39:58,632::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912632::DEBUG::2013-11-20 02:39:58,633::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912632::DEBUG::2013-11-20 02:39:58,633::task::495::TaskManager.Task::(_debug) Task 9ba0234a-e9a5-4bfb-ba9e-1ae9b92fde4b: ref 0 aborting False >Thread-15912632::INFO::2013-11-20 02:39:58,634::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:58,821::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:58,824::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:58,825::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_LVER=286', u'_SHA_CKSUM=d45d960da2c767fa0cdea03c6c2e2fb94c077f58', u'POOL_SPM_ID=-1'] >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:58,826::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:58,827::persistentDict::146::Storage.PersistentDict::(transaction) Flushing changes >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:58,827::persistentDict::265::Storage.PersistentDict::(flush) about to write lines (VGTagMetadataRW)=[u'CLASS=Data', u'DESCRIPTION=OS', u'IOOPTIMEOUTSEC=10', u'LEASERETRIES=3', u'LEASETIMESEC=60', u'LOCKPOLICY=', u'LOCKRENEWALINTERVALSEC=5', u'LOGBLKSIZE=512', u'MASTER_VERSION=1', u'PHYBLKSIZE=512', u'POOL_DESCRIPTION=Default', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'POOL_SPM_ID=2', u'POOL_SPM_LVER=287', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'ROLE=Master', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'TYPE=FCP', u'VERSION=2', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', '_SHA_CKSUM=716ea7492d9134a4b13de27ac61a336ad474727c'] >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:58,828::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:58,829::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:58,830::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:58,830::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:58,831::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:58,832::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15912633::INFO::2013-11-20 02:39:59,092::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912633::DEBUG::2013-11-20 02:39:59,094::task::495::TaskManager.Task::(_debug) Task 22dda694-6fc7-42eb-9b6b-8c417ea6e60f: moving from state init -> state preparing >Thread-15912633::DEBUG::2013-11-20 02:39:59,097::task::495::TaskManager.Task::(_debug) Task 22dda694-6fc7-42eb-9b6b-8c417ea6e60f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000309944152832', 'lastCheck': 1384915193.6592431, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0448310375214', 'lastCheck': 1384915194.825798, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0895879268646', 'lastCheck': 1384915195.1958771, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000488996505737', 'lastCheck': 1384915191.8335409, 'valid': True, 'code': 0}} >Thread-15912633::DEBUG::2013-11-20 02:39:59,106::task::495::TaskManager.Task::(_debug) Task 22dda694-6fc7-42eb-9b6b-8c417ea6e60f: moving from state preparing -> state finished >Thread-15912633::DEBUG::2013-11-20 02:39:59,107::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912633::DEBUG::2013-11-20 02:39:59,108::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912633::DEBUG::2013-11-20 02:39:59,109::task::495::TaskManager.Task::(_debug) Task 22dda694-6fc7-42eb-9b6b-8c417ea6e60f: ref 0 aborting False >Thread-15912633::INFO::2013-11-20 02:39:59,109::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000309944152832', 'lastCheck': 1384915193.6592431, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0448310375214', 'lastCheck': 1384915194.825798, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0895879268646', 'lastCheck': 1384915195.1958771, 'valid': False, 'code': 200}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000488996505737', 'lastCheck': 1384915191.8335409, 'valid': True, 'code': 0}} >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:59,436::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:59,439::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:59,440::blockSD::145::storage.Metadata.VGTagMetadataRW::(writelines) Updating metadata adding=MDT_POOL_SPM_LVER=287, MDT__SHA_CKSUM=716ea7492d9134a4b13de27ac61a336ad474727c, MDT_POOL_SPM_ID=2 removing=MDT_POOL_SPM_ID=-1, MDT_POOL_SPM_LVER=286, MDT__SHA_CKSUM=d45d960da2c767fa0cdea03c6c2e2fb94c077f58 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:39:59,441::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --deltag MDT__SHA_CKSUM=d45d960da2c767fa0cdea03c6c2e2fb94c077f58 --deltag MDT_POOL_SPM_ID=-1 --deltag MDT_POOL_SPM_LVER=286 --addtag MDT__SHA_CKSUM=716ea7492d9134a4b13de27ac61a336ad474727c --addtag MDT_POOL_SPM_ID=2 --addtag MDT_POOL_SPM_LVER=287 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15912635::DEBUG::2013-11-20 02:39:59,663::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912635::INFO::2013-11-20 02:39:59,666::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912635::DEBUG::2013-11-20 02:39:59,666::task::495::TaskManager.Task::(_debug) Task a5b1348d-20f4-4822-98ff-027d04c83619: moving from state init -> state preparing >Thread-15912635::DEBUG::2013-11-20 02:39:59,667::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912635::DEBUG::2013-11-20 02:39:59,667::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912635::DEBUG::2013-11-20 02:39:59,668::task::495::TaskManager.Task::(_debug) Task a5b1348d-20f4-4822-98ff-027d04c83619: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912635::DEBUG::2013-11-20 02:39:59,669::task::495::TaskManager.Task::(_debug) Task a5b1348d-20f4-4822-98ff-027d04c83619: moving from state preparing -> state finished >Thread-15912635::DEBUG::2013-11-20 02:39:59,669::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912635::DEBUG::2013-11-20 02:39:59,670::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912635::DEBUG::2013-11-20 02:39:59,670::task::495::TaskManager.Task::(_debug) Task a5b1348d-20f4-4822-98ff-027d04c83619: ref 0 aborting False >Thread-15912635::INFO::2013-11-20 02:39:59,671::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,166::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,168::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,169::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,169::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,170::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f6d18b26-248b-4d01-a2c1-b87667e945f7`::Request was made in '/usr/share/vdsm/storage/spm.py' line '330' at '_upgradePool' >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,171::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'exclusive' >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,172::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'exclusive' (1 active user) >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,172::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f6d18b26-248b-4d01-a2c1-b87667e945f7`::Granted request >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::INFO::2013-11-20 02:40:00,173::spm::336::Storage.SPM::(_upgradePool) Trying to upgrade master domain `6b892067-ac7e-4039-a7be-5b7387525b16` >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,174::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`fdf6264a-962e-4fa0-a9c7-9bef6c23ba02`::Request was made in '/usr/share/vdsm/storage/spm.py' line '337' at '_upgradePool' >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,175::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'exclusive' >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,175::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'exclusive' (1 active user) >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,176::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`fdf6264a-962e-4fa0-a9c7-9bef6c23ba02`::Granted request >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,177::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,177::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,178::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,179::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,179::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,180::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15912636::DEBUG::2013-11-20 02:40:00,682::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912636::INFO::2013-11-20 02:40:00,683::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912636::DEBUG::2013-11-20 02:40:00,684::task::495::TaskManager.Task::(_debug) Task 71c47a34-3fc8-415d-8f00-332a2ffd587f: moving from state init -> state preparing >Thread-15912636::DEBUG::2013-11-20 02:40:00,684::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912636::DEBUG::2013-11-20 02:40:00,685::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912636::DEBUG::2013-11-20 02:40:00,686::task::495::TaskManager.Task::(_debug) Task 71c47a34-3fc8-415d-8f00-332a2ffd587f: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912636::DEBUG::2013-11-20 02:40:00,686::task::495::TaskManager.Task::(_debug) Task 71c47a34-3fc8-415d-8f00-332a2ffd587f: moving from state preparing -> state finished >Thread-15912636::DEBUG::2013-11-20 02:40:00,687::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912636::DEBUG::2013-11-20 02:40:00,687::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912636::DEBUG::2013-11-20 02:40:00,688::task::495::TaskManager.Task::(_debug) Task 71c47a34-3fc8-415d-8f00-332a2ffd587f: ref 0 aborting False >Thread-15912636::INFO::2013-11-20 02:40:00,688::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912637::DEBUG::2013-11-20 02:40:00,811::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912637::INFO::2013-11-20 02:40:00,811::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: validateStorageServerConnection, args: (domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{'connection': 'kairo.brd-klee.dk:/export', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36', 'port': ''}]) >Thread-15912637::DEBUG::2013-11-20 02:40:00,812::task::495::TaskManager.Task::(_debug) Task c068ce1b-3a5e-4369-ab64-0cd423cbc79f: moving from state init -> state preparing >Thread-15912637::INFO::2013-11-20 02:40:00,812::storage_connection::127::Storage.ServerConnection::(validate) Request to validate NFS storage server >Thread-15912637::INFO::2013-11-20 02:40:00,812::storage_connection::53::Storage.ServerConnection::(__validateConnectionParams) conList=[{'connection': 'kairo.brd-klee.dk:/export', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36', 'port': ''}] >Thread-15912637::DEBUG::2013-11-20 02:40:00,815::fileUtils::83::Storage.Misc.excCmd::(mount) '/usr/bin/sudo -n /bin/mount -o soft,timeo=600,retrans=6,nosharecache,vers=3 -t nfs kairo.brd-klee.dk:/export /tmp/tmpgimtBa' (cwd None) >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,841::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,844::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,845::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'_SHA_CKSUM=716ea7492d9134a4b13de27ac61a336ad474727c', u'POOL_SPM_ID=2', u'POOL_SPM_LVER=287'] >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,846::blockSD::625::Storage.StorageDomain::(upgrade) Trying to upgrade domain `6b892067-ac7e-4039-a7be-5b7387525b16` from version 2 to version 2 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,847::blockSD::631::Storage.StorageDomain::(upgrade) No need to upgrade domain `6b892067-ac7e-4039-a7be-5b7387525b16`, leaving unchanged >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,847::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,848::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,849::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,849::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,850::spm::340::Storage.SPM::(_upgradePool) Marking all domains for upgrade >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,850::spm::347::Storage.SPM::(_upgradePool) Registering with state change event >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,851::spm::349::Storage.SPM::(_upgradePool) Running initial domain upgrade threads >Thread-15912638::DEBUG::2013-11-20 02:40:00,852::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 45cdff8b-89c0-4926-a8c7-80354064bfa6 >Thread-15912638::DEBUG::2013-11-20 02:40:00,856::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`c72e1389-8910-4ceb-9eaa-b8511b783f6d`::Request was made in '/usr/share/vdsm/storage/spm.py' line '376' at '_upgradePoolDomain' >Thread-15912639::DEBUG::2013-11-20 02:40:00,857::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 772472cb-2429-466d-96cd-00b3301ef78a >Thread-15912638::DEBUG::2013-11-20 02:40:00,858::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' for lock type 'exclusive' >Thread-15912640::DEBUG::2013-11-20 02:40:00,858::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 5d7b5178-3790-472c-a13b-b481eb27a61e >Thread-15912639::DEBUG::2013-11-20 02:40:00,861::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`daa5fb58-109f-4f40-b8a9-d4afee706da5`::Request was made in '/usr/share/vdsm/storage/spm.py' line '376' at '_upgradePoolDomain' >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,861::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912638::DEBUG::2013-11-20 02:40:00,862::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' is free. Now locking as 'exclusive' (1 active user) >Thread-15912640::ERROR::2013-11-20 02:40:00,863::spm::373::Storage.SPM::(_upgradePoolDomain) Error while preparing domain `5d7b5178-3790-472c-a13b-b481eb27a61e` upgrade >Traceback (most recent call last): > File "/usr/share/vdsm/storage/spm.py", line 369, in _upgradePoolDomain >IndexError: list index out of range >Thread-15912639::DEBUG::2013-11-20 02:40:00,864::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'exclusive' >Thread-15912638::DEBUG::2013-11-20 02:40:00,866::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`c72e1389-8910-4ceb-9eaa-b8511b783f6d`::Granted request >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,867::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912638::DEBUG::2013-11-20 02:40:00,868::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`c9d5d78e-7c56-4c4b-a669-67afb34711df`::Request was made in '/usr/share/vdsm/storage/spm.py' line '377' at '_upgradePoolDomain' >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,869::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912638::DEBUG::2013-11-20 02:40:00,875::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' for lock type 'exclusive' >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,876::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912639::DEBUG::2013-11-20 02:40:00,877::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'exclusive' (1 active user) >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:00,878::blockSD::791::Storage.Misc.excCmd::(mountMaster) '/usr/bin/sudo -n /sbin/e2fsck -p /dev/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15912639::DEBUG::2013-11-20 02:40:00,879::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`daa5fb58-109f-4f40-b8a9-d4afee706da5`::Granted request >Thread-15912638::DEBUG::2013-11-20 02:40:00,880::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' is free. Now locking as 'exclusive' (1 active user) >Thread-15912638::DEBUG::2013-11-20 02:40:01,124::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6`ReqID=`c9d5d78e-7c56-4c4b-a669-67afb34711df`::Granted request >Thread-15912639::DEBUG::2013-11-20 02:40:01,127::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`4761fdaa-e6b3-4f32-88e9-d00933ea94e4`::Request was made in '/usr/share/vdsm/storage/spm.py' line '377' at '_upgradePoolDomain' >Thread-15912639::DEBUG::2013-11-20 02:40:01,129::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'exclusive' >Thread-15912638::DEBUG::2013-11-20 02:40:01,131::spm::385::Storage.SPM::(_upgradePoolDomain) Domain `45cdff8b-89c0-4926-a8c7-80354064bfa6` is not a data domain it is an 2 domain, not upgrading >Thread-15912639::DEBUG::2013-11-20 02:40:01,132::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'exclusive' (1 active user) >Thread-15912638::DEBUG::2013-11-20 02:40:01,133::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' >Thread-15912639::DEBUG::2013-11-20 02:40:01,133::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`4761fdaa-e6b3-4f32-88e9-d00933ea94e4`::Granted request >Thread-15912638::DEBUG::2013-11-20 02:40:01,135::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' (0 active users) >Thread-15912639::DEBUG::2013-11-20 02:40:01,137::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15912638::DEBUG::2013-11-20 02:40:01,137::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6' is free, finding out if anyone is waiting for it. >Thread-15912639::DEBUG::2013-11-20 02:40:01,138::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15912638::DEBUG::2013-11-20 02:40:01,139::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.45cdff8b-89c0-4926-a8c7-80354064bfa6', Clearing records. >Thread-15912639::DEBUG::2013-11-20 02:40:01,140::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15912639::DEBUG::2013-11-20 02:40:01,142::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15912638::DEBUG::2013-11-20 02:40:01,143::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' >Thread-15912639::DEBUG::2013-11-20 02:40:01,143::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15912638::DEBUG::2013-11-20 02:40:01,144::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' (0 active users) >Thread-15912639::DEBUG::2013-11-20 02:40:01,146::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15912638::DEBUG::2013-11-20 02:40:01,148::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6' is free, finding out if anyone is waiting for it. >Thread-15912638::DEBUG::2013-11-20 02:40:01,605::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_45cdff8b-89c0-4926-a8c7-80354064bfa6', Clearing records. >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:01,633::blockSD::791::Storage.Misc.excCmd::(mountMaster) SUCCESS: <err> = ''; <rc> = 0 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:01,634::blockSD::816::Storage.Misc.excCmd::(mountMaster) '/usr/bin/sudo -n /sbin/tune2fs -j /dev/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15912637::DEBUG::2013-11-20 02:40:01,650::fileUtils::83::Storage.Misc.excCmd::(mount) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912641::DEBUG::2013-11-20 02:40:01,975::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912641::INFO::2013-11-20 02:40:01,984::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912641::DEBUG::2013-11-20 02:40:01,985::task::495::TaskManager.Task::(_debug) Task f01de43b-cccb-48e1-8bb4-15802fa89a97: moving from state init -> state preparing >Thread-15912637::DEBUG::2013-11-20 02:40:01,985::fileUtils::112::Storage.Misc.excCmd::(umount) '/usr/bin/sudo -n /bin/umount -t nfs -f /tmp/tmpgimtBa' (cwd None) >Thread-15912641::DEBUG::2013-11-20 02:40:01,986::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912641::DEBUG::2013-11-20 02:40:01,987::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912641::DEBUG::2013-11-20 02:40:01,992::task::495::TaskManager.Task::(_debug) Task f01de43b-cccb-48e1-8bb4-15802fa89a97: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912641::DEBUG::2013-11-20 02:40:01,993::task::495::TaskManager.Task::(_debug) Task f01de43b-cccb-48e1-8bb4-15802fa89a97: moving from state preparing -> state finished >Thread-15912641::DEBUG::2013-11-20 02:40:01,994::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912641::DEBUG::2013-11-20 02:40:01,994::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912641::DEBUG::2013-11-20 02:40:01,995::task::495::TaskManager.Task::(_debug) Task f01de43b-cccb-48e1-8bb4-15802fa89a97: ref 0 aborting False >Thread-15912641::INFO::2013-11-20 02:40:01,995::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-42840::DEBUG::2013-11-20 02:40:02,122::lvm::439::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-42840::DEBUG::2013-11-20 02:40:02,124::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15912639::DEBUG::2013-11-20 02:40:02,510::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:02,513::blockSD::816::Storage.Misc.excCmd::(mountMaster) FAILED: <err> = 'The filesystem already has a journal.\n'; <rc> = 1 >Thread-15912639::DEBUG::2013-11-20 02:40:02,516::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'SDUUID=772472cb-2429-466d-96cd-00b3301ef78a', u'VERSION=2', u'LOCKPOLICY=', u'VGUUID=XTb294-NYv6-dVpO-bB1n-4f9P-d0GL-dhd2DI', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'LEASETIMESEC=5', u'LOCKRENEWALINTERVALSEC=5', u'ROLE=Regular', u'PHYBLKSIZE=512', u'DESCRIPTION=Database', u'IOOPTIMEOUTSEC=1', u'CLASS=Data', u'TYPE=FCP', u'PV0=pv:3600c0ff0001520c7414f605001000000,uuid:NujM33-vNiD-sBHY-mrPo-EQ2f-1qsz-HXfn0q,pestart:0,pecount:8929,mapoffset:0', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'MASTER_VERSION=0', u'_SHA_CKSUM=ef8840d9fdef24516a56fd970bed1960d2b1ae92'] >Thread-15912639::DEBUG::2013-11-20 02:40:02,519::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:02,522::fileUtils::83::Storage.Misc.excCmd::(mount) '/usr/bin/sudo -n /bin/mount /dev/6b892067-ac7e-4039-a7be-5b7387525b16/master /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >Thread-15912639::DEBUG::2013-11-20 02:40:02,523::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15912639::DEBUG::2013-11-20 02:40:02,524::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15912637::DEBUG::2013-11-20 02:40:02,916::fileUtils::112::Storage.Misc.excCmd::(umount) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912639::DEBUG::2013-11-20 02:40:03,007::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15912637::DEBUG::2013-11-20 02:40:03,008::task::495::TaskManager.Task::(_debug) Task c068ce1b-3a5e-4369-ab64-0cd423cbc79f: finished: {'statuslist': [{'status': 0, 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36'}]} >Thread-15912637::DEBUG::2013-11-20 02:40:03,014::task::495::TaskManager.Task::(_debug) Task c068ce1b-3a5e-4369-ab64-0cd423cbc79f: moving from state preparing -> state finished >Thread-15912639::DEBUG::2013-11-20 02:40:03,016::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15912637::DEBUG::2013-11-20 02:40:03,017::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912637::DEBUG::2013-11-20 02:40:03,018::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912639::DEBUG::2013-11-20 02:40:03,019::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15912637::DEBUG::2013-11-20 02:40:03,020::task::495::TaskManager.Task::(_debug) Task c068ce1b-3a5e-4369-ab64-0cd423cbc79f: ref 0 aborting False >Thread-15912639::DEBUG::2013-11-20 02:40:03,020::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15912637::INFO::2013-11-20 02:40:03,021::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: validateStorageServerConnection, Return response: {'status': {'message': 'OK', 'code': 0}, 'statuslist': [{'status': 0, 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36'}]} >Thread-15912639::DEBUG::2013-11-20 02:40:03,022::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15912643::DEBUG::2013-11-20 02:40:03,182::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912643::INFO::2013-11-20 02:40:03,184::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912643::DEBUG::2013-11-20 02:40:03,184::task::495::TaskManager.Task::(_debug) Task cc03b74d-37b5-4411-9ca5-70ef5db383d2: moving from state init -> state preparing >Thread-15912643::DEBUG::2013-11-20 02:40:03,185::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912643::DEBUG::2013-11-20 02:40:03,185::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912643::DEBUG::2013-11-20 02:40:03,186::task::495::TaskManager.Task::(_debug) Task cc03b74d-37b5-4411-9ca5-70ef5db383d2: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912643::DEBUG::2013-11-20 02:40:03,186::task::495::TaskManager.Task::(_debug) Task cc03b74d-37b5-4411-9ca5-70ef5db383d2: moving from state preparing -> state finished >Thread-15912643::DEBUG::2013-11-20 02:40:03,187::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912643::DEBUG::2013-11-20 02:40:03,187::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912643::DEBUG::2013-11-20 02:40:03,188::task::495::TaskManager.Task::(_debug) Task cc03b74d-37b5-4411-9ca5-70ef5db383d2: ref 0 aborting False >Thread-15912643::INFO::2013-11-20 02:40:03,189::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912644::DEBUG::2013-11-20 02:40:03,194::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912644::INFO::2013-11-20 02:40:03,195::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: connectStorageServer, args: (domType=1, spUUID=00000000-0000-0000-0000-000000000000, conList=[{'connection': 'kairo.brd-klee.dk:/export', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36', 'port': ''}]) >Thread-15912644::DEBUG::2013-11-20 02:40:03,196::task::495::TaskManager.Task::(_debug) Task 40fc9d51-2146-46c1-b5ac-38c6a7012dd5: moving from state init -> state preparing >Thread-15912644::INFO::2013-11-20 02:40:03,197::storage_connection::95::Storage.ServerConnection::(connect) Request to connect NFS storage server >Thread-15912644::INFO::2013-11-20 02:40:03,197::storage_connection::53::Storage.ServerConnection::(__validateConnectionParams) conList=[{'connection': 'kairo.brd-klee.dk:/export', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36', 'port': ''}] >Thread-15912644::DEBUG::2013-11-20 02:40:03,210::lvm::547::OperationMutex::(_invalidateAllPvs) Operation 'lvm reload operation' is holding the operation mutex, waiting... >Thread-42840::DEBUG::2013-11-20 02:40:03,237::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-42840::DEBUG::2013-11-20 02:40:03,240::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15912644::DEBUG::2013-11-20 02:40:03,242::lvm::547::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15912644::DEBUG::2013-11-20 02:40:03,243::lvm::549::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15912644::DEBUG::2013-11-20 02:40:03,244::lvm::559::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15912644::DEBUG::2013-11-20 02:40:03,244::lvm::561::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15912644::DEBUG::2013-11-20 02:40:03,245::lvm::580::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15912644::DEBUG::2013-11-20 02:40:03,246::lvm::582::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15912644::DEBUG::2013-11-20 02:40:03,246::task::495::TaskManager.Task::(_debug) Task 40fc9d51-2146-46c1-b5ac-38c6a7012dd5: finished: {'statuslist': [{'status': 0, 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36'}]} >Thread-15912644::DEBUG::2013-11-20 02:40:03,247::task::495::TaskManager.Task::(_debug) Task 40fc9d51-2146-46c1-b5ac-38c6a7012dd5: moving from state preparing -> state finished >Thread-15912644::DEBUG::2013-11-20 02:40:03,247::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912644::DEBUG::2013-11-20 02:40:03,248::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912644::DEBUG::2013-11-20 02:40:03,248::task::495::TaskManager.Task::(_debug) Task 40fc9d51-2146-46c1-b5ac-38c6a7012dd5: ref 0 aborting False >Thread-15912644::INFO::2013-11-20 02:40:03,249::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: connectStorageServer, Return response: {'status': {'message': 'OK', 'code': 0}, 'statuslist': [{'status': 0, 'id': 'ec9380b7-cfc1-41d8-9237-d5c7c505ef36'}]} >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:03,268::fileUtils::83::Storage.Misc.excCmd::(mount) SUCCESS: <err> = ''; <rc> = 0 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:03,269::blockSD::833::Storage.Misc.excCmd::(mountMaster) '/usr/bin/sudo -n /bin/chown vdsm:kvm /rhev/data-center/mnt/blockSD/6b892067-ac7e-4039-a7be-5b7387525b16/master' (cwd None) >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:03,688::blockSD::833::Storage.Misc.excCmd::(mountMaster) SUCCESS: <err> = ''; <rc> = 0 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::WARNING::2013-11-20 02:40:03,699::spm::490::Storage.SPM::(start) Backup domain validation failed >Traceback (most recent call last): > File "/usr/share/vdsm/storage/spm.py", line 488, in start > File "/usr/share/vdsm/storage/sp.py", line 1208, in checkBackupDomain > File "/usr/share/vdsm/storage/sd.py", line 666, in isBackup > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:03,700::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:03,701::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:03,702::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:03,702::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:03,703::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:03,703::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:03,704::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:03,704::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:03,705::persistentDict::140::Storage.PersistentDict::(transaction) Starting transaction >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:03,705::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:03,706::persistentDict::133::Storage.PersistentDict::(transaction) Reusing active transaction >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:03,706::persistentDict::146::Storage.PersistentDict::(transaction) Flushing changes >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:03,707::persistentDict::265::Storage.PersistentDict::(flush) about to write lines (FileMetadataRW)=['MASTER_VERSION=0', 'ROLE=Regular', '_SHA_CKSUM=1deb73127f89496e2cb17e51bbfae1df6c1879d1'] >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:03,940::persistentDict::148::Storage.PersistentDict::(transaction) Finished transaction >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:03,943::threadPool::25::Misc.ThreadPool::(__init__) Enter - numThreads: 5.0, waitTimeout: 3, maxTasks: 500.0 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:03,948::storage_mailbox::486::Storage.MailBox.SpmMailMonitor::(__init__) SPM_MailMonitor - clearing outgoing mail, command is: ['dd', 'of=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/outbox', 'oflag=direct', 'iflag=fullblock', 'conv=notrunc', 'count=1'] >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:03,949::storage_mailbox::488::Storage.Misc.excCmd::(__init__) 'dd of=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/outbox oflag=direct iflag=fullblock conv=notrunc count=1 bs=1024000' (cwd None) >Thread-42838::DEBUG::2013-11-20 02:40:04,154::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-42838::DEBUG::2013-11-20 02:40:04,161::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15912650::DEBUG::2013-11-20 02:40:04,478::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912650::INFO::2013-11-20 02:40:04,485::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912650::DEBUG::2013-11-20 02:40:04,486::task::495::TaskManager.Task::(_debug) Task c1fc4616-b3bb-472d-aa21-d7386e34d9d6: moving from state init -> state preparing >Thread-15912650::DEBUG::2013-11-20 02:40:04,488::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912650::DEBUG::2013-11-20 02:40:04,489::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912650::DEBUG::2013-11-20 02:40:04,490::task::495::TaskManager.Task::(_debug) Task c1fc4616-b3bb-472d-aa21-d7386e34d9d6: finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912650::DEBUG::2013-11-20 02:40:04,492::task::495::TaskManager.Task::(_debug) Task c1fc4616-b3bb-472d-aa21-d7386e34d9d6: moving from state preparing -> state finished >Thread-15912650::DEBUG::2013-11-20 02:40:04,494::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912650::DEBUG::2013-11-20 02:40:04,495::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912650::DEBUG::2013-11-20 02:40:04,496::task::495::TaskManager.Task::(_debug) Task c1fc4616-b3bb-472d-aa21-d7386e34d9d6: ref 0 aborting False >Thread-15912650::INFO::2013-11-20 02:40:04,498::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:04,538::storage_mailbox::488::Storage.Misc.excCmd::(__init__) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0572943 s, 17.9 MB/s\n'; <rc> = 0 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:04,539::storage_mailbox::493::Storage.MailBox.SpmMailMonitor::(__init__) SPM_MailMonitor created for pool c4e5889a-348d-11e2-83fe-9f72369eab30 >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:04,541::spm::523::Storage.SPM::(start) ended. >Dummy-15912651::DEBUG::2013-11-20 02:40:04,540::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:04,542::task::495::TaskManager.Task::(_debug) Task 59c70706-854b-4ae3-91cc-4a2cfb6f46d4: moving from state running -> state finished >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:04,809::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:04,817::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:04,819::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:04,820::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:04,820::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:04,821::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:04,822::task::915::TaskManager.Task::(_runJobs) Task.run: exit - success: result >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:04,822::task::495::TaskManager.Task::(_debug) Task 59c70706-854b-4ae3-91cc-4a2cfb6f46d4: ref 0 aborting False >59c70706-854b-4ae3-91cc-4a2cfb6f46d4::DEBUG::2013-11-20 02:40:04,823::threadPool::47::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0 >Thread-42838::DEBUG::2013-11-20 02:40:04,959::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-42838::DEBUG::2013-11-20 02:40:04,963::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Dummy-15912651::DEBUG::2013-11-20 02:40:05,173::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.131043 s, 7.8 MB/s\n'; <rc> = 0 >Thread-15912652::DEBUG::2013-11-20 02:40:05,517::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912652::INFO::2013-11-20 02:40:05,519::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912652::DEBUG::2013-11-20 02:40:05,520::task::495::TaskManager.Task::(_debug) Task 11533244-3712-4617-a3fb-9a81b8711f90: moving from state init -> state preparing >Thread-15912652::DEBUG::2013-11-20 02:40:05,520::taskManager::82::TaskManager::(getTaskStatus) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912652::DEBUG::2013-11-20 02:40:05,521::taskManager::85::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': '1 jobs completed successfuly', 'taskState': 'finished', 'taskResult': 'success', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'} >Thread-15912652::DEBUG::2013-11-20 02:40:05,522::task::495::TaskManager.Task::(_debug) Task 11533244-3712-4617-a3fb-9a81b8711f90: finished: {'taskStatus': {'code': 0, 'message': '1 jobs completed successfuly', 'taskState': 'finished', 'taskResult': 'success', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912652::DEBUG::2013-11-20 02:40:05,522::task::495::TaskManager.Task::(_debug) Task 11533244-3712-4617-a3fb-9a81b8711f90: moving from state preparing -> state finished >Thread-15912652::DEBUG::2013-11-20 02:40:05,523::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912652::DEBUG::2013-11-20 02:40:05,523::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912652::DEBUG::2013-11-20 02:40:05,524::task::495::TaskManager.Task::(_debug) Task 11533244-3712-4617-a3fb-9a81b8711f90: ref 0 aborting False >Thread-15912652::INFO::2013-11-20 02:40:05,525::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getTaskStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'taskStatus': {'code': 0, 'message': '1 jobs completed successfuly', 'taskState': 'finished', 'taskResult': 'success', 'taskID': '59c70706-854b-4ae3-91cc-4a2cfb6f46d4'}} >Thread-15912653::DEBUG::2013-11-20 02:40:05,531::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912653::INFO::2013-11-20 02:40:05,532::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912653::DEBUG::2013-11-20 02:40:05,533::task::495::TaskManager.Task::(_debug) Task 5aaf0398-bb64-485c-9209-2713b6f1c57d: moving from state init -> state preparing >Thread-15912653::DEBUG::2013-11-20 02:40:05,534::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912653::DEBUG::2013-11-20 02:40:05,534::task::495::TaskManager.Task::(_debug) Task 5aaf0398-bb64-485c-9209-2713b6f1c57d: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912653::DEBUG::2013-11-20 02:40:05,535::task::495::TaskManager.Task::(_debug) Task 5aaf0398-bb64-485c-9209-2713b6f1c57d: moving from state preparing -> state finished >Thread-15912653::DEBUG::2013-11-20 02:40:05,536::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912653::DEBUG::2013-11-20 02:40:05,536::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912653::DEBUG::2013-11-20 02:40:05,537::task::495::TaskManager.Task::(_debug) Task 5aaf0398-bb64-485c-9209-2713b6f1c57d: ref 0 aborting False >Thread-15912653::INFO::2013-11-20 02:40:05,537::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912654::DEBUG::2013-11-20 02:40:05,677::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912654::INFO::2013-11-20 02:40:05,678::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: clearTask, args: ( taskID=59c70706-854b-4ae3-91cc-4a2cfb6f46d4) >Thread-15912654::DEBUG::2013-11-20 02:40:05,678::task::495::TaskManager.Task::(_debug) Task 7cce22c7-921e-46c9-b0cd-5db46111da1b: moving from state init -> state preparing >Thread-15912654::DEBUG::2013-11-20 02:40:05,679::taskManager::146::TaskManager::(clearTask) Entry. taskID: 59c70706-854b-4ae3-91cc-4a2cfb6f46d4 >Thread-15912654::DEBUG::2013-11-20 02:40:05,680::taskManager::151::TaskManager::(clearTask) Return. >Thread-15912654::DEBUG::2013-11-20 02:40:05,680::task::495::TaskManager.Task::(_debug) Task 7cce22c7-921e-46c9-b0cd-5db46111da1b: finished: None >Thread-15912654::DEBUG::2013-11-20 02:40:05,681::task::495::TaskManager.Task::(_debug) Task 7cce22c7-921e-46c9-b0cd-5db46111da1b: moving from state preparing -> state finished >Thread-15912654::DEBUG::2013-11-20 02:40:05,681::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912654::DEBUG::2013-11-20 02:40:05,682::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912654::DEBUG::2013-11-20 02:40:05,682::task::495::TaskManager.Task::(_debug) Task 7cce22c7-921e-46c9-b0cd-5db46111da1b: ref 0 aborting False >Thread-15912654::INFO::2013-11-20 02:40:05,683::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: clearTask, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15912655::DEBUG::2013-11-20 02:40:06,164::misc::1156::Event.StatsThread.onDomainConnectivityStateChange::(_emit) Emitting event >Thread-15912655::DEBUG::2013-11-20 02:40:06,165::misc::1166::Event.StatsThread.onDomainConnectivityStateChange::(_emit) Calling registered method `run` >Thread-15912656::DEBUG::2013-11-20 02:40:06,167::spm::365::Storage.SPM::(_upgradePoolDomain) Preparing to upgrade domain 5d7b5178-3790-472c-a13b-b481eb27a61e >Thread-15912655::DEBUG::2013-11-20 02:40:06,174::misc::1174::Event.StatsThread.onDomainConnectivityStateChange::(_emit) Event emitted >Thread-15912656::ERROR::2013-11-20 02:40:06,176::spm::373::Storage.SPM::(_upgradePoolDomain) Error while preparing domain `5d7b5178-3790-472c-a13b-b481eb27a61e` upgrade >Traceback (most recent call last): > File "/usr/share/vdsm/storage/spm.py", line 369, in _upgradePoolDomain >IndexError: list index out of range >Dummy-15912651::DEBUG::2013-11-20 02:40:07,187::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:40:07,719::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0788495 s, 13.0 MB/s\n'; <rc> = 0 >Thread-15912658::DEBUG::2013-11-20 02:40:07,870::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912658::INFO::2013-11-20 02:40:07,872::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912658::DEBUG::2013-11-20 02:40:07,873::task::495::TaskManager.Task::(_debug) Task 59c8e6a2-993f-43a1-aff9-573924f88106: moving from state init -> state preparing >Thread-15912658::DEBUG::2013-11-20 02:40:07,874::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`62d951f8-11c0-438a-99aa-63dc8d4b9086`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912658::DEBUG::2013-11-20 02:40:07,881::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912658::DEBUG::2013-11-20 02:40:07,882::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912658::DEBUG::2013-11-20 02:40:07,893::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`62d951f8-11c0-438a-99aa-63dc8d4b9086`::Granted request >Thread-15912658::DEBUG::2013-11-20 02:40:07,894::task::495::TaskManager.Task::(_debug) Task 59c8e6a2-993f-43a1-aff9-573924f88106: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912658::DEBUG::2013-11-20 02:40:07,894::task::495::TaskManager.Task::(_debug) Task 59c8e6a2-993f-43a1-aff9-573924f88106: ref 1 aborting False >Thread-15912658::WARNING::2013-11-20 02:40:07,899::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912658::DEBUG::2013-11-20 02:40:07,900::task::495::TaskManager.Task::(_debug) Task 59c8e6a2-993f-43a1-aff9-573924f88106: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912658::DEBUG::2013-11-20 02:40:07,903::task::495::TaskManager.Task::(_debug) Task 59c8e6a2-993f-43a1-aff9-573924f88106: moving from state preparing -> state finished >Thread-15912658::DEBUG::2013-11-20 02:40:07,904::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912658::DEBUG::2013-11-20 02:40:07,904::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912658::DEBUG::2013-11-20 02:40:07,911::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912658::DEBUG::2013-11-20 02:40:07,912::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912658::DEBUG::2013-11-20 02:40:07,912::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912658::DEBUG::2013-11-20 02:40:07,919::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912658::DEBUG::2013-11-20 02:40:07,923::task::495::TaskManager.Task::(_debug) Task 59c8e6a2-993f-43a1-aff9-573924f88106: ref 0 aborting False >Thread-15912658::INFO::2013-11-20 02:40:07,924::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912659::DEBUG::2013-11-20 02:40:08,412::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912659::INFO::2013-11-20 02:40:08,413::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksInfo, args: () >Thread-15912659::DEBUG::2013-11-20 02:40:08,414::task::495::TaskManager.Task::(_debug) Task e5cdcaca-dfa9-4a97-9e11-c5a14019288f: moving from state init -> state preparing >Thread-15912659::DEBUG::2013-11-20 02:40:08,415::taskManager::168::TaskManager::(getAllTasksInfo) Entry. >Thread-15912659::DEBUG::2013-11-20 02:40:08,416::taskManager::173::TaskManager::(getAllTasksInfo) Return. Response: {} >Thread-15912659::DEBUG::2013-11-20 02:40:08,417::task::495::TaskManager.Task::(_debug) Task e5cdcaca-dfa9-4a97-9e11-c5a14019288f: finished: {'allTasksInfo': {}} >Thread-15912659::DEBUG::2013-11-20 02:40:08,417::task::495::TaskManager.Task::(_debug) Task e5cdcaca-dfa9-4a97-9e11-c5a14019288f: moving from state preparing -> state finished >Thread-15912659::DEBUG::2013-11-20 02:40:08,418::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912659::DEBUG::2013-11-20 02:40:08,419::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912659::DEBUG::2013-11-20 02:40:08,420::task::495::TaskManager.Task::(_debug) Task e5cdcaca-dfa9-4a97-9e11-c5a14019288f: ref 0 aborting False >Thread-15912659::INFO::2013-11-20 02:40:08,424::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getAllTasksInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'allTasksInfo': {}} >Dummy-15912651::DEBUG::2013-11-20 02:40:09,735::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:40:10,261::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0727364 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:40:12,276::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:40:12,827::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0726634 s, 14.1 MB/s\n'; <rc> = 0 >Thread-42840::DEBUG::2013-11-20 02:40:13,501::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-42840::DEBUG::2013-11-20 02:40:13,503::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 02:40:14,185::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-42840::DEBUG::2013-11-20 02:40:14,188::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Dummy-15912651::DEBUG::2013-11-20 02:40:14,843::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:40:15,372::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0755846 s, 13.5 MB/s\n'; <rc> = 0 >Thread-15912662::INFO::2013-11-20 02:40:16,037::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912662::DEBUG::2013-11-20 02:40:16,038::task::495::TaskManager.Task::(_debug) Task dad08f6c-2a15-4dd8-84dc-12f3ceba85ab: moving from state init -> state preparing >Thread-15912662::DEBUG::2013-11-20 02:40:16,039::task::495::TaskManager.Task::(_debug) Task dad08f6c-2a15-4dd8-84dc-12f3ceba85ab: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000531911849976', 'lastCheck': 1384915215.063504, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00179600715637', 'lastCheck': 1384915215.1816781, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00251483917236', 'lastCheck': 1384915206.1634541, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000345230102539', 'lastCheck': 1384915214.1902399, 'valid': True, 'code': 0}} >Thread-15912662::DEBUG::2013-11-20 02:40:16,040::task::495::TaskManager.Task::(_debug) Task dad08f6c-2a15-4dd8-84dc-12f3ceba85ab: moving from state preparing -> state finished >Thread-15912662::DEBUG::2013-11-20 02:40:16,040::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912662::DEBUG::2013-11-20 02:40:16,041::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912662::DEBUG::2013-11-20 02:40:16,041::task::495::TaskManager.Task::(_debug) Task dad08f6c-2a15-4dd8-84dc-12f3ceba85ab: ref 0 aborting False >Thread-15912662::INFO::2013-11-20 02:40:16,042::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000531911849976', 'lastCheck': 1384915215.063504, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00179600715637', 'lastCheck': 1384915215.1816781, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00251483917236', 'lastCheck': 1384915206.1634541, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000345230102539', 'lastCheck': 1384915214.1902399, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:40:17,405::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:40:17,962::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.07667 s, 13.4 MB/s\n'; <rc> = 0 >Thread-15912664::DEBUG::2013-11-20 02:40:18,399::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912664::INFO::2013-11-20 02:40:18,400::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912664::DEBUG::2013-11-20 02:40:18,401::task::495::TaskManager.Task::(_debug) Task 41e28ba9-ef74-4011-80d8-cd1536382651: moving from state init -> state preparing >Thread-15912664::DEBUG::2013-11-20 02:40:18,402::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912664::DEBUG::2013-11-20 02:40:18,402::task::495::TaskManager.Task::(_debug) Task 41e28ba9-ef74-4011-80d8-cd1536382651: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912664::DEBUG::2013-11-20 02:40:18,403::task::495::TaskManager.Task::(_debug) Task 41e28ba9-ef74-4011-80d8-cd1536382651: moving from state preparing -> state finished >Thread-15912664::DEBUG::2013-11-20 02:40:18,403::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912664::DEBUG::2013-11-20 02:40:18,404::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912664::DEBUG::2013-11-20 02:40:18,405::task::495::TaskManager.Task::(_debug) Task 41e28ba9-ef74-4011-80d8-cd1536382651: ref 0 aborting False >Thread-15912664::INFO::2013-11-20 02:40:18,411::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912665::DEBUG::2013-11-20 02:40:18,422::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912665::INFO::2013-11-20 02:40:18,426::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912665::DEBUG::2013-11-20 02:40:18,427::task::495::TaskManager.Task::(_debug) Task 86fe80bb-891e-44a0-922c-e09d50d6ab26: moving from state init -> state preparing >Thread-15912665::DEBUG::2013-11-20 02:40:18,428::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`06dd6e41-2e74-41a8-b64a-ee4331c9ee04`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912665::DEBUG::2013-11-20 02:40:18,434::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912665::DEBUG::2013-11-20 02:40:18,435::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912665::DEBUG::2013-11-20 02:40:18,436::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`06dd6e41-2e74-41a8-b64a-ee4331c9ee04`::Granted request >Thread-15912665::DEBUG::2013-11-20 02:40:18,437::task::495::TaskManager.Task::(_debug) Task 86fe80bb-891e-44a0-922c-e09d50d6ab26: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912665::DEBUG::2013-11-20 02:40:18,438::task::495::TaskManager.Task::(_debug) Task 86fe80bb-891e-44a0-922c-e09d50d6ab26: ref 1 aborting False >Thread-15912665::WARNING::2013-11-20 02:40:18,440::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912665::DEBUG::2013-11-20 02:40:18,441::task::495::TaskManager.Task::(_debug) Task 86fe80bb-891e-44a0-922c-e09d50d6ab26: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912665::DEBUG::2013-11-20 02:40:18,446::task::495::TaskManager.Task::(_debug) Task 86fe80bb-891e-44a0-922c-e09d50d6ab26: moving from state preparing -> state finished >Thread-15912665::DEBUG::2013-11-20 02:40:18,446::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912665::DEBUG::2013-11-20 02:40:18,447::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912665::DEBUG::2013-11-20 02:40:18,447::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912665::DEBUG::2013-11-20 02:40:18,448::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912665::DEBUG::2013-11-20 02:40:18,449::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912665::DEBUG::2013-11-20 02:40:18,449::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912665::DEBUG::2013-11-20 02:40:18,450::task::495::TaskManager.Task::(_debug) Task 86fe80bb-891e-44a0-922c-e09d50d6ab26: ref 0 aborting False >Thread-15912665::INFO::2013-11-20 02:40:18,450::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:40:19,979::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:40:20,532::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0744498 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:40:22,547::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:40:23,137::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0745944 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:40:25,150::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:40:25,721::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0797565 s, 12.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:40:27,735::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:40:28,189::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0447813 s, 22.9 MB/s\n'; <rc> = 0 >Thread-15912670::DEBUG::2013-11-20 02:40:29,254::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912670::INFO::2013-11-20 02:40:29,267::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912670::DEBUG::2013-11-20 02:40:29,273::task::495::TaskManager.Task::(_debug) Task 56e36ab3-fd13-48ba-8c16-fa595e25245c: moving from state init -> state preparing >Thread-15912670::DEBUG::2013-11-20 02:40:29,276::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912670::DEBUG::2013-11-20 02:40:29,276::task::495::TaskManager.Task::(_debug) Task 56e36ab3-fd13-48ba-8c16-fa595e25245c: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912670::DEBUG::2013-11-20 02:40:29,282::task::495::TaskManager.Task::(_debug) Task 56e36ab3-fd13-48ba-8c16-fa595e25245c: moving from state preparing -> state finished >Thread-15912670::DEBUG::2013-11-20 02:40:29,282::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912670::DEBUG::2013-11-20 02:40:29,283::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912670::DEBUG::2013-11-20 02:40:29,283::task::495::TaskManager.Task::(_debug) Task 56e36ab3-fd13-48ba-8c16-fa595e25245c: ref 0 aborting False >Thread-15912670::INFO::2013-11-20 02:40:29,284::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912671::DEBUG::2013-11-20 02:40:29,297::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912671::INFO::2013-11-20 02:40:29,298::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912671::DEBUG::2013-11-20 02:40:29,298::task::495::TaskManager.Task::(_debug) Task 719ffbc2-9d8d-42ce-998f-06fcb4936794: moving from state init -> state preparing >Thread-15912671::DEBUG::2013-11-20 02:40:29,299::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bdcf08cc-f57b-4be6-a9ad-2aa47ef0a5d7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912671::DEBUG::2013-11-20 02:40:29,300::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912671::DEBUG::2013-11-20 02:40:29,306::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912671::DEBUG::2013-11-20 02:40:29,307::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bdcf08cc-f57b-4be6-a9ad-2aa47ef0a5d7`::Granted request >Thread-15912671::DEBUG::2013-11-20 02:40:29,308::task::495::TaskManager.Task::(_debug) Task 719ffbc2-9d8d-42ce-998f-06fcb4936794: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912671::DEBUG::2013-11-20 02:40:29,308::task::495::TaskManager.Task::(_debug) Task 719ffbc2-9d8d-42ce-998f-06fcb4936794: ref 1 aborting False >Thread-15912671::WARNING::2013-11-20 02:40:29,310::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912671::DEBUG::2013-11-20 02:40:29,314::task::495::TaskManager.Task::(_debug) Task 719ffbc2-9d8d-42ce-998f-06fcb4936794: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912671::DEBUG::2013-11-20 02:40:29,315::task::495::TaskManager.Task::(_debug) Task 719ffbc2-9d8d-42ce-998f-06fcb4936794: moving from state preparing -> state finished >Thread-15912671::DEBUG::2013-11-20 02:40:29,316::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912671::DEBUG::2013-11-20 02:40:29,316::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912671::DEBUG::2013-11-20 02:40:29,317::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912671::DEBUG::2013-11-20 02:40:29,317::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912671::DEBUG::2013-11-20 02:40:29,318::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912671::DEBUG::2013-11-20 02:40:29,318::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912671::DEBUG::2013-11-20 02:40:29,319::task::495::TaskManager.Task::(_debug) Task 719ffbc2-9d8d-42ce-998f-06fcb4936794: ref 0 aborting False >Thread-15912671::INFO::2013-11-20 02:40:29,320::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:40:30,199::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:40:30,702::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0758841 s, 13.5 MB/s\n'; <rc> = 0 >Thread-15912672::INFO::2013-11-20 02:40:32,025::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912672::DEBUG::2013-11-20 02:40:32,026::task::495::TaskManager.Task::(_debug) Task defccff8-9554-40a7-84cb-36430f7de42c: moving from state init -> state preparing >Thread-15912672::DEBUG::2013-11-20 02:40:32,027::task::495::TaskManager.Task::(_debug) Task defccff8-9554-40a7-84cb-36430f7de42c: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000823020935059', 'lastCheck': 1384915225.39729, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0166461467743', 'lastCheck': 1384915225.413501, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00139307975769', 'lastCheck': 1384915226.3934691, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000494003295898', 'lastCheck': 1384915224.209826, 'valid': True, 'code': 0}} >Thread-15912672::DEBUG::2013-11-20 02:40:32,028::task::495::TaskManager.Task::(_debug) Task defccff8-9554-40a7-84cb-36430f7de42c: moving from state preparing -> state finished >Thread-15912672::DEBUG::2013-11-20 02:40:32,028::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912672::DEBUG::2013-11-20 02:40:32,029::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912672::DEBUG::2013-11-20 02:40:32,029::task::495::TaskManager.Task::(_debug) Task defccff8-9554-40a7-84cb-36430f7de42c: ref 0 aborting False >Thread-15912672::INFO::2013-11-20 02:40:32,030::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000823020935059', 'lastCheck': 1384915225.39729, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0166461467743', 'lastCheck': 1384915225.413501, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00139307975769', 'lastCheck': 1384915226.3934691, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000494003295898', 'lastCheck': 1384915224.209826, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:40:32,718::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:40:33,246::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0793916 s, 12.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:40:35,262::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:40:35,821::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0712237 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:40:37,836::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:40:38,337::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0617664 s, 16.6 MB/s\n'; <rc> = 0 >Thread-15912676::DEBUG::2013-11-20 02:40:39,779::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912676::INFO::2013-11-20 02:40:39,783::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912676::DEBUG::2013-11-20 02:40:39,783::task::495::TaskManager.Task::(_debug) Task 0aa12d4c-faa2-4b28-9fee-ae6bf1997a45: moving from state init -> state preparing >Thread-15912676::DEBUG::2013-11-20 02:40:39,784::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912676::DEBUG::2013-11-20 02:40:39,785::task::495::TaskManager.Task::(_debug) Task 0aa12d4c-faa2-4b28-9fee-ae6bf1997a45: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912676::DEBUG::2013-11-20 02:40:39,785::task::495::TaskManager.Task::(_debug) Task 0aa12d4c-faa2-4b28-9fee-ae6bf1997a45: moving from state preparing -> state finished >Thread-15912676::DEBUG::2013-11-20 02:40:39,786::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912676::DEBUG::2013-11-20 02:40:39,786::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912676::DEBUG::2013-11-20 02:40:39,787::task::495::TaskManager.Task::(_debug) Task 0aa12d4c-faa2-4b28-9fee-ae6bf1997a45: ref 0 aborting False >Thread-15912676::INFO::2013-11-20 02:40:39,787::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912677::DEBUG::2013-11-20 02:40:39,797::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912677::INFO::2013-11-20 02:40:39,798::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912677::DEBUG::2013-11-20 02:40:39,799::task::495::TaskManager.Task::(_debug) Task 38efb9a1-bf16-4373-93ea-6bcdb16b97a8: moving from state init -> state preparing >Thread-15912677::DEBUG::2013-11-20 02:40:39,800::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c5492dd3-ef3f-44de-be86-1f5d9b20ef64`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912677::DEBUG::2013-11-20 02:40:39,800::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912677::DEBUG::2013-11-20 02:40:39,801::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912677::DEBUG::2013-11-20 02:40:39,802::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c5492dd3-ef3f-44de-be86-1f5d9b20ef64`::Granted request >Thread-15912677::DEBUG::2013-11-20 02:40:39,803::task::495::TaskManager.Task::(_debug) Task 38efb9a1-bf16-4373-93ea-6bcdb16b97a8: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912677::DEBUG::2013-11-20 02:40:39,803::task::495::TaskManager.Task::(_debug) Task 38efb9a1-bf16-4373-93ea-6bcdb16b97a8: ref 1 aborting False >Thread-15912677::WARNING::2013-11-20 02:40:39,805::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912677::DEBUG::2013-11-20 02:40:39,806::task::495::TaskManager.Task::(_debug) Task 38efb9a1-bf16-4373-93ea-6bcdb16b97a8: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115766984704', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115766984704', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912677::DEBUG::2013-11-20 02:40:39,807::task::495::TaskManager.Task::(_debug) Task 38efb9a1-bf16-4373-93ea-6bcdb16b97a8: moving from state preparing -> state finished >Thread-15912677::DEBUG::2013-11-20 02:40:39,807::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912677::DEBUG::2013-11-20 02:40:39,808::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912677::DEBUG::2013-11-20 02:40:39,809::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912677::DEBUG::2013-11-20 02:40:39,809::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912677::DEBUG::2013-11-20 02:40:39,810::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912677::DEBUG::2013-11-20 02:40:39,810::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912677::DEBUG::2013-11-20 02:40:39,811::task::495::TaskManager.Task::(_debug) Task 38efb9a1-bf16-4373-93ea-6bcdb16b97a8: ref 0 aborting False >Thread-15912677::INFO::2013-11-20 02:40:39,812::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115766984704', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115766984704', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:40:40,347::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:40:40,879::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0800273 s, 12.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:40:42,893::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:40:43,456::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0761718 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:40:45,472::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:40:46,009::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0758962 s, 13.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:40:48,024::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912680::INFO::2013-11-20 02:40:48,390::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912680::DEBUG::2013-11-20 02:40:48,391::task::495::TaskManager.Task::(_debug) Task 3708e432-1ac1-4697-a40f-38a3371d39e4: moving from state init -> state preparing >Thread-15912680::DEBUG::2013-11-20 02:40:48,392::task::495::TaskManager.Task::(_debug) Task 3708e432-1ac1-4697-a40f-38a3371d39e4: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000463962554932', 'lastCheck': 1384915245.6879389, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00145602226257', 'lastCheck': 1384915245.691154, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00155782699585', 'lastCheck': 1384915246.6898489, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000530004501343', 'lastCheck': 1384915244.559463, 'valid': True, 'code': 0}} >Thread-15912680::DEBUG::2013-11-20 02:40:48,393::task::495::TaskManager.Task::(_debug) Task 3708e432-1ac1-4697-a40f-38a3371d39e4: moving from state preparing -> state finished >Thread-15912680::DEBUG::2013-11-20 02:40:48,394::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912680::DEBUG::2013-11-20 02:40:48,394::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912680::DEBUG::2013-11-20 02:40:48,395::task::495::TaskManager.Task::(_debug) Task 3708e432-1ac1-4697-a40f-38a3371d39e4: ref 0 aborting False >Thread-15912680::INFO::2013-11-20 02:40:48,396::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000463962554932', 'lastCheck': 1384915245.6879389, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00145602226257', 'lastCheck': 1384915245.691154, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00155782699585', 'lastCheck': 1384915246.6898489, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000530004501343', 'lastCheck': 1384915244.559463, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:40:48,573::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0735003 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:40:50,588::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:40:51,113::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0736667 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15912682::DEBUG::2013-11-20 02:40:51,608::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912682::INFO::2013-11-20 02:40:51,614::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912682::DEBUG::2013-11-20 02:40:51,616::task::495::TaskManager.Task::(_debug) Task 78709e12-23ec-438e-b3ac-2471ec9429d2: moving from state init -> state preparing >Thread-15912682::DEBUG::2013-11-20 02:40:51,617::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912682::DEBUG::2013-11-20 02:40:51,618::task::495::TaskManager.Task::(_debug) Task 78709e12-23ec-438e-b3ac-2471ec9429d2: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912682::DEBUG::2013-11-20 02:40:51,618::task::495::TaskManager.Task::(_debug) Task 78709e12-23ec-438e-b3ac-2471ec9429d2: moving from state preparing -> state finished >Thread-15912682::DEBUG::2013-11-20 02:40:51,619::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912682::DEBUG::2013-11-20 02:40:51,619::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912682::DEBUG::2013-11-20 02:40:51,620::task::495::TaskManager.Task::(_debug) Task 78709e12-23ec-438e-b3ac-2471ec9429d2: ref 0 aborting False >Thread-15912682::INFO::2013-11-20 02:40:51,621::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912683::DEBUG::2013-11-20 02:40:51,625::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912683::INFO::2013-11-20 02:40:51,626::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912683::DEBUG::2013-11-20 02:40:51,627::task::495::TaskManager.Task::(_debug) Task 2095afd3-b952-4581-a453-7514eaed8e40: moving from state init -> state preparing >Thread-15912683::DEBUG::2013-11-20 02:40:51,628::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`25099e1b-8049-4631-9579-3cd56598e998`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912683::DEBUG::2013-11-20 02:40:51,629::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912683::DEBUG::2013-11-20 02:40:51,629::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912683::DEBUG::2013-11-20 02:40:51,630::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`25099e1b-8049-4631-9579-3cd56598e998`::Granted request >Thread-15912683::DEBUG::2013-11-20 02:40:51,631::task::495::TaskManager.Task::(_debug) Task 2095afd3-b952-4581-a453-7514eaed8e40: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912683::DEBUG::2013-11-20 02:40:51,631::task::495::TaskManager.Task::(_debug) Task 2095afd3-b952-4581-a453-7514eaed8e40: ref 1 aborting False >Thread-15912683::WARNING::2013-11-20 02:40:51,633::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912683::DEBUG::2013-11-20 02:40:51,634::task::495::TaskManager.Task::(_debug) Task 2095afd3-b952-4581-a453-7514eaed8e40: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115766984704', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115766984704', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912683::DEBUG::2013-11-20 02:40:51,635::task::495::TaskManager.Task::(_debug) Task 2095afd3-b952-4581-a453-7514eaed8e40: moving from state preparing -> state finished >Thread-15912683::DEBUG::2013-11-20 02:40:51,636::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912683::DEBUG::2013-11-20 02:40:51,636::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912683::DEBUG::2013-11-20 02:40:51,637::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912683::DEBUG::2013-11-20 02:40:51,638::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912683::DEBUG::2013-11-20 02:40:51,638::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912683::DEBUG::2013-11-20 02:40:51,639::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912683::DEBUG::2013-11-20 02:40:51,639::task::495::TaskManager.Task::(_debug) Task 2095afd3-b952-4581-a453-7514eaed8e40: ref 0 aborting False >Thread-15912683::INFO::2013-11-20 02:40:51,643::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115766984704', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115766984704', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:40:53,128::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:40:53,671::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0719806 s, 14.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:40:55,687::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:40:56,240::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0851882 s, 12.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:40:58,265::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:40:58,820::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0758267 s, 13.5 MB/s\n'; <rc> = 0 >Thread-15912392::INFO::2013-11-20 02:40:59,374::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:40:59,375::task::495::TaskManager.Task::(_debug) Task f2e1455a-f461-49cd-84fe-46e4fc8d1a20: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:40:59,380::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e288a725-3541-4c84-ad17-941fb4da2ece`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:40:59,385::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:40:59,392::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:40:59,394::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e288a725-3541-4c84-ad17-941fb4da2ece`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:40:59,400::task::495::TaskManager.Task::(_debug) Task f2e1455a-f461-49cd-84fe-46e4fc8d1a20: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:40:59,401::task::495::TaskManager.Task::(_debug) Task f2e1455a-f461-49cd-84fe-46e4fc8d1a20: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:40:59,403::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15912392::DEBUG::2013-11-20 02:40:59,406::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15912392::DEBUG::2013-11-20 02:41:00,358::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15912392::DEBUG::2013-11-20 02:41:00,562::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15912392::DEBUG::2013-11-20 02:41:00,562::task::495::TaskManager.Task::(_debug) Task f2e1455a-f461-49cd-84fe-46e4fc8d1a20: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:41:00,563::task::495::TaskManager.Task::(_debug) Task f2e1455a-f461-49cd-84fe-46e4fc8d1a20: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:41:00,563::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:41:00,563::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:41:00,564::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:41:00,564::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:41:00,565::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:41:00,565::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:41:00,565::task::495::TaskManager.Task::(_debug) Task f2e1455a-f461-49cd-84fe-46e4fc8d1a20: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:41:00,566::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 02:41:00,835::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:41:01,333::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0709116 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:41:03,347::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:41:03,858::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0707051 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15912688::DEBUG::2013-11-20 02:41:03,924::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912688::INFO::2013-11-20 02:41:03,925::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912688::DEBUG::2013-11-20 02:41:03,926::task::495::TaskManager.Task::(_debug) Task 55d6403a-8044-4267-be3a-2f2a02699a17: moving from state init -> state preparing >Thread-15912688::DEBUG::2013-11-20 02:41:03,927::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912688::DEBUG::2013-11-20 02:41:03,927::task::495::TaskManager.Task::(_debug) Task 55d6403a-8044-4267-be3a-2f2a02699a17: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912688::DEBUG::2013-11-20 02:41:03,928::task::495::TaskManager.Task::(_debug) Task 55d6403a-8044-4267-be3a-2f2a02699a17: moving from state preparing -> state finished >Thread-15912688::DEBUG::2013-11-20 02:41:03,928::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912688::DEBUG::2013-11-20 02:41:03,929::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912688::DEBUG::2013-11-20 02:41:03,930::task::495::TaskManager.Task::(_debug) Task 55d6403a-8044-4267-be3a-2f2a02699a17: ref 0 aborting False >Thread-15912688::INFO::2013-11-20 02:41:03,930::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912689::DEBUG::2013-11-20 02:41:03,935::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912689::INFO::2013-11-20 02:41:03,936::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912689::DEBUG::2013-11-20 02:41:03,936::task::495::TaskManager.Task::(_debug) Task c2d1d8cb-6467-4583-812d-3ab5cd4878de: moving from state init -> state preparing >Thread-15912689::DEBUG::2013-11-20 02:41:03,937::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`670b4f02-049d-4c9c-868d-cd12ea5fb4b5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912689::DEBUG::2013-11-20 02:41:03,938::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912689::DEBUG::2013-11-20 02:41:03,939::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912689::DEBUG::2013-11-20 02:41:03,939::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`670b4f02-049d-4c9c-868d-cd12ea5fb4b5`::Granted request >Thread-15912689::DEBUG::2013-11-20 02:41:03,950::task::495::TaskManager.Task::(_debug) Task c2d1d8cb-6467-4583-812d-3ab5cd4878de: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912689::DEBUG::2013-11-20 02:41:03,950::task::495::TaskManager.Task::(_debug) Task c2d1d8cb-6467-4583-812d-3ab5cd4878de: ref 1 aborting False >Thread-15912689::WARNING::2013-11-20 02:41:03,952::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912689::DEBUG::2013-11-20 02:41:03,953::task::495::TaskManager.Task::(_debug) Task c2d1d8cb-6467-4583-812d-3ab5cd4878de: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115766984704', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115766984704', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912689::DEBUG::2013-11-20 02:41:03,960::task::495::TaskManager.Task::(_debug) Task c2d1d8cb-6467-4583-812d-3ab5cd4878de: moving from state preparing -> state finished >Thread-15912689::DEBUG::2013-11-20 02:41:03,961::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912689::DEBUG::2013-11-20 02:41:03,961::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912689::DEBUG::2013-11-20 02:41:03,962::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912689::DEBUG::2013-11-20 02:41:03,962::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912689::DEBUG::2013-11-20 02:41:03,963::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912689::DEBUG::2013-11-20 02:41:03,963::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912689::DEBUG::2013-11-20 02:41:03,964::task::495::TaskManager.Task::(_debug) Task c2d1d8cb-6467-4583-812d-3ab5cd4878de: ref 0 aborting False >Thread-15912689::INFO::2013-11-20 02:41:03,965::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115766984704', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115766984704', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912690::INFO::2013-11-20 02:41:04,197::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912690::DEBUG::2013-11-20 02:41:04,198::task::495::TaskManager.Task::(_debug) Task 28cdbd1e-adf3-4234-ba10-156da287ed99: moving from state init -> state preparing >Thread-15912690::DEBUG::2013-11-20 02:41:04,198::task::495::TaskManager.Task::(_debug) Task 28cdbd1e-adf3-4234-ba10-156da287ed99: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000423908233643', 'lastCheck': 1384915255.918988, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137495994568', 'lastCheck': 1384915255.9221311, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0519759654999', 'lastCheck': 1384915256.962745, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00055193901062', 'lastCheck': 1384915254.807621, 'valid': True, 'code': 0}} >Thread-15912690::DEBUG::2013-11-20 02:41:04,199::task::495::TaskManager.Task::(_debug) Task 28cdbd1e-adf3-4234-ba10-156da287ed99: moving from state preparing -> state finished >Thread-15912690::DEBUG::2013-11-20 02:41:04,200::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912690::DEBUG::2013-11-20 02:41:04,200::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912690::DEBUG::2013-11-20 02:41:04,201::task::495::TaskManager.Task::(_debug) Task 28cdbd1e-adf3-4234-ba10-156da287ed99: ref 0 aborting False >Thread-15912690::INFO::2013-11-20 02:41:04,201::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000423908233643', 'lastCheck': 1384915255.918988, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137495994568', 'lastCheck': 1384915255.9221311, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0519759654999', 'lastCheck': 1384915256.962745, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00055193901062', 'lastCheck': 1384915254.807621, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:41:05,873::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-42838::DEBUG::2013-11-20 02:41:06,104::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:41:06,560::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0779848 s, 13.1 MB/s\n'; <rc> = 0 >Thread-42838::DEBUG::2013-11-20 02:41:06,856::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:41:08,575::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:41:09,136::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0860558 s, 11.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:41:11,152::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:41:11,717::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0788987 s, 13.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:41:13,732::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:41:14,291::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0754048 s, 13.6 MB/s\n'; <rc> = 0 >Thread-15912695::DEBUG::2013-11-20 02:41:15,939::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912695::INFO::2013-11-20 02:41:15,941::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912695::DEBUG::2013-11-20 02:41:15,941::task::495::TaskManager.Task::(_debug) Task e0aa5f83-b9e7-49ba-b821-956540166695: moving from state init -> state preparing >Thread-15912695::DEBUG::2013-11-20 02:41:15,942::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912695::DEBUG::2013-11-20 02:41:15,943::task::495::TaskManager.Task::(_debug) Task e0aa5f83-b9e7-49ba-b821-956540166695: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912695::DEBUG::2013-11-20 02:41:15,943::task::495::TaskManager.Task::(_debug) Task e0aa5f83-b9e7-49ba-b821-956540166695: moving from state preparing -> state finished >Thread-15912695::DEBUG::2013-11-20 02:41:15,944::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912695::DEBUG::2013-11-20 02:41:15,944::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912695::DEBUG::2013-11-20 02:41:15,945::task::495::TaskManager.Task::(_debug) Task e0aa5f83-b9e7-49ba-b821-956540166695: ref 0 aborting False >Thread-15912695::INFO::2013-11-20 02:41:15,946::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912696::DEBUG::2013-11-20 02:41:15,950::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912696::INFO::2013-11-20 02:41:15,951::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912696::DEBUG::2013-11-20 02:41:15,951::task::495::TaskManager.Task::(_debug) Task 4102de90-a2dd-4059-90d1-5bccc68aa1a5: moving from state init -> state preparing >Thread-15912696::DEBUG::2013-11-20 02:41:15,953::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`78673e99-b3dd-4552-9933-092f1a901ce2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912696::DEBUG::2013-11-20 02:41:15,953::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912696::DEBUG::2013-11-20 02:41:15,954::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912696::DEBUG::2013-11-20 02:41:15,955::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`78673e99-b3dd-4552-9933-092f1a901ce2`::Granted request >Thread-15912696::DEBUG::2013-11-20 02:41:15,956::task::495::TaskManager.Task::(_debug) Task 4102de90-a2dd-4059-90d1-5bccc68aa1a5: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912696::DEBUG::2013-11-20 02:41:15,956::task::495::TaskManager.Task::(_debug) Task 4102de90-a2dd-4059-90d1-5bccc68aa1a5: ref 1 aborting False >Thread-15912696::WARNING::2013-11-20 02:41:15,958::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912696::DEBUG::2013-11-20 02:41:15,959::task::495::TaskManager.Task::(_debug) Task 4102de90-a2dd-4059-90d1-5bccc68aa1a5: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115766984704', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115766984704', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912696::DEBUG::2013-11-20 02:41:15,960::task::495::TaskManager.Task::(_debug) Task 4102de90-a2dd-4059-90d1-5bccc68aa1a5: moving from state preparing -> state finished >Thread-15912696::DEBUG::2013-11-20 02:41:15,961::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912696::DEBUG::2013-11-20 02:41:15,961::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912696::DEBUG::2013-11-20 02:41:15,962::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912696::DEBUG::2013-11-20 02:41:15,962::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912696::DEBUG::2013-11-20 02:41:15,963::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912696::DEBUG::2013-11-20 02:41:15,963::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912696::DEBUG::2013-11-20 02:41:15,964::task::495::TaskManager.Task::(_debug) Task 4102de90-a2dd-4059-90d1-5bccc68aa1a5: ref 0 aborting False >Thread-15912696::INFO::2013-11-20 02:41:15,965::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115766984704', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115766984704', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:41:16,307::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:41:16,857::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0699548 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:41:18,872::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:41:19,409::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0741993 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15912698::INFO::2013-11-20 02:41:21,324::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912698::DEBUG::2013-11-20 02:41:21,325::task::495::TaskManager.Task::(_debug) Task 2f452b6b-adce-4f62-b399-2dd95d6dd468: moving from state init -> state preparing >Thread-15912698::DEBUG::2013-11-20 02:41:21,326::task::495::TaskManager.Task::(_debug) Task 2f452b6b-adce-4f62-b399-2dd95d6dd468: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000442028045654', 'lastCheck': 1384915276.9627709, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00165891647339', 'lastCheck': 1384915276.5488529, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00156903266907', 'lastCheck': 1384915277.1301949, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000427007675171', 'lastCheck': 1384915275.113627, 'valid': True, 'code': 0}} >Thread-15912698::DEBUG::2013-11-20 02:41:21,326::task::495::TaskManager.Task::(_debug) Task 2f452b6b-adce-4f62-b399-2dd95d6dd468: moving from state preparing -> state finished >Thread-15912698::DEBUG::2013-11-20 02:41:21,327::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912698::DEBUG::2013-11-20 02:41:21,328::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912698::DEBUG::2013-11-20 02:41:21,328::task::495::TaskManager.Task::(_debug) Task 2f452b6b-adce-4f62-b399-2dd95d6dd468: ref 0 aborting False >Thread-15912698::INFO::2013-11-20 02:41:21,329::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000442028045654', 'lastCheck': 1384915276.9627709, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00165891647339', 'lastCheck': 1384915276.5488529, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00156903266907', 'lastCheck': 1384915277.1301949, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000427007675171', 'lastCheck': 1384915275.113627, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:41:21,424::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:41:21,968::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.07415 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:41:23,980::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:41:24,519::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0748645 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:41:26,534::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:41:27,088::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0744492 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15912701::DEBUG::2013-11-20 02:41:28,887::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912701::INFO::2013-11-20 02:41:28,889::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912701::DEBUG::2013-11-20 02:41:28,889::task::495::TaskManager.Task::(_debug) Task c39d1344-00d7-4b69-b3a3-a58d9326a606: moving from state init -> state preparing >Thread-15912701::DEBUG::2013-11-20 02:41:28,890::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912701::DEBUG::2013-11-20 02:41:28,891::task::495::TaskManager.Task::(_debug) Task c39d1344-00d7-4b69-b3a3-a58d9326a606: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912701::DEBUG::2013-11-20 02:41:28,891::task::495::TaskManager.Task::(_debug) Task c39d1344-00d7-4b69-b3a3-a58d9326a606: moving from state preparing -> state finished >Thread-15912701::DEBUG::2013-11-20 02:41:28,892::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912701::DEBUG::2013-11-20 02:41:28,892::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912701::DEBUG::2013-11-20 02:41:28,893::task::495::TaskManager.Task::(_debug) Task c39d1344-00d7-4b69-b3a3-a58d9326a606: ref 0 aborting False >Thread-15912701::INFO::2013-11-20 02:41:28,894::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912702::DEBUG::2013-11-20 02:41:28,898::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912702::INFO::2013-11-20 02:41:28,899::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912702::DEBUG::2013-11-20 02:41:28,900::task::495::TaskManager.Task::(_debug) Task 41898ff0-8700-4d03-a6b5-7d51dff93274: moving from state init -> state preparing >Thread-15912702::DEBUG::2013-11-20 02:41:28,901::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f9b2081f-9a31-451d-96a7-ddfa5edaeb07`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912702::DEBUG::2013-11-20 02:41:28,901::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912702::DEBUG::2013-11-20 02:41:28,902::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912702::DEBUG::2013-11-20 02:41:28,903::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f9b2081f-9a31-451d-96a7-ddfa5edaeb07`::Granted request >Thread-15912702::DEBUG::2013-11-20 02:41:28,904::task::495::TaskManager.Task::(_debug) Task 41898ff0-8700-4d03-a6b5-7d51dff93274: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912702::DEBUG::2013-11-20 02:41:28,904::task::495::TaskManager.Task::(_debug) Task 41898ff0-8700-4d03-a6b5-7d51dff93274: ref 1 aborting False >Thread-15912702::WARNING::2013-11-20 02:41:28,906::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912702::DEBUG::2013-11-20 02:41:28,907::task::495::TaskManager.Task::(_debug) Task 41898ff0-8700-4d03-a6b5-7d51dff93274: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115766984704', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115766984704', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912702::DEBUG::2013-11-20 02:41:28,908::task::495::TaskManager.Task::(_debug) Task 41898ff0-8700-4d03-a6b5-7d51dff93274: moving from state preparing -> state finished >Thread-15912702::DEBUG::2013-11-20 02:41:28,908::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912702::DEBUG::2013-11-20 02:41:28,909::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912702::DEBUG::2013-11-20 02:41:28,910::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912702::DEBUG::2013-11-20 02:41:28,910::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912702::DEBUG::2013-11-20 02:41:28,911::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912702::DEBUG::2013-11-20 02:41:28,911::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912702::DEBUG::2013-11-20 02:41:28,912::task::495::TaskManager.Task::(_debug) Task 41898ff0-8700-4d03-a6b5-7d51dff93274: ref 0 aborting False >Thread-15912702::INFO::2013-11-20 02:41:28,912::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115766984704', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115766984704', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:41:29,104::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:41:29,634::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0773064 s, 13.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:41:31,649::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:41:32,232::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.075587 s, 13.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:41:34,247::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:41:34,791::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.075739 s, 13.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:41:36,807::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:41:37,369::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0765814 s, 13.4 MB/s\n'; <rc> = 0 >Thread-15912706::INFO::2013-11-20 02:41:38,861::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912706::DEBUG::2013-11-20 02:41:38,862::task::495::TaskManager.Task::(_debug) Task 11cf2abf-c45a-4353-99d7-3969b9a818c9: moving from state init -> state preparing >Thread-15912706::DEBUG::2013-11-20 02:41:38,863::task::495::TaskManager.Task::(_debug) Task 11cf2abf-c45a-4353-99d7-3969b9a818c9: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000447034835815', 'lastCheck': 1384915297.4855809, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154900550842', 'lastCheck': 1384915297.0441411, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00154519081116', 'lastCheck': 1384915297.4886761, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000396013259888', 'lastCheck': 1384915295.4777319, 'valid': True, 'code': 0}} >Thread-15912706::DEBUG::2013-11-20 02:41:38,864::task::495::TaskManager.Task::(_debug) Task 11cf2abf-c45a-4353-99d7-3969b9a818c9: moving from state preparing -> state finished >Thread-15912706::DEBUG::2013-11-20 02:41:38,864::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912706::DEBUG::2013-11-20 02:41:38,865::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912706::DEBUG::2013-11-20 02:41:38,865::task::495::TaskManager.Task::(_debug) Task 11cf2abf-c45a-4353-99d7-3969b9a818c9: ref 0 aborting False >Thread-15912706::INFO::2013-11-20 02:41:38,866::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000447034835815', 'lastCheck': 1384915297.4855809, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154900550842', 'lastCheck': 1384915297.0441411, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00154519081116', 'lastCheck': 1384915297.4886761, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000396013259888', 'lastCheck': 1384915295.4777319, 'valid': True, 'code': 0}} >Thread-15912708::DEBUG::2013-11-20 02:41:39,352::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912708::INFO::2013-11-20 02:41:39,353::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912708::DEBUG::2013-11-20 02:41:39,354::task::495::TaskManager.Task::(_debug) Task f1bec3e4-e7cd-4739-b278-0d05f0288d0a: moving from state init -> state preparing >Thread-15912708::DEBUG::2013-11-20 02:41:39,354::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912708::DEBUG::2013-11-20 02:41:39,355::task::495::TaskManager.Task::(_debug) Task f1bec3e4-e7cd-4739-b278-0d05f0288d0a: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912708::DEBUG::2013-11-20 02:41:39,355::task::495::TaskManager.Task::(_debug) Task f1bec3e4-e7cd-4739-b278-0d05f0288d0a: moving from state preparing -> state finished >Thread-15912708::DEBUG::2013-11-20 02:41:39,356::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912708::DEBUG::2013-11-20 02:41:39,356::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912708::DEBUG::2013-11-20 02:41:39,357::task::495::TaskManager.Task::(_debug) Task f1bec3e4-e7cd-4739-b278-0d05f0288d0a: ref 0 aborting False >Thread-15912708::INFO::2013-11-20 02:41:39,367::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912709::DEBUG::2013-11-20 02:41:39,371::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912709::INFO::2013-11-20 02:41:39,372::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912709::DEBUG::2013-11-20 02:41:39,373::task::495::TaskManager.Task::(_debug) Task 43d11c4c-b07e-49af-a1e5-10f9c67822e8: moving from state init -> state preparing >Thread-15912709::DEBUG::2013-11-20 02:41:39,374::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ea6fd0b8-ed24-4023-bfae-011ddf9ecbf5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912709::DEBUG::2013-11-20 02:41:39,375::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912709::DEBUG::2013-11-20 02:41:39,376::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912709::DEBUG::2013-11-20 02:41:39,376::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ea6fd0b8-ed24-4023-bfae-011ddf9ecbf5`::Granted request >Thread-15912709::DEBUG::2013-11-20 02:41:39,377::task::495::TaskManager.Task::(_debug) Task 43d11c4c-b07e-49af-a1e5-10f9c67822e8: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912709::DEBUG::2013-11-20 02:41:39,378::task::495::TaskManager.Task::(_debug) Task 43d11c4c-b07e-49af-a1e5-10f9c67822e8: ref 1 aborting False >Thread-15912709::WARNING::2013-11-20 02:41:39,379::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912709::DEBUG::2013-11-20 02:41:39,380::task::495::TaskManager.Task::(_debug) Task 43d11c4c-b07e-49af-a1e5-10f9c67822e8: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912709::DEBUG::2013-11-20 02:41:39,381::task::495::TaskManager.Task::(_debug) Task 43d11c4c-b07e-49af-a1e5-10f9c67822e8: moving from state preparing -> state finished >Thread-15912709::DEBUG::2013-11-20 02:41:39,382::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912709::DEBUG::2013-11-20 02:41:39,382::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912709::DEBUG::2013-11-20 02:41:39,383::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912709::DEBUG::2013-11-20 02:41:39,383::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912709::DEBUG::2013-11-20 02:41:39,384::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912709::DEBUG::2013-11-20 02:41:39,384::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912709::DEBUG::2013-11-20 02:41:39,385::task::495::TaskManager.Task::(_debug) Task 43d11c4c-b07e-49af-a1e5-10f9c67822e8: ref 0 aborting False >Dummy-15912651::DEBUG::2013-11-20 02:41:39,386::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912709::INFO::2013-11-20 02:41:39,386::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:41:39,983::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0775364 s, 13.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:41:41,993::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:41:42,593::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0737248 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:41:44,608::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:41:45,149::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0701212 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:41:47,164::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:41:47,625::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0472678 s, 21.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:41:49,635::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912713::DEBUG::2013-11-20 02:41:50,065::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912713::INFO::2013-11-20 02:41:50,067::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912713::DEBUG::2013-11-20 02:41:50,068::task::495::TaskManager.Task::(_debug) Task 8cabe733-be44-4862-b6e2-ce2bfb9e3e8d: moving from state init -> state preparing >Thread-15912713::DEBUG::2013-11-20 02:41:50,069::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912713::DEBUG::2013-11-20 02:41:50,070::task::495::TaskManager.Task::(_debug) Task 8cabe733-be44-4862-b6e2-ce2bfb9e3e8d: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912713::DEBUG::2013-11-20 02:41:50,071::task::495::TaskManager.Task::(_debug) Task 8cabe733-be44-4862-b6e2-ce2bfb9e3e8d: moving from state preparing -> state finished >Thread-15912713::DEBUG::2013-11-20 02:41:50,071::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912713::DEBUG::2013-11-20 02:41:50,072::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912713::DEBUG::2013-11-20 02:41:50,078::task::495::TaskManager.Task::(_debug) Task 8cabe733-be44-4862-b6e2-ce2bfb9e3e8d: ref 0 aborting False >Thread-15912713::INFO::2013-11-20 02:41:50,079::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912714::DEBUG::2013-11-20 02:41:50,084::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912714::INFO::2013-11-20 02:41:50,085::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912714::DEBUG::2013-11-20 02:41:50,086::task::495::TaskManager.Task::(_debug) Task 4f50c36e-c486-4b5d-8d3a-906704787593: moving from state init -> state preparing >Thread-15912714::DEBUG::2013-11-20 02:41:50,087::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9b694b36-0a08-4b40-a1d6-d1878f5779cc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912714::DEBUG::2013-11-20 02:41:50,088::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912714::DEBUG::2013-11-20 02:41:50,089::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912714::DEBUG::2013-11-20 02:41:50,090::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9b694b36-0a08-4b40-a1d6-d1878f5779cc`::Granted request >Thread-15912714::DEBUG::2013-11-20 02:41:50,092::task::495::TaskManager.Task::(_debug) Task 4f50c36e-c486-4b5d-8d3a-906704787593: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912714::DEBUG::2013-11-20 02:41:50,092::task::495::TaskManager.Task::(_debug) Task 4f50c36e-c486-4b5d-8d3a-906704787593: ref 1 aborting False >Thread-15912714::WARNING::2013-11-20 02:41:50,095::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912714::DEBUG::2013-11-20 02:41:50,096::task::495::TaskManager.Task::(_debug) Task 4f50c36e-c486-4b5d-8d3a-906704787593: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912714::DEBUG::2013-11-20 02:41:50,098::task::495::TaskManager.Task::(_debug) Task 4f50c36e-c486-4b5d-8d3a-906704787593: moving from state preparing -> state finished >Thread-15912714::DEBUG::2013-11-20 02:41:50,098::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912714::DEBUG::2013-11-20 02:41:50,099::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912714::DEBUG::2013-11-20 02:41:50,101::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912714::DEBUG::2013-11-20 02:41:50,102::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912714::DEBUG::2013-11-20 02:41:50,103::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912714::DEBUG::2013-11-20 02:41:50,103::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912714::DEBUG::2013-11-20 02:41:50,104::task::495::TaskManager.Task::(_debug) Task 4f50c36e-c486-4b5d-8d3a-906704787593: ref 0 aborting False >Thread-15912714::INFO::2013-11-20 02:41:50,107::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:41:50,222::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0931993 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:41:52,237::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:41:52,779::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0721959 s, 14.2 MB/s\n'; <rc> = 0 >Thread-15912716::INFO::2013-11-20 02:41:54,317::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912716::DEBUG::2013-11-20 02:41:54,318::task::495::TaskManager.Task::(_debug) Task 39624bcf-4acf-44a5-af1e-5d4995fd9f78: moving from state init -> state preparing >Thread-15912716::DEBUG::2013-11-20 02:41:54,319::task::495::TaskManager.Task::(_debug) Task 39624bcf-4acf-44a5-af1e-5d4995fd9f78: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000433921813965', 'lastCheck': 1384915307.825901, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0012309551239', 'lastCheck': 1384915307.4356811, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00122404098511', 'lastCheck': 1384915307.8317511, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000479936599731', 'lastCheck': 1384915305.826436, 'valid': True, 'code': 0}} >Thread-15912716::DEBUG::2013-11-20 02:41:54,320::task::495::TaskManager.Task::(_debug) Task 39624bcf-4acf-44a5-af1e-5d4995fd9f78: moving from state preparing -> state finished >Thread-15912716::DEBUG::2013-11-20 02:41:54,320::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912716::DEBUG::2013-11-20 02:41:54,321::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912716::DEBUG::2013-11-20 02:41:54,322::task::495::TaskManager.Task::(_debug) Task 39624bcf-4acf-44a5-af1e-5d4995fd9f78: ref 0 aborting False >Thread-15912716::INFO::2013-11-20 02:41:54,322::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000433921813965', 'lastCheck': 1384915307.825901, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0012309551239', 'lastCheck': 1384915307.4356811, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00122404098511', 'lastCheck': 1384915307.8317511, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000479936599731', 'lastCheck': 1384915305.826436, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:41:54,794::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:41:55,359::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0749692 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:41:57,374::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:41:57,941::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0758348 s, 13.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:41:59,956::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:42:00,496::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.074524 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15912719::DEBUG::2013-11-20 02:42:00,850::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912719::INFO::2013-11-20 02:42:00,851::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912719::DEBUG::2013-11-20 02:42:00,852::task::495::TaskManager.Task::(_debug) Task 8a2e89f9-df06-4ce8-b33d-e467be94f11a: moving from state init -> state preparing >Thread-15912719::DEBUG::2013-11-20 02:42:00,853::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912719::DEBUG::2013-11-20 02:42:00,854::task::495::TaskManager.Task::(_debug) Task 8a2e89f9-df06-4ce8-b33d-e467be94f11a: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912719::DEBUG::2013-11-20 02:42:00,854::task::495::TaskManager.Task::(_debug) Task 8a2e89f9-df06-4ce8-b33d-e467be94f11a: moving from state preparing -> state finished >Thread-15912719::DEBUG::2013-11-20 02:42:00,855::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912719::DEBUG::2013-11-20 02:42:00,855::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912719::DEBUG::2013-11-20 02:42:00,856::task::495::TaskManager.Task::(_debug) Task 8a2e89f9-df06-4ce8-b33d-e467be94f11a: ref 0 aborting False >Thread-15912719::INFO::2013-11-20 02:42:00,856::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912720::DEBUG::2013-11-20 02:42:00,861::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912720::INFO::2013-11-20 02:42:00,862::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912720::DEBUG::2013-11-20 02:42:00,862::task::495::TaskManager.Task::(_debug) Task 944d892f-683d-4646-8b08-61a0b7c25914: moving from state init -> state preparing >Thread-15912720::DEBUG::2013-11-20 02:42:00,864::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d4725b73-6630-49ff-a9ba-29bb988efa51`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912720::DEBUG::2013-11-20 02:42:00,864::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912720::DEBUG::2013-11-20 02:42:00,865::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912720::DEBUG::2013-11-20 02:42:00,866::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d4725b73-6630-49ff-a9ba-29bb988efa51`::Granted request >Thread-15912720::DEBUG::2013-11-20 02:42:00,867::task::495::TaskManager.Task::(_debug) Task 944d892f-683d-4646-8b08-61a0b7c25914: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912720::DEBUG::2013-11-20 02:42:00,867::task::495::TaskManager.Task::(_debug) Task 944d892f-683d-4646-8b08-61a0b7c25914: ref 1 aborting False >Thread-15912720::WARNING::2013-11-20 02:42:00,869::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912720::DEBUG::2013-11-20 02:42:00,870::task::495::TaskManager.Task::(_debug) Task 944d892f-683d-4646-8b08-61a0b7c25914: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912720::DEBUG::2013-11-20 02:42:00,871::task::495::TaskManager.Task::(_debug) Task 944d892f-683d-4646-8b08-61a0b7c25914: moving from state preparing -> state finished >Thread-15912720::DEBUG::2013-11-20 02:42:00,872::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912720::DEBUG::2013-11-20 02:42:00,872::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912720::DEBUG::2013-11-20 02:42:00,873::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912720::DEBUG::2013-11-20 02:42:00,873::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912720::DEBUG::2013-11-20 02:42:00,874::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912720::DEBUG::2013-11-20 02:42:00,874::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912720::DEBUG::2013-11-20 02:42:00,875::task::495::TaskManager.Task::(_debug) Task 944d892f-683d-4646-8b08-61a0b7c25914: ref 0 aborting False >Thread-15912720::INFO::2013-11-20 02:42:00,876::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912392::INFO::2013-11-20 02:42:01,053::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:42:01,054::task::495::TaskManager.Task::(_debug) Task d1871768-f8e6-4240-b157-6668c7c760c1: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:42:01,056::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f7a2d776-3e23-439e-af22-3c2719b5a3f2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:42:01,056::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:42:01,057::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:42:01,057::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f7a2d776-3e23-439e-af22-3c2719b5a3f2`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:42:01,058::task::495::TaskManager.Task::(_debug) Task d1871768-f8e6-4240-b157-6668c7c760c1: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:42:01,059::task::495::TaskManager.Task::(_debug) Task d1871768-f8e6-4240-b157-6668c7c760c1: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:42:01,059::task::495::TaskManager.Task::(_debug) Task d1871768-f8e6-4240-b157-6668c7c760c1: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:42:01,060::task::495::TaskManager.Task::(_debug) Task d1871768-f8e6-4240-b157-6668c7c760c1: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:42:01,061::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:42:01,061::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:42:01,062::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:42:01,062::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:42:01,063::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:42:01,064::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:42:01,064::task::495::TaskManager.Task::(_debug) Task d1871768-f8e6-4240-b157-6668c7c760c1: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:42:01,065::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 02:42:02,512::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:42:03,099::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0800868 s, 12.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:42:05,116::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:42:05,657::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0902047 s, 11.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:42:07,672::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:42:08,238::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0746545 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15912724::INFO::2013-11-20 02:42:10,008::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912724::DEBUG::2013-11-20 02:42:10,008::task::495::TaskManager.Task::(_debug) Task 06a5c44d-436c-44ae-9826-d329bb55075a: moving from state init -> state preparing >Thread-15912724::DEBUG::2013-11-20 02:42:10,009::task::495::TaskManager.Task::(_debug) Task 06a5c44d-436c-44ae-9826-d329bb55075a: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000664949417114', 'lastCheck': 1384915328.3280859, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00224781036377', 'lastCheck': 1384915327.90611, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00123286247253', 'lastCheck': 1384915328.3297441, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000549077987671', 'lastCheck': 1384915326.322731, 'valid': True, 'code': 0}} >Thread-15912724::DEBUG::2013-11-20 02:42:10,010::task::495::TaskManager.Task::(_debug) Task 06a5c44d-436c-44ae-9826-d329bb55075a: moving from state preparing -> state finished >Thread-15912724::DEBUG::2013-11-20 02:42:10,011::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912724::DEBUG::2013-11-20 02:42:10,011::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912724::DEBUG::2013-11-20 02:42:10,012::task::495::TaskManager.Task::(_debug) Task 06a5c44d-436c-44ae-9826-d329bb55075a: ref 0 aborting False >Thread-15912724::INFO::2013-11-20 02:42:10,012::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000664949417114', 'lastCheck': 1384915328.3280859, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00224781036377', 'lastCheck': 1384915327.90611, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00123286247253', 'lastCheck': 1384915328.3297441, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000549077987671', 'lastCheck': 1384915326.322731, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:42:10,247::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:42:10,768::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0700235 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15912726::DEBUG::2013-11-20 02:42:11,918::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912726::INFO::2013-11-20 02:42:11,919::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912726::DEBUG::2013-11-20 02:42:11,931::task::495::TaskManager.Task::(_debug) Task 65ea6596-3e20-4ce1-b030-bb2cbe486498: moving from state init -> state preparing >Thread-15912726::DEBUG::2013-11-20 02:42:11,932::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912726::DEBUG::2013-11-20 02:42:11,933::task::495::TaskManager.Task::(_debug) Task 65ea6596-3e20-4ce1-b030-bb2cbe486498: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912726::DEBUG::2013-11-20 02:42:11,933::task::495::TaskManager.Task::(_debug) Task 65ea6596-3e20-4ce1-b030-bb2cbe486498: moving from state preparing -> state finished >Thread-15912726::DEBUG::2013-11-20 02:42:11,934::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912726::DEBUG::2013-11-20 02:42:11,934::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912726::DEBUG::2013-11-20 02:42:11,935::task::495::TaskManager.Task::(_debug) Task 65ea6596-3e20-4ce1-b030-bb2cbe486498: ref 0 aborting False >Thread-15912726::INFO::2013-11-20 02:42:11,936::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912727::DEBUG::2013-11-20 02:42:11,940::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912727::INFO::2013-11-20 02:42:11,941::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912727::DEBUG::2013-11-20 02:42:11,942::task::495::TaskManager.Task::(_debug) Task eec90911-c950-4316-8209-ce35378be8f1: moving from state init -> state preparing >Thread-15912727::DEBUG::2013-11-20 02:42:11,943::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1f52d38a-3ec9-464e-885c-395dead1e9ff`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912727::DEBUG::2013-11-20 02:42:11,943::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912727::DEBUG::2013-11-20 02:42:11,944::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912727::DEBUG::2013-11-20 02:42:11,945::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1f52d38a-3ec9-464e-885c-395dead1e9ff`::Granted request >Thread-15912727::DEBUG::2013-11-20 02:42:11,946::task::495::TaskManager.Task::(_debug) Task eec90911-c950-4316-8209-ce35378be8f1: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912727::DEBUG::2013-11-20 02:42:11,946::task::495::TaskManager.Task::(_debug) Task eec90911-c950-4316-8209-ce35378be8f1: ref 1 aborting False >Thread-15912727::WARNING::2013-11-20 02:42:11,948::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912727::DEBUG::2013-11-20 02:42:11,949::task::495::TaskManager.Task::(_debug) Task eec90911-c950-4316-8209-ce35378be8f1: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912727::DEBUG::2013-11-20 02:42:11,950::task::495::TaskManager.Task::(_debug) Task eec90911-c950-4316-8209-ce35378be8f1: moving from state preparing -> state finished >Thread-15912727::DEBUG::2013-11-20 02:42:11,950::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912727::DEBUG::2013-11-20 02:42:11,951::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912727::DEBUG::2013-11-20 02:42:11,952::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912727::DEBUG::2013-11-20 02:42:11,952::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912727::DEBUG::2013-11-20 02:42:11,953::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912727::DEBUG::2013-11-20 02:42:11,953::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912727::DEBUG::2013-11-20 02:42:11,954::task::495::TaskManager.Task::(_debug) Task eec90911-c950-4316-8209-ce35378be8f1: ref 0 aborting False >Thread-15912727::INFO::2013-11-20 02:42:11,955::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:42:12,784::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:42:13,320::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0713513 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:42:15,335::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:42:15,854::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0701752 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:42:17,869::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:42:18,413::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0748935 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:42:20,428::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:42:20,989::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0843428 s, 12.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:42:23,004::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:42:23,557::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0796514 s, 12.9 MB/s\n'; <rc> = 0 >Thread-15912732::DEBUG::2013-11-20 02:42:24,502::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912732::INFO::2013-11-20 02:42:24,504::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912732::DEBUG::2013-11-20 02:42:24,504::task::495::TaskManager.Task::(_debug) Task 78dbdfb2-fe4e-4fe9-878a-754713c40455: moving from state init -> state preparing >Thread-15912732::DEBUG::2013-11-20 02:42:24,505::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912732::DEBUG::2013-11-20 02:42:24,506::task::495::TaskManager.Task::(_debug) Task 78dbdfb2-fe4e-4fe9-878a-754713c40455: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912732::DEBUG::2013-11-20 02:42:24,506::task::495::TaskManager.Task::(_debug) Task 78dbdfb2-fe4e-4fe9-878a-754713c40455: moving from state preparing -> state finished >Thread-15912732::DEBUG::2013-11-20 02:42:24,507::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912732::DEBUG::2013-11-20 02:42:24,508::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912732::DEBUG::2013-11-20 02:42:24,508::task::495::TaskManager.Task::(_debug) Task 78dbdfb2-fe4e-4fe9-878a-754713c40455: ref 0 aborting False >Thread-15912732::INFO::2013-11-20 02:42:24,509::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912733::DEBUG::2013-11-20 02:42:24,513::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912733::INFO::2013-11-20 02:42:24,514::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912733::DEBUG::2013-11-20 02:42:24,515::task::495::TaskManager.Task::(_debug) Task a043f461-2e1e-4b59-8c25-b6c55bf17ad5: moving from state init -> state preparing >Thread-15912733::DEBUG::2013-11-20 02:42:24,516::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e108d009-8c79-4cb1-81e0-c4d467414342`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912733::DEBUG::2013-11-20 02:42:24,517::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912733::DEBUG::2013-11-20 02:42:24,517::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912733::DEBUG::2013-11-20 02:42:24,518::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e108d009-8c79-4cb1-81e0-c4d467414342`::Granted request >Thread-15912733::DEBUG::2013-11-20 02:42:24,519::task::495::TaskManager.Task::(_debug) Task a043f461-2e1e-4b59-8c25-b6c55bf17ad5: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912733::DEBUG::2013-11-20 02:42:24,520::task::495::TaskManager.Task::(_debug) Task a043f461-2e1e-4b59-8c25-b6c55bf17ad5: ref 1 aborting False >Thread-15912733::WARNING::2013-11-20 02:42:24,521::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912733::DEBUG::2013-11-20 02:42:24,522::task::495::TaskManager.Task::(_debug) Task a043f461-2e1e-4b59-8c25-b6c55bf17ad5: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912733::DEBUG::2013-11-20 02:42:24,523::task::495::TaskManager.Task::(_debug) Task a043f461-2e1e-4b59-8c25-b6c55bf17ad5: moving from state preparing -> state finished >Thread-15912733::DEBUG::2013-11-20 02:42:24,524::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912733::DEBUG::2013-11-20 02:42:24,524::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912733::DEBUG::2013-11-20 02:42:24,525::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912733::DEBUG::2013-11-20 02:42:24,526::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912733::DEBUG::2013-11-20 02:42:24,526::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912733::DEBUG::2013-11-20 02:42:24,527::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912733::DEBUG::2013-11-20 02:42:24,527::task::495::TaskManager.Task::(_debug) Task a043f461-2e1e-4b59-8c25-b6c55bf17ad5: ref 0 aborting False >Thread-15912733::INFO::2013-11-20 02:42:24,528::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:42:25,572::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912734::INFO::2013-11-20 02:42:25,793::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912734::DEBUG::2013-11-20 02:42:25,794::task::495::TaskManager.Task::(_debug) Task a3d7befd-81fc-46e6-a8f5-94ff5f25899e: moving from state init -> state preparing >Thread-15912734::DEBUG::2013-11-20 02:42:25,795::task::495::TaskManager.Task::(_debug) Task a3d7befd-81fc-46e6-a8f5-94ff5f25899e: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000395059585571', 'lastCheck': 1384915338.5476961, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.124522924423', 'lastCheck': 1384915338.222285, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00137615203857', 'lastCheck': 1384915338.5493159, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000427007675171', 'lastCheck': 1384915336.5404551, 'valid': True, 'code': 0}} >Thread-15912734::DEBUG::2013-11-20 02:42:25,796::task::495::TaskManager.Task::(_debug) Task a3d7befd-81fc-46e6-a8f5-94ff5f25899e: moving from state preparing -> state finished >Thread-15912734::DEBUG::2013-11-20 02:42:25,797::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912734::DEBUG::2013-11-20 02:42:25,797::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912734::DEBUG::2013-11-20 02:42:25,798::task::495::TaskManager.Task::(_debug) Task a3d7befd-81fc-46e6-a8f5-94ff5f25899e: ref 0 aborting False >Thread-15912734::INFO::2013-11-20 02:42:25,799::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000395059585571', 'lastCheck': 1384915338.5476961, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.124522924423', 'lastCheck': 1384915338.222285, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00137615203857', 'lastCheck': 1384915338.5493159, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000427007675171', 'lastCheck': 1384915336.5404551, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:42:26,099::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.070764 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:42:28,114::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:42:28,643::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0762249 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:42:30,657::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:42:31,219::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0763 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:42:33,236::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:42:33,773::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0704052 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15912738::DEBUG::2013-11-20 02:42:35,324::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912738::INFO::2013-11-20 02:42:35,326::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912738::DEBUG::2013-11-20 02:42:35,326::task::495::TaskManager.Task::(_debug) Task 1ccbbf53-3f84-45b0-b087-b46d906f0b65: moving from state init -> state preparing >Thread-15912738::DEBUG::2013-11-20 02:42:35,327::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912738::DEBUG::2013-11-20 02:42:35,328::task::495::TaskManager.Task::(_debug) Task 1ccbbf53-3f84-45b0-b087-b46d906f0b65: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912738::DEBUG::2013-11-20 02:42:35,328::task::495::TaskManager.Task::(_debug) Task 1ccbbf53-3f84-45b0-b087-b46d906f0b65: moving from state preparing -> state finished >Thread-15912738::DEBUG::2013-11-20 02:42:35,329::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912738::DEBUG::2013-11-20 02:42:35,330::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912738::DEBUG::2013-11-20 02:42:35,331::task::495::TaskManager.Task::(_debug) Task 1ccbbf53-3f84-45b0-b087-b46d906f0b65: ref 0 aborting False >Thread-15912738::INFO::2013-11-20 02:42:35,332::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912739::DEBUG::2013-11-20 02:42:35,337::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912739::INFO::2013-11-20 02:42:35,338::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912739::DEBUG::2013-11-20 02:42:35,338::task::495::TaskManager.Task::(_debug) Task 7ee15d31-5158-461d-ae73-7828ca7a6aa7: moving from state init -> state preparing >Thread-15912739::DEBUG::2013-11-20 02:42:35,339::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`54ce41e1-64d1-48d3-bc7f-c2a97110a908`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912739::DEBUG::2013-11-20 02:42:35,340::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912739::DEBUG::2013-11-20 02:42:35,341::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912739::DEBUG::2013-11-20 02:42:35,341::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`54ce41e1-64d1-48d3-bc7f-c2a97110a908`::Granted request >Thread-15912739::DEBUG::2013-11-20 02:42:35,342::task::495::TaskManager.Task::(_debug) Task 7ee15d31-5158-461d-ae73-7828ca7a6aa7: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912739::DEBUG::2013-11-20 02:42:35,343::task::495::TaskManager.Task::(_debug) Task 7ee15d31-5158-461d-ae73-7828ca7a6aa7: ref 1 aborting False >Thread-15912739::WARNING::2013-11-20 02:42:35,345::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912739::DEBUG::2013-11-20 02:42:35,346::task::495::TaskManager.Task::(_debug) Task 7ee15d31-5158-461d-ae73-7828ca7a6aa7: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912739::DEBUG::2013-11-20 02:42:35,347::task::495::TaskManager.Task::(_debug) Task 7ee15d31-5158-461d-ae73-7828ca7a6aa7: moving from state preparing -> state finished >Thread-15912739::DEBUG::2013-11-20 02:42:35,348::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912739::DEBUG::2013-11-20 02:42:35,348::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912739::DEBUG::2013-11-20 02:42:35,349::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912739::DEBUG::2013-11-20 02:42:35,350::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912739::DEBUG::2013-11-20 02:42:35,350::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912739::DEBUG::2013-11-20 02:42:35,351::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912739::DEBUG::2013-11-20 02:42:35,351::task::495::TaskManager.Task::(_debug) Task 7ee15d31-5158-461d-ae73-7828ca7a6aa7: ref 0 aborting False >Thread-15912739::INFO::2013-11-20 02:42:35,352::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:42:35,788::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:42:36,322::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.070235 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:42:38,338::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:42:38,872::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0819654 s, 12.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:42:40,887::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:42:41,445::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0766569 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:42:43,471::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912742::INFO::2013-11-20 02:42:43,720::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912742::DEBUG::2013-11-20 02:42:43,722::task::495::TaskManager.Task::(_debug) Task 61b61543-d296-4eba-b5d8-3591263f4582: moving from state init -> state preparing >Thread-15912742::DEBUG::2013-11-20 02:42:43,723::task::495::TaskManager.Task::(_debug) Task 61b61543-d296-4eba-b5d8-3591263f4582: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000471830368042', 'lastCheck': 1384915359.0243089, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00143384933472', 'lastCheck': 1384915358.581356, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00233507156372', 'lastCheck': 1384915359.0230131, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00065803527832', 'lastCheck': 1384915357.0111251, 'valid': True, 'code': 0}} >Thread-15912742::DEBUG::2013-11-20 02:42:43,724::task::495::TaskManager.Task::(_debug) Task 61b61543-d296-4eba-b5d8-3591263f4582: moving from state preparing -> state finished >Thread-15912742::DEBUG::2013-11-20 02:42:43,724::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912742::DEBUG::2013-11-20 02:42:43,725::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912742::DEBUG::2013-11-20 02:42:43,726::task::495::TaskManager.Task::(_debug) Task 61b61543-d296-4eba-b5d8-3591263f4582: ref 0 aborting False >Thread-15912742::INFO::2013-11-20 02:42:43,726::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000471830368042', 'lastCheck': 1384915359.0243089, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00143384933472', 'lastCheck': 1384915358.581356, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00233507156372', 'lastCheck': 1384915359.0230131, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00065803527832', 'lastCheck': 1384915357.0111251, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:42:43,979::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0641547 s, 16.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:42:45,993::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912744::DEBUG::2013-11-20 02:42:46,225::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912744::INFO::2013-11-20 02:42:46,227::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912744::DEBUG::2013-11-20 02:42:46,228::task::495::TaskManager.Task::(_debug) Task e4d28eb3-4182-4576-8707-5ab4378c94f6: moving from state init -> state preparing >Thread-15912744::DEBUG::2013-11-20 02:42:46,229::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912744::DEBUG::2013-11-20 02:42:46,230::task::495::TaskManager.Task::(_debug) Task e4d28eb3-4182-4576-8707-5ab4378c94f6: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912744::DEBUG::2013-11-20 02:42:46,230::task::495::TaskManager.Task::(_debug) Task e4d28eb3-4182-4576-8707-5ab4378c94f6: moving from state preparing -> state finished >Thread-15912744::DEBUG::2013-11-20 02:42:46,231::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912744::DEBUG::2013-11-20 02:42:46,232::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912744::DEBUG::2013-11-20 02:42:46,232::task::495::TaskManager.Task::(_debug) Task e4d28eb3-4182-4576-8707-5ab4378c94f6: ref 0 aborting False >Thread-15912744::INFO::2013-11-20 02:42:46,233::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912745::DEBUG::2013-11-20 02:42:46,239::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912745::INFO::2013-11-20 02:42:46,240::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912745::DEBUG::2013-11-20 02:42:46,241::task::495::TaskManager.Task::(_debug) Task 1feb7869-6b83-43b2-be2a-2eb62509d8de: moving from state init -> state preparing >Thread-15912745::DEBUG::2013-11-20 02:42:46,242::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2658b0d4-190a-4bed-a05c-5038709070ad`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912745::DEBUG::2013-11-20 02:42:46,243::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912745::DEBUG::2013-11-20 02:42:46,244::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912745::DEBUG::2013-11-20 02:42:46,245::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2658b0d4-190a-4bed-a05c-5038709070ad`::Granted request >Thread-15912745::DEBUG::2013-11-20 02:42:46,246::task::495::TaskManager.Task::(_debug) Task 1feb7869-6b83-43b2-be2a-2eb62509d8de: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912745::DEBUG::2013-11-20 02:42:46,246::task::495::TaskManager.Task::(_debug) Task 1feb7869-6b83-43b2-be2a-2eb62509d8de: ref 1 aborting False >Thread-15912745::WARNING::2013-11-20 02:42:46,249::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912745::DEBUG::2013-11-20 02:42:46,250::task::495::TaskManager.Task::(_debug) Task 1feb7869-6b83-43b2-be2a-2eb62509d8de: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912745::DEBUG::2013-11-20 02:42:46,251::task::495::TaskManager.Task::(_debug) Task 1feb7869-6b83-43b2-be2a-2eb62509d8de: moving from state preparing -> state finished >Thread-15912745::DEBUG::2013-11-20 02:42:46,251::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912745::DEBUG::2013-11-20 02:42:46,252::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912745::DEBUG::2013-11-20 02:42:46,253::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912745::DEBUG::2013-11-20 02:42:46,253::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912745::DEBUG::2013-11-20 02:42:46,254::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912745::DEBUG::2013-11-20 02:42:46,254::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912745::DEBUG::2013-11-20 02:42:46,255::task::495::TaskManager.Task::(_debug) Task 1feb7869-6b83-43b2-be2a-2eb62509d8de: ref 0 aborting False >Thread-15912745::INFO::2013-11-20 02:42:46,256::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:42:46,535::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0771923 s, 13.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:42:48,551::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:42:49,092::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0781407 s, 13.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:42:51,109::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:42:51,628::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0753141 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:42:53,643::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:42:54,193::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.074161 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:42:56,208::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:42:56,722::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0758035 s, 13.5 MB/s\n'; <rc> = 0 >Thread-15912750::DEBUG::2013-11-20 02:42:57,407::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912750::INFO::2013-11-20 02:42:57,409::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912750::DEBUG::2013-11-20 02:42:57,409::task::495::TaskManager.Task::(_debug) Task 89f301a3-785c-47af-bcda-a621e8ea41e7: moving from state init -> state preparing >Thread-15912750::DEBUG::2013-11-20 02:42:57,410::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912750::DEBUG::2013-11-20 02:42:57,411::task::495::TaskManager.Task::(_debug) Task 89f301a3-785c-47af-bcda-a621e8ea41e7: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912750::DEBUG::2013-11-20 02:42:57,411::task::495::TaskManager.Task::(_debug) Task 89f301a3-785c-47af-bcda-a621e8ea41e7: moving from state preparing -> state finished >Thread-15912750::DEBUG::2013-11-20 02:42:57,412::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912750::DEBUG::2013-11-20 02:42:57,412::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912750::DEBUG::2013-11-20 02:42:57,413::task::495::TaskManager.Task::(_debug) Task 89f301a3-785c-47af-bcda-a621e8ea41e7: ref 0 aborting False >Thread-15912750::INFO::2013-11-20 02:42:57,414::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912751::DEBUG::2013-11-20 02:42:57,425::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912751::INFO::2013-11-20 02:42:57,426::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912751::DEBUG::2013-11-20 02:42:57,427::task::495::TaskManager.Task::(_debug) Task a2834101-b195-4030-b5f9-8ca771e7b8f6: moving from state init -> state preparing >Thread-15912751::DEBUG::2013-11-20 02:42:57,428::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`da26024c-3411-4463-b659-c9f4ce7c72b3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912751::DEBUG::2013-11-20 02:42:57,428::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912751::DEBUG::2013-11-20 02:42:57,429::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912751::DEBUG::2013-11-20 02:42:57,430::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`da26024c-3411-4463-b659-c9f4ce7c72b3`::Granted request >Thread-15912751::DEBUG::2013-11-20 02:42:57,431::task::495::TaskManager.Task::(_debug) Task a2834101-b195-4030-b5f9-8ca771e7b8f6: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912751::DEBUG::2013-11-20 02:42:57,431::task::495::TaskManager.Task::(_debug) Task a2834101-b195-4030-b5f9-8ca771e7b8f6: ref 1 aborting False >Thread-15912751::WARNING::2013-11-20 02:42:57,433::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912751::DEBUG::2013-11-20 02:42:57,434::task::495::TaskManager.Task::(_debug) Task a2834101-b195-4030-b5f9-8ca771e7b8f6: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912751::DEBUG::2013-11-20 02:42:57,435::task::495::TaskManager.Task::(_debug) Task a2834101-b195-4030-b5f9-8ca771e7b8f6: moving from state preparing -> state finished >Thread-15912751::DEBUG::2013-11-20 02:42:57,435::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912751::DEBUG::2013-11-20 02:42:57,436::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912751::DEBUG::2013-11-20 02:42:57,436::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912751::DEBUG::2013-11-20 02:42:57,437::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912751::DEBUG::2013-11-20 02:42:57,437::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912751::DEBUG::2013-11-20 02:42:57,438::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912751::DEBUG::2013-11-20 02:42:57,439::task::495::TaskManager.Task::(_debug) Task a2834101-b195-4030-b5f9-8ca771e7b8f6: ref 0 aborting False >Thread-15912751::INFO::2013-11-20 02:42:57,439::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:42:58,738::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:42:59,160::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0608333 s, 16.8 MB/s\n'; <rc> = 0 >Thread-15912752::INFO::2013-11-20 02:42:59,187::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912752::DEBUG::2013-11-20 02:42:59,188::task::495::TaskManager.Task::(_debug) Task c969cfda-c264-48bc-890a-c2a07de03c7c: moving from state init -> state preparing >Thread-15912752::DEBUG::2013-11-20 02:42:59,188::task::495::TaskManager.Task::(_debug) Task c969cfda-c264-48bc-890a-c2a07de03c7c: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000367879867554', 'lastCheck': 1384915369.2288799, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00118899345398', 'lastCheck': 1384915378.9247739, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00131702423096', 'lastCheck': 1384915369.231473, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00059986114502', 'lastCheck': 1384915377.416348, 'valid': True, 'code': 0}} >Thread-15912752::DEBUG::2013-11-20 02:42:59,189::task::495::TaskManager.Task::(_debug) Task c969cfda-c264-48bc-890a-c2a07de03c7c: moving from state preparing -> state finished >Thread-15912752::DEBUG::2013-11-20 02:42:59,190::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912752::DEBUG::2013-11-20 02:42:59,190::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912752::DEBUG::2013-11-20 02:42:59,191::task::495::TaskManager.Task::(_debug) Task c969cfda-c264-48bc-890a-c2a07de03c7c: ref 0 aborting False >Thread-15912752::INFO::2013-11-20 02:42:59,192::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000367879867554', 'lastCheck': 1384915369.2288799, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00118899345398', 'lastCheck': 1384915378.9247739, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00131702423096', 'lastCheck': 1384915369.231473, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00059986114502', 'lastCheck': 1384915377.416348, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:43:01,171::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 02:43:01,503::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:43:01,504::task::495::TaskManager.Task::(_debug) Task 5540029b-c8a9-4248-9f82-b622954b2e15: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:43:01,506::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`17bcd170-89de-42a3-9009-94daf227da06`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:43:01,507::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:43:01,507::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:43:01,508::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`17bcd170-89de-42a3-9009-94daf227da06`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:43:01,510::task::495::TaskManager.Task::(_debug) Task 5540029b-c8a9-4248-9f82-b622954b2e15: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:43:01,511::task::495::TaskManager.Task::(_debug) Task 5540029b-c8a9-4248-9f82-b622954b2e15: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:43:01,512::task::495::TaskManager.Task::(_debug) Task 5540029b-c8a9-4248-9f82-b622954b2e15: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:43:01,513::task::495::TaskManager.Task::(_debug) Task 5540029b-c8a9-4248-9f82-b622954b2e15: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:43:01,514::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:43:01,515::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:43:01,516::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:43:01,517::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:43:01,518::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:43:01,518::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:43:01,519::task::495::TaskManager.Task::(_debug) Task 5540029b-c8a9-4248-9f82-b622954b2e15: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:43:01,520::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 02:43:01,760::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0878887 s, 11.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:43:03,775::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:04,309::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0749967 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:43:06,325::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:06,835::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.049113 s, 20.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:43:08,849::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:09,344::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0756869 s, 13.5 MB/s\n'; <rc> = 0 >Thread-15912757::DEBUG::2013-11-20 02:43:09,589::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912757::INFO::2013-11-20 02:43:09,591::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912757::DEBUG::2013-11-20 02:43:09,591::task::495::TaskManager.Task::(_debug) Task d6f3855c-d892-4c91-a665-815c20e83b99: moving from state init -> state preparing >Thread-15912757::DEBUG::2013-11-20 02:43:09,592::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912757::DEBUG::2013-11-20 02:43:09,592::task::495::TaskManager.Task::(_debug) Task d6f3855c-d892-4c91-a665-815c20e83b99: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912757::DEBUG::2013-11-20 02:43:09,593::task::495::TaskManager.Task::(_debug) Task d6f3855c-d892-4c91-a665-815c20e83b99: moving from state preparing -> state finished >Thread-15912757::DEBUG::2013-11-20 02:43:09,594::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912757::DEBUG::2013-11-20 02:43:09,594::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912757::DEBUG::2013-11-20 02:43:09,595::task::495::TaskManager.Task::(_debug) Task d6f3855c-d892-4c91-a665-815c20e83b99: ref 0 aborting False >Thread-15912757::INFO::2013-11-20 02:43:09,595::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912758::DEBUG::2013-11-20 02:43:09,599::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912758::INFO::2013-11-20 02:43:09,600::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912758::DEBUG::2013-11-20 02:43:09,601::task::495::TaskManager.Task::(_debug) Task 58c39ccd-eb48-4bc2-87d3-63a14d302f3b: moving from state init -> state preparing >Thread-15912758::DEBUG::2013-11-20 02:43:09,602::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`4918b0ff-90a6-4ff2-8451-f75bbb75393c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912758::DEBUG::2013-11-20 02:43:09,603::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912758::DEBUG::2013-11-20 02:43:09,603::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912758::DEBUG::2013-11-20 02:43:09,604::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`4918b0ff-90a6-4ff2-8451-f75bbb75393c`::Granted request >Thread-15912758::DEBUG::2013-11-20 02:43:09,605::task::495::TaskManager.Task::(_debug) Task 58c39ccd-eb48-4bc2-87d3-63a14d302f3b: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912758::DEBUG::2013-11-20 02:43:09,605::task::495::TaskManager.Task::(_debug) Task 58c39ccd-eb48-4bc2-87d3-63a14d302f3b: ref 1 aborting False >Thread-15912758::WARNING::2013-11-20 02:43:09,607::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912758::DEBUG::2013-11-20 02:43:09,608::task::495::TaskManager.Task::(_debug) Task 58c39ccd-eb48-4bc2-87d3-63a14d302f3b: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912758::DEBUG::2013-11-20 02:43:09,609::task::495::TaskManager.Task::(_debug) Task 58c39ccd-eb48-4bc2-87d3-63a14d302f3b: moving from state preparing -> state finished >Thread-15912758::DEBUG::2013-11-20 02:43:09,610::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912758::DEBUG::2013-11-20 02:43:09,610::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912758::DEBUG::2013-11-20 02:43:09,611::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912758::DEBUG::2013-11-20 02:43:09,612::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912758::DEBUG::2013-11-20 02:43:09,612::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912758::DEBUG::2013-11-20 02:43:09,613::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912758::DEBUG::2013-11-20 02:43:09,613::task::495::TaskManager.Task::(_debug) Task 58c39ccd-eb48-4bc2-87d3-63a14d302f3b: ref 0 aborting False >Thread-15912758::INFO::2013-11-20 02:43:09,614::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:43:11,359::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:11,801::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0486161 s, 21.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:43:13,811::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:14,219::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0474025 s, 21.6 MB/s\n'; <rc> = 0 >Thread-15912760::INFO::2013-11-20 02:43:15,157::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912760::DEBUG::2013-11-20 02:43:15,158::task::495::TaskManager.Task::(_debug) Task 6cd41819-d98e-4959-a961-6903c9efa119: moving from state init -> state preparing >Thread-15912760::DEBUG::2013-11-20 02:43:15,159::task::495::TaskManager.Task::(_debug) Task 6cd41819-d98e-4959-a961-6903c9efa119: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000289916992188', 'lastCheck': 1384915389.544889, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00190305709839', 'lastCheck': 1384915389.0541351, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00107383728027', 'lastCheck': 1384915389.545351, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000518083572388', 'lastCheck': 1384915387.5389841, 'valid': True, 'code': 0}} >Thread-15912760::DEBUG::2013-11-20 02:43:15,160::task::495::TaskManager.Task::(_debug) Task 6cd41819-d98e-4959-a961-6903c9efa119: moving from state preparing -> state finished >Thread-15912760::DEBUG::2013-11-20 02:43:15,161::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912760::DEBUG::2013-11-20 02:43:15,161::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912760::DEBUG::2013-11-20 02:43:15,162::task::495::TaskManager.Task::(_debug) Task 6cd41819-d98e-4959-a961-6903c9efa119: ref 0 aborting False >Thread-15912760::INFO::2013-11-20 02:43:15,162::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000289916992188', 'lastCheck': 1384915389.544889, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00190305709839', 'lastCheck': 1384915389.0541351, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00107383728027', 'lastCheck': 1384915389.545351, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000518083572388', 'lastCheck': 1384915387.5389841, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:43:16,230::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:16,763::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0745511 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:43:18,778::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:19,271::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0704296 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:43:21,286::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:21,813::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0732069 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15912763::DEBUG::2013-11-20 02:43:22,758::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912763::INFO::2013-11-20 02:43:22,759::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912763::DEBUG::2013-11-20 02:43:22,760::task::495::TaskManager.Task::(_debug) Task 5c18aaae-08cb-4a8a-90f3-0cc972afcd12: moving from state init -> state preparing >Thread-15912763::DEBUG::2013-11-20 02:43:22,761::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912763::DEBUG::2013-11-20 02:43:22,761::task::495::TaskManager.Task::(_debug) Task 5c18aaae-08cb-4a8a-90f3-0cc972afcd12: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912763::DEBUG::2013-11-20 02:43:22,762::task::495::TaskManager.Task::(_debug) Task 5c18aaae-08cb-4a8a-90f3-0cc972afcd12: moving from state preparing -> state finished >Thread-15912763::DEBUG::2013-11-20 02:43:22,762::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912763::DEBUG::2013-11-20 02:43:22,763::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912763::DEBUG::2013-11-20 02:43:22,764::task::495::TaskManager.Task::(_debug) Task 5c18aaae-08cb-4a8a-90f3-0cc972afcd12: ref 0 aborting False >Thread-15912763::INFO::2013-11-20 02:43:22,764::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912764::DEBUG::2013-11-20 02:43:22,769::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912764::INFO::2013-11-20 02:43:22,769::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912764::DEBUG::2013-11-20 02:43:22,770::task::495::TaskManager.Task::(_debug) Task 5b39ee8c-89b2-4362-adc9-faa20d2322f7: moving from state init -> state preparing >Thread-15912764::DEBUG::2013-11-20 02:43:22,771::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c3e4df4f-5c3a-49e4-99ba-ed01abf64e1d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912764::DEBUG::2013-11-20 02:43:22,772::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912764::DEBUG::2013-11-20 02:43:22,773::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912764::DEBUG::2013-11-20 02:43:22,773::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c3e4df4f-5c3a-49e4-99ba-ed01abf64e1d`::Granted request >Thread-15912764::DEBUG::2013-11-20 02:43:22,781::task::495::TaskManager.Task::(_debug) Task 5b39ee8c-89b2-4362-adc9-faa20d2322f7: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912764::DEBUG::2013-11-20 02:43:22,781::task::495::TaskManager.Task::(_debug) Task 5b39ee8c-89b2-4362-adc9-faa20d2322f7: ref 1 aborting False >Thread-15912764::WARNING::2013-11-20 02:43:22,783::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912764::DEBUG::2013-11-20 02:43:22,784::task::495::TaskManager.Task::(_debug) Task 5b39ee8c-89b2-4362-adc9-faa20d2322f7: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912764::DEBUG::2013-11-20 02:43:22,785::task::495::TaskManager.Task::(_debug) Task 5b39ee8c-89b2-4362-adc9-faa20d2322f7: moving from state preparing -> state finished >Thread-15912764::DEBUG::2013-11-20 02:43:22,786::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912764::DEBUG::2013-11-20 02:43:22,786::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912764::DEBUG::2013-11-20 02:43:22,787::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912764::DEBUG::2013-11-20 02:43:22,787::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912764::DEBUG::2013-11-20 02:43:22,788::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912764::DEBUG::2013-11-20 02:43:22,788::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912764::DEBUG::2013-11-20 02:43:22,789::task::495::TaskManager.Task::(_debug) Task 5b39ee8c-89b2-4362-adc9-faa20d2322f7: ref 0 aborting False >Thread-15912764::INFO::2013-11-20 02:43:22,790::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:43:23,828::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:24,375::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0736866 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:43:26,389::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:26,929::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0746287 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:43:28,944::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:29,486::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0778107 s, 13.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:43:31,500::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:32,017::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0723419 s, 14.2 MB/s\n'; <rc> = 0 >Thread-15912768::INFO::2013-11-20 02:43:32,683::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912768::DEBUG::2013-11-20 02:43:32,683::task::495::TaskManager.Task::(_debug) Task fd4569dc-f4ee-4175-a75e-39297c8e197c: moving from state init -> state preparing >Thread-15912768::DEBUG::2013-11-20 02:43:32,684::task::495::TaskManager.Task::(_debug) Task fd4569dc-f4ee-4175-a75e-39297c8e197c: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000468015670776', 'lastCheck': 1384915409.6229081, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.111585140228', 'lastCheck': 1384915409.2822199, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.1806640625', 'lastCheck': 1384915409.8253901, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000432968139648', 'lastCheck': 1384915407.623462, 'valid': True, 'code': 0}} >Thread-15912768::DEBUG::2013-11-20 02:43:32,684::task::495::TaskManager.Task::(_debug) Task fd4569dc-f4ee-4175-a75e-39297c8e197c: moving from state preparing -> state finished >Thread-15912768::DEBUG::2013-11-20 02:43:32,685::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912768::DEBUG::2013-11-20 02:43:32,685::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912768::DEBUG::2013-11-20 02:43:32,686::task::495::TaskManager.Task::(_debug) Task fd4569dc-f4ee-4175-a75e-39297c8e197c: ref 0 aborting False >Thread-15912768::INFO::2013-11-20 02:43:32,686::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000468015670776', 'lastCheck': 1384915409.6229081, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.111585140228', 'lastCheck': 1384915409.2822199, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.1806640625', 'lastCheck': 1384915409.8253901, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000432968139648', 'lastCheck': 1384915407.623462, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:43:34,032::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912770::DEBUG::2013-11-20 02:43:34,473::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912770::INFO::2013-11-20 02:43:34,478::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912770::DEBUG::2013-11-20 02:43:34,481::task::495::TaskManager.Task::(_debug) Task 7e4bd729-208c-49a2-b1a5-194cefcffb73: moving from state init -> state preparing >Thread-15912770::DEBUG::2013-11-20 02:43:34,486::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912770::DEBUG::2013-11-20 02:43:34,486::task::495::TaskManager.Task::(_debug) Task 7e4bd729-208c-49a2-b1a5-194cefcffb73: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912770::DEBUG::2013-11-20 02:43:34,487::task::495::TaskManager.Task::(_debug) Task 7e4bd729-208c-49a2-b1a5-194cefcffb73: moving from state preparing -> state finished >Thread-15912770::DEBUG::2013-11-20 02:43:34,488::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912770::DEBUG::2013-11-20 02:43:34,488::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912770::DEBUG::2013-11-20 02:43:34,489::task::495::TaskManager.Task::(_debug) Task 7e4bd729-208c-49a2-b1a5-194cefcffb73: ref 0 aborting False >Thread-15912770::INFO::2013-11-20 02:43:34,490::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912771::DEBUG::2013-11-20 02:43:34,494::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912771::INFO::2013-11-20 02:43:34,495::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912771::DEBUG::2013-11-20 02:43:34,496::task::495::TaskManager.Task::(_debug) Task 2f4ea8c3-89e6-41d7-b798-c11379916d31: moving from state init -> state preparing >Thread-15912771::DEBUG::2013-11-20 02:43:34,497::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f0afc5fa-0110-4c13-a584-14d99e383d54`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912771::DEBUG::2013-11-20 02:43:34,497::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912771::DEBUG::2013-11-20 02:43:34,498::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912771::DEBUG::2013-11-20 02:43:34,499::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f0afc5fa-0110-4c13-a584-14d99e383d54`::Granted request >Thread-15912771::DEBUG::2013-11-20 02:43:34,500::task::495::TaskManager.Task::(_debug) Task 2f4ea8c3-89e6-41d7-b798-c11379916d31: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912771::DEBUG::2013-11-20 02:43:34,500::task::495::TaskManager.Task::(_debug) Task 2f4ea8c3-89e6-41d7-b798-c11379916d31: ref 1 aborting False >Thread-15912771::WARNING::2013-11-20 02:43:34,502::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912771::DEBUG::2013-11-20 02:43:34,503::task::495::TaskManager.Task::(_debug) Task 2f4ea8c3-89e6-41d7-b798-c11379916d31: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912771::DEBUG::2013-11-20 02:43:34,504::task::495::TaskManager.Task::(_debug) Task 2f4ea8c3-89e6-41d7-b798-c11379916d31: moving from state preparing -> state finished >Thread-15912771::DEBUG::2013-11-20 02:43:34,505::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912771::DEBUG::2013-11-20 02:43:34,508::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912771::DEBUG::2013-11-20 02:43:34,510::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912771::DEBUG::2013-11-20 02:43:34,513::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912771::DEBUG::2013-11-20 02:43:34,514::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912771::DEBUG::2013-11-20 02:43:34,515::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912771::DEBUG::2013-11-20 02:43:34,517::task::495::TaskManager.Task::(_debug) Task 2f4ea8c3-89e6-41d7-b798-c11379916d31: ref 0 aborting False >Thread-15912771::INFO::2013-11-20 02:43:34,518::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:43:34,548::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.112431 s, 9.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:43:36,557::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:37,118::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.076404 s, 13.4 MB/s\n'; <rc> = 0 >Thread-42840::DEBUG::2013-11-20 02:43:37,803::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 02:43:38,456::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:43:39,134::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:39,649::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0737767 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:43:41,664::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:42,197::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0866151 s, 11.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:43:44,212::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:44,743::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.092527 s, 11.1 MB/s\n'; <rc> = 0 >Thread-15912776::DEBUG::2013-11-20 02:43:45,885::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912776::INFO::2013-11-20 02:43:45,886::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912776::DEBUG::2013-11-20 02:43:45,886::task::495::TaskManager.Task::(_debug) Task 6deb5dbb-89fc-4472-a4fa-312a5b0b7687: moving from state init -> state preparing >Thread-15912776::DEBUG::2013-11-20 02:43:45,887::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912776::DEBUG::2013-11-20 02:43:45,888::task::495::TaskManager.Task::(_debug) Task 6deb5dbb-89fc-4472-a4fa-312a5b0b7687: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912776::DEBUG::2013-11-20 02:43:45,888::task::495::TaskManager.Task::(_debug) Task 6deb5dbb-89fc-4472-a4fa-312a5b0b7687: moving from state preparing -> state finished >Thread-15912776::DEBUG::2013-11-20 02:43:45,889::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912776::DEBUG::2013-11-20 02:43:45,889::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912776::DEBUG::2013-11-20 02:43:45,889::task::495::TaskManager.Task::(_debug) Task 6deb5dbb-89fc-4472-a4fa-312a5b0b7687: ref 0 aborting False >Thread-15912776::INFO::2013-11-20 02:43:45,890::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912777::DEBUG::2013-11-20 02:43:45,894::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912777::INFO::2013-11-20 02:43:45,895::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912777::DEBUG::2013-11-20 02:43:45,896::task::495::TaskManager.Task::(_debug) Task 1cf633c4-41f5-46e3-bc12-80a24f9c0904: moving from state init -> state preparing >Thread-15912777::DEBUG::2013-11-20 02:43:45,897::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3911d822-6a24-4798-9ae5-913598813732`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912777::DEBUG::2013-11-20 02:43:45,897::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912777::DEBUG::2013-11-20 02:43:45,898::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912777::DEBUG::2013-11-20 02:43:45,898::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3911d822-6a24-4798-9ae5-913598813732`::Granted request >Thread-15912777::DEBUG::2013-11-20 02:43:45,899::task::495::TaskManager.Task::(_debug) Task 1cf633c4-41f5-46e3-bc12-80a24f9c0904: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912777::DEBUG::2013-11-20 02:43:45,899::task::495::TaskManager.Task::(_debug) Task 1cf633c4-41f5-46e3-bc12-80a24f9c0904: ref 1 aborting False >Thread-15912777::WARNING::2013-11-20 02:43:45,901::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912777::DEBUG::2013-11-20 02:43:45,902::task::495::TaskManager.Task::(_debug) Task 1cf633c4-41f5-46e3-bc12-80a24f9c0904: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115764887552', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115764887552', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912777::DEBUG::2013-11-20 02:43:45,902::task::495::TaskManager.Task::(_debug) Task 1cf633c4-41f5-46e3-bc12-80a24f9c0904: moving from state preparing -> state finished >Thread-15912777::DEBUG::2013-11-20 02:43:45,903::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912777::DEBUG::2013-11-20 02:43:45,903::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912777::DEBUG::2013-11-20 02:43:45,904::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912777::DEBUG::2013-11-20 02:43:45,904::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912777::DEBUG::2013-11-20 02:43:45,905::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912777::DEBUG::2013-11-20 02:43:45,905::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912777::DEBUG::2013-11-20 02:43:45,906::task::495::TaskManager.Task::(_debug) Task 1cf633c4-41f5-46e3-bc12-80a24f9c0904: ref 0 aborting False >Thread-15912777::INFO::2013-11-20 02:43:45,906::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115764887552', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115764887552', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:43:46,759::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:47,287::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0770389 s, 13.3 MB/s\n'; <rc> = 0 >Thread-15912778::INFO::2013-11-20 02:43:48,133::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912778::DEBUG::2013-11-20 02:43:48,134::task::495::TaskManager.Task::(_debug) Task 49536ba2-14ef-4c16-b530-70c8e5e29376: moving from state init -> state preparing >Thread-15912778::DEBUG::2013-11-20 02:43:48,134::task::495::TaskManager.Task::(_debug) Task 49536ba2-14ef-4c16-b530-70c8e5e29376: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000349044799805', 'lastCheck': 1384915419.8084481, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0014169216156', 'lastCheck': 1384915419.357512, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00179100036621', 'lastCheck': 1384915420.0153861, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00040602684021', 'lastCheck': 1384915418.458756, 'valid': True, 'code': 0}} >Thread-15912778::DEBUG::2013-11-20 02:43:48,135::task::495::TaskManager.Task::(_debug) Task 49536ba2-14ef-4c16-b530-70c8e5e29376: moving from state preparing -> state finished >Thread-15912778::DEBUG::2013-11-20 02:43:48,136::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912778::DEBUG::2013-11-20 02:43:48,136::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912778::DEBUG::2013-11-20 02:43:48,136::task::495::TaskManager.Task::(_debug) Task 49536ba2-14ef-4c16-b530-70c8e5e29376: ref 0 aborting False >Thread-15912778::INFO::2013-11-20 02:43:48,137::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000349044799805', 'lastCheck': 1384915419.8084481, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0014169216156', 'lastCheck': 1384915419.357512, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00179100036621', 'lastCheck': 1384915420.0153861, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00040602684021', 'lastCheck': 1384915418.458756, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:43:49,302::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:49,831::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0743811 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:43:51,846::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:52,350::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0661062 s, 15.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:43:54,365::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:54,879::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0539989 s, 19.0 MB/s\n'; <rc> = 0 >Thread-15912782::DEBUG::2013-11-20 02:43:56,620::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912782::INFO::2013-11-20 02:43:56,624::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912782::DEBUG::2013-11-20 02:43:56,625::task::495::TaskManager.Task::(_debug) Task c670da17-46e8-41e0-8695-8e76f5306fe6: moving from state init -> state preparing >Thread-15912782::DEBUG::2013-11-20 02:43:56,629::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912782::DEBUG::2013-11-20 02:43:56,634::task::495::TaskManager.Task::(_debug) Task c670da17-46e8-41e0-8695-8e76f5306fe6: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912782::DEBUG::2013-11-20 02:43:56,636::task::495::TaskManager.Task::(_debug) Task c670da17-46e8-41e0-8695-8e76f5306fe6: moving from state preparing -> state finished >Thread-15912782::DEBUG::2013-11-20 02:43:56,638::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912782::DEBUG::2013-11-20 02:43:56,640::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912782::DEBUG::2013-11-20 02:43:56,642::task::495::TaskManager.Task::(_debug) Task c670da17-46e8-41e0-8695-8e76f5306fe6: ref 0 aborting False >Thread-15912782::INFO::2013-11-20 02:43:56,643::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912783::DEBUG::2013-11-20 02:43:56,650::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912783::INFO::2013-11-20 02:43:56,651::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912783::DEBUG::2013-11-20 02:43:56,651::task::495::TaskManager.Task::(_debug) Task c311c567-9daa-4334-8290-7d48f1088e25: moving from state init -> state preparing >Thread-15912783::DEBUG::2013-11-20 02:43:56,652::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`868aeb49-5b22-47d1-a185-e152c1e77b94`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912783::DEBUG::2013-11-20 02:43:56,653::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912783::DEBUG::2013-11-20 02:43:56,654::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912783::DEBUG::2013-11-20 02:43:56,654::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`868aeb49-5b22-47d1-a185-e152c1e77b94`::Granted request >Thread-15912783::DEBUG::2013-11-20 02:43:56,655::task::495::TaskManager.Task::(_debug) Task c311c567-9daa-4334-8290-7d48f1088e25: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912783::DEBUG::2013-11-20 02:43:56,656::task::495::TaskManager.Task::(_debug) Task c311c567-9daa-4334-8290-7d48f1088e25: ref 1 aborting False >Thread-15912783::WARNING::2013-11-20 02:43:56,658::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912783::DEBUG::2013-11-20 02:43:56,659::task::495::TaskManager.Task::(_debug) Task c311c567-9daa-4334-8290-7d48f1088e25: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115764887552', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115764887552', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912783::DEBUG::2013-11-20 02:43:56,660::task::495::TaskManager.Task::(_debug) Task c311c567-9daa-4334-8290-7d48f1088e25: moving from state preparing -> state finished >Thread-15912783::DEBUG::2013-11-20 02:43:56,660::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912783::DEBUG::2013-11-20 02:43:56,661::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912783::DEBUG::2013-11-20 02:43:56,661::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912783::DEBUG::2013-11-20 02:43:56,662::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912783::DEBUG::2013-11-20 02:43:56,662::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912783::DEBUG::2013-11-20 02:43:56,663::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912783::DEBUG::2013-11-20 02:43:56,664::task::495::TaskManager.Task::(_debug) Task c311c567-9daa-4334-8290-7d48f1088e25: ref 0 aborting False >Thread-15912783::INFO::2013-11-20 02:43:56,664::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115764887552', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115764887552', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:43:56,891::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:57,372::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.070034 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:43:59,387::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:43:59,929::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.070444 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:44:01,943::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 02:44:02,144::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:44:02,146::task::495::TaskManager.Task::(_debug) Task 806d697c-65b7-4090-9327-1435734d92ff: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:44:02,149::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`3dcdce6b-d67c-42f1-97f7-0e69fefa57b9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:44:02,149::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:44:02,151::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:44:02,151::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`3dcdce6b-d67c-42f1-97f7-0e69fefa57b9`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:44:02,152::task::495::TaskManager.Task::(_debug) Task 806d697c-65b7-4090-9327-1435734d92ff: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:44:02,153::task::495::TaskManager.Task::(_debug) Task 806d697c-65b7-4090-9327-1435734d92ff: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:44:02,154::task::495::TaskManager.Task::(_debug) Task 806d697c-65b7-4090-9327-1435734d92ff: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:44:02,154::task::495::TaskManager.Task::(_debug) Task 806d697c-65b7-4090-9327-1435734d92ff: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:44:02,155::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:44:02,155::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:44:02,156::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:44:02,157::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:44:02,157::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:44:02,158::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:44:02,158::task::495::TaskManager.Task::(_debug) Task 806d697c-65b7-4090-9327-1435734d92ff: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:44:02,159::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 02:44:02,468::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0703908 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15912786::INFO::2013-11-20 02:44:04,170::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912786::DEBUG::2013-11-20 02:44:04,171::task::495::TaskManager.Task::(_debug) Task 0475393c-3481-46a6-a15c-f6a6f13056c5: moving from state init -> state preparing >Thread-15912786::DEBUG::2013-11-20 02:44:04,171::task::495::TaskManager.Task::(_debug) Task 0475393c-3481-46a6-a15c-f6a6f13056c5: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000381946563721', 'lastCheck': 1384915440.0831151, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0015721321106', 'lastCheck': 1384915439.6194639, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00213003158569', 'lastCheck': 1384915440.0817189, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000475883483887', 'lastCheck': 1384915438.5876279, 'valid': True, 'code': 0}} >Thread-15912786::DEBUG::2013-11-20 02:44:04,172::task::495::TaskManager.Task::(_debug) Task 0475393c-3481-46a6-a15c-f6a6f13056c5: moving from state preparing -> state finished >Thread-15912786::DEBUG::2013-11-20 02:44:04,173::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912786::DEBUG::2013-11-20 02:44:04,173::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912786::DEBUG::2013-11-20 02:44:04,174::task::495::TaskManager.Task::(_debug) Task 0475393c-3481-46a6-a15c-f6a6f13056c5: ref 0 aborting False >Thread-15912786::INFO::2013-11-20 02:44:04,174::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000381946563721', 'lastCheck': 1384915440.0831151, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0015721321106', 'lastCheck': 1384915439.6194639, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00213003158569', 'lastCheck': 1384915440.0817189, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000475883483887', 'lastCheck': 1384915438.5876279, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:44:04,483::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:44:05,040::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.102558 s, 10.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:44:07,055::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:44:07,577::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0766399 s, 13.4 MB/s\n'; <rc> = 0 >Thread-15912788::DEBUG::2013-11-20 02:44:09,008::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912788::INFO::2013-11-20 02:44:09,010::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912788::DEBUG::2013-11-20 02:44:09,010::task::495::TaskManager.Task::(_debug) Task 76983ea6-c98f-493e-9570-b783d4c544fb: moving from state init -> state preparing >Thread-15912788::DEBUG::2013-11-20 02:44:09,011::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912788::DEBUG::2013-11-20 02:44:09,012::task::495::TaskManager.Task::(_debug) Task 76983ea6-c98f-493e-9570-b783d4c544fb: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912788::DEBUG::2013-11-20 02:44:09,012::task::495::TaskManager.Task::(_debug) Task 76983ea6-c98f-493e-9570-b783d4c544fb: moving from state preparing -> state finished >Thread-15912788::DEBUG::2013-11-20 02:44:09,013::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912788::DEBUG::2013-11-20 02:44:09,013::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912788::DEBUG::2013-11-20 02:44:09,014::task::495::TaskManager.Task::(_debug) Task 76983ea6-c98f-493e-9570-b783d4c544fb: ref 0 aborting False >Thread-15912788::INFO::2013-11-20 02:44:09,015::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912789::DEBUG::2013-11-20 02:44:09,019::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912789::INFO::2013-11-20 02:44:09,019::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912789::DEBUG::2013-11-20 02:44:09,020::task::495::TaskManager.Task::(_debug) Task 244815f9-8550-4948-86eb-2208b92e88e5: moving from state init -> state preparing >Thread-15912789::DEBUG::2013-11-20 02:44:09,021::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b4bceb6d-8890-46be-a78d-a3abb7d856c6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912789::DEBUG::2013-11-20 02:44:09,022::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912789::DEBUG::2013-11-20 02:44:09,023::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912789::DEBUG::2013-11-20 02:44:09,023::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b4bceb6d-8890-46be-a78d-a3abb7d856c6`::Granted request >Thread-15912789::DEBUG::2013-11-20 02:44:09,024::task::495::TaskManager.Task::(_debug) Task 244815f9-8550-4948-86eb-2208b92e88e5: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912789::DEBUG::2013-11-20 02:44:09,025::task::495::TaskManager.Task::(_debug) Task 244815f9-8550-4948-86eb-2208b92e88e5: ref 1 aborting False >Thread-15912789::WARNING::2013-11-20 02:44:09,027::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912789::DEBUG::2013-11-20 02:44:09,027::task::495::TaskManager.Task::(_debug) Task 244815f9-8550-4948-86eb-2208b92e88e5: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912789::DEBUG::2013-11-20 02:44:09,028::task::495::TaskManager.Task::(_debug) Task 244815f9-8550-4948-86eb-2208b92e88e5: moving from state preparing -> state finished >Thread-15912789::DEBUG::2013-11-20 02:44:09,029::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912789::DEBUG::2013-11-20 02:44:09,029::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912789::DEBUG::2013-11-20 02:44:09,030::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912789::DEBUG::2013-11-20 02:44:09,031::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912789::DEBUG::2013-11-20 02:44:09,031::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912789::DEBUG::2013-11-20 02:44:09,032::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912789::DEBUG::2013-11-20 02:44:09,032::task::495::TaskManager.Task::(_debug) Task 244815f9-8550-4948-86eb-2208b92e88e5: ref 0 aborting False >Thread-15912789::INFO::2013-11-20 02:44:09,033::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:44:09,592::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:44:10,023::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0469762 s, 21.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:44:12,036::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:44:12,568::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0739549 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:44:14,583::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:44:15,148::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.103527 s, 9.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:44:17,170::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:44:17,709::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0709951 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15912794::DEBUG::2013-11-20 02:44:19,611::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912794::INFO::2013-11-20 02:44:19,612::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912794::DEBUG::2013-11-20 02:44:19,613::task::495::TaskManager.Task::(_debug) Task b5d7058a-b206-4b88-b089-1b1267bbc317: moving from state init -> state preparing >Thread-15912794::DEBUG::2013-11-20 02:44:19,614::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912794::DEBUG::2013-11-20 02:44:19,614::task::495::TaskManager.Task::(_debug) Task b5d7058a-b206-4b88-b089-1b1267bbc317: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912794::DEBUG::2013-11-20 02:44:19,615::task::495::TaskManager.Task::(_debug) Task b5d7058a-b206-4b88-b089-1b1267bbc317: moving from state preparing -> state finished >Thread-15912794::DEBUG::2013-11-20 02:44:19,615::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912794::DEBUG::2013-11-20 02:44:19,616::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912794::DEBUG::2013-11-20 02:44:19,616::task::495::TaskManager.Task::(_debug) Task b5d7058a-b206-4b88-b089-1b1267bbc317: ref 0 aborting False >Thread-15912794::INFO::2013-11-20 02:44:19,617::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912795::DEBUG::2013-11-20 02:44:19,621::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912795::INFO::2013-11-20 02:44:19,621::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912795::DEBUG::2013-11-20 02:44:19,622::task::495::TaskManager.Task::(_debug) Task d70953cd-d9b7-48b6-880f-2487bb2377f9: moving from state init -> state preparing >Thread-15912795::DEBUG::2013-11-20 02:44:19,623::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c087b8d5-ca5b-4b83-b877-3f21e7b4aa20`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912795::DEBUG::2013-11-20 02:44:19,624::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912795::DEBUG::2013-11-20 02:44:19,625::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912795::DEBUG::2013-11-20 02:44:19,625::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c087b8d5-ca5b-4b83-b877-3f21e7b4aa20`::Granted request >Thread-15912795::DEBUG::2013-11-20 02:44:19,626::task::495::TaskManager.Task::(_debug) Task d70953cd-d9b7-48b6-880f-2487bb2377f9: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912795::DEBUG::2013-11-20 02:44:19,627::task::495::TaskManager.Task::(_debug) Task d70953cd-d9b7-48b6-880f-2487bb2377f9: ref 1 aborting False >Thread-15912795::WARNING::2013-11-20 02:44:19,629::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912795::DEBUG::2013-11-20 02:44:19,629::task::495::TaskManager.Task::(_debug) Task d70953cd-d9b7-48b6-880f-2487bb2377f9: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912795::DEBUG::2013-11-20 02:44:19,630::task::495::TaskManager.Task::(_debug) Task d70953cd-d9b7-48b6-880f-2487bb2377f9: moving from state preparing -> state finished >Thread-15912795::DEBUG::2013-11-20 02:44:19,631::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912795::DEBUG::2013-11-20 02:44:19,631::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912795::DEBUG::2013-11-20 02:44:19,632::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912795::DEBUG::2013-11-20 02:44:19,633::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912795::DEBUG::2013-11-20 02:44:19,633::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912795::DEBUG::2013-11-20 02:44:19,634::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912795::DEBUG::2013-11-20 02:44:19,634::task::495::TaskManager.Task::(_debug) Task d70953cd-d9b7-48b6-880f-2487bb2377f9: ref 0 aborting False >Thread-15912795::INFO::2013-11-20 02:44:19,635::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:44:19,725::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:44:20,142::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0425894 s, 24.0 MB/s\n'; <rc> = 0 >Thread-15912796::INFO::2013-11-20 02:44:21,887::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912796::DEBUG::2013-11-20 02:44:21,888::task::495::TaskManager.Task::(_debug) Task 455ef442-6356-4bad-adb5-6e895eed6fdb: moving from state init -> state preparing >Thread-15912796::DEBUG::2013-11-20 02:44:21,889::task::495::TaskManager.Task::(_debug) Task 455ef442-6356-4bad-adb5-6e895eed6fdb: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000364065170288', 'lastCheck': 1384915460.4077201, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00102591514587', 'lastCheck': 1384915459.9261141, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00135397911072', 'lastCheck': 1384915460.409332, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000445127487183', 'lastCheck': 1384915458.81552, 'valid': True, 'code': 0}} >Thread-15912796::DEBUG::2013-11-20 02:44:21,890::task::495::TaskManager.Task::(_debug) Task 455ef442-6356-4bad-adb5-6e895eed6fdb: moving from state preparing -> state finished >Thread-15912796::DEBUG::2013-11-20 02:44:21,890::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912796::DEBUG::2013-11-20 02:44:21,891::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912796::DEBUG::2013-11-20 02:44:21,891::task::495::TaskManager.Task::(_debug) Task 455ef442-6356-4bad-adb5-6e895eed6fdb: ref 0 aborting False >Thread-15912796::INFO::2013-11-20 02:44:21,892::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000364065170288', 'lastCheck': 1384915460.4077201, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00102591514587', 'lastCheck': 1384915459.9261141, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00135397911072', 'lastCheck': 1384915460.409332, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000445127487183', 'lastCheck': 1384915458.81552, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:44:22,156::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:44:22,687::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0671611 s, 15.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:44:24,703::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:44:25,142::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0433918 s, 23.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:44:27,152::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:44:27,570::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0529798 s, 19.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:44:29,584::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:44:30,101::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0736466 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15912800::DEBUG::2013-11-20 02:44:30,123::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912800::INFO::2013-11-20 02:44:30,125::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912800::DEBUG::2013-11-20 02:44:30,125::task::495::TaskManager.Task::(_debug) Task d1406a35-6c0e-4b84-a917-12137c414c24: moving from state init -> state preparing >Thread-15912800::DEBUG::2013-11-20 02:44:30,126::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912800::DEBUG::2013-11-20 02:44:30,127::task::495::TaskManager.Task::(_debug) Task d1406a35-6c0e-4b84-a917-12137c414c24: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912800::DEBUG::2013-11-20 02:44:30,127::task::495::TaskManager.Task::(_debug) Task d1406a35-6c0e-4b84-a917-12137c414c24: moving from state preparing -> state finished >Thread-15912800::DEBUG::2013-11-20 02:44:30,128::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912800::DEBUG::2013-11-20 02:44:30,128::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912800::DEBUG::2013-11-20 02:44:30,129::task::495::TaskManager.Task::(_debug) Task d1406a35-6c0e-4b84-a917-12137c414c24: ref 0 aborting False >Thread-15912800::INFO::2013-11-20 02:44:30,129::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912801::DEBUG::2013-11-20 02:44:30,134::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912801::INFO::2013-11-20 02:44:30,135::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912801::DEBUG::2013-11-20 02:44:30,135::task::495::TaskManager.Task::(_debug) Task 1d16eced-e816-4618-9b88-1ff6dd810b8c: moving from state init -> state preparing >Thread-15912801::DEBUG::2013-11-20 02:44:30,136::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5392d610-c7a1-48da-a643-098270325696`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912801::DEBUG::2013-11-20 02:44:30,137::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912801::DEBUG::2013-11-20 02:44:30,138::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912801::DEBUG::2013-11-20 02:44:30,138::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5392d610-c7a1-48da-a643-098270325696`::Granted request >Thread-15912801::DEBUG::2013-11-20 02:44:30,139::task::495::TaskManager.Task::(_debug) Task 1d16eced-e816-4618-9b88-1ff6dd810b8c: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912801::DEBUG::2013-11-20 02:44:30,140::task::495::TaskManager.Task::(_debug) Task 1d16eced-e816-4618-9b88-1ff6dd810b8c: ref 1 aborting False >Thread-15912801::WARNING::2013-11-20 02:44:30,142::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912801::DEBUG::2013-11-20 02:44:30,143::task::495::TaskManager.Task::(_debug) Task 1d16eced-e816-4618-9b88-1ff6dd810b8c: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912801::DEBUG::2013-11-20 02:44:30,143::task::495::TaskManager.Task::(_debug) Task 1d16eced-e816-4618-9b88-1ff6dd810b8c: moving from state preparing -> state finished >Thread-15912801::DEBUG::2013-11-20 02:44:30,144::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912801::DEBUG::2013-11-20 02:44:30,145::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912801::DEBUG::2013-11-20 02:44:30,145::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912801::DEBUG::2013-11-20 02:44:30,146::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912801::DEBUG::2013-11-20 02:44:30,146::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912801::DEBUG::2013-11-20 02:44:30,147::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912801::DEBUG::2013-11-20 02:44:30,147::task::495::TaskManager.Task::(_debug) Task 1d16eced-e816-4618-9b88-1ff6dd810b8c: ref 0 aborting False >Thread-15912801::INFO::2013-11-20 02:44:30,148::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:44:32,116::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:44:32,639::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0728841 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:44:34,654::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:44:35,168::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.05375 s, 19.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:44:37,180::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:44:37,708::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0741235 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15912804::INFO::2013-11-20 02:44:38,487::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912804::DEBUG::2013-11-20 02:44:38,489::task::495::TaskManager.Task::(_debug) Task fff36ac9-b2af-4d4b-8f36-d356f2e43e84: moving from state init -> state preparing >Thread-15912804::DEBUG::2013-11-20 02:44:38,489::task::495::TaskManager.Task::(_debug) Task fff36ac9-b2af-4d4b-8f36-d356f2e43e84: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000385999679565', 'lastCheck': 1384915470.421278, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0140869617462', 'lastCheck': 1384915469.954282, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00330495834351', 'lastCheck': 1384915470.4252591, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000514030456543', 'lastCheck': 1384915468.930141, 'valid': True, 'code': 0}} >Thread-15912804::DEBUG::2013-11-20 02:44:38,490::task::495::TaskManager.Task::(_debug) Task fff36ac9-b2af-4d4b-8f36-d356f2e43e84: moving from state preparing -> state finished >Thread-15912804::DEBUG::2013-11-20 02:44:38,491::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912804::DEBUG::2013-11-20 02:44:38,491::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912804::DEBUG::2013-11-20 02:44:38,492::task::495::TaskManager.Task::(_debug) Task fff36ac9-b2af-4d4b-8f36-d356f2e43e84: ref 0 aborting False >Thread-15912804::INFO::2013-11-20 02:44:38,493::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000385999679565', 'lastCheck': 1384915470.421278, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0140869617462', 'lastCheck': 1384915469.954282, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00330495834351', 'lastCheck': 1384915470.4252591, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000514030456543', 'lastCheck': 1384915468.930141, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:44:39,723::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:44:40,228::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0614619 s, 16.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:44:42,241::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912806::DEBUG::2013-11-20 02:44:42,521::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912806::INFO::2013-11-20 02:44:42,523::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912806::DEBUG::2013-11-20 02:44:42,523::task::495::TaskManager.Task::(_debug) Task a10de214-05f8-4754-9e41-2684e3418bbe: moving from state init -> state preparing >Thread-15912806::DEBUG::2013-11-20 02:44:42,525::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912806::DEBUG::2013-11-20 02:44:42,525::task::495::TaskManager.Task::(_debug) Task a10de214-05f8-4754-9e41-2684e3418bbe: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912806::DEBUG::2013-11-20 02:44:42,526::task::495::TaskManager.Task::(_debug) Task a10de214-05f8-4754-9e41-2684e3418bbe: moving from state preparing -> state finished >Thread-15912806::DEBUG::2013-11-20 02:44:42,526::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912806::DEBUG::2013-11-20 02:44:42,527::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912806::DEBUG::2013-11-20 02:44:42,527::task::495::TaskManager.Task::(_debug) Task a10de214-05f8-4754-9e41-2684e3418bbe: ref 0 aborting False >Thread-15912806::INFO::2013-11-20 02:44:42,528::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912807::DEBUG::2013-11-20 02:44:42,534::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912807::INFO::2013-11-20 02:44:42,535::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912807::DEBUG::2013-11-20 02:44:42,535::task::495::TaskManager.Task::(_debug) Task 2d1a5657-268b-4fc0-a4e4-6b9e07ca9baf: moving from state init -> state preparing >Thread-15912807::DEBUG::2013-11-20 02:44:42,537::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bf095424-05d9-4d2b-8947-f40842145c87`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912807::DEBUG::2013-11-20 02:44:42,537::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912807::DEBUG::2013-11-20 02:44:42,538::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912807::DEBUG::2013-11-20 02:44:42,539::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bf095424-05d9-4d2b-8947-f40842145c87`::Granted request >Thread-15912807::DEBUG::2013-11-20 02:44:42,540::task::495::TaskManager.Task::(_debug) Task 2d1a5657-268b-4fc0-a4e4-6b9e07ca9baf: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912807::DEBUG::2013-11-20 02:44:42,540::task::495::TaskManager.Task::(_debug) Task 2d1a5657-268b-4fc0-a4e4-6b9e07ca9baf: ref 1 aborting False >Thread-15912807::WARNING::2013-11-20 02:44:42,543::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912807::DEBUG::2013-11-20 02:44:42,544::task::495::TaskManager.Task::(_debug) Task 2d1a5657-268b-4fc0-a4e4-6b9e07ca9baf: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912807::DEBUG::2013-11-20 02:44:42,545::task::495::TaskManager.Task::(_debug) Task 2d1a5657-268b-4fc0-a4e4-6b9e07ca9baf: moving from state preparing -> state finished >Thread-15912807::DEBUG::2013-11-20 02:44:42,545::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912807::DEBUG::2013-11-20 02:44:42,546::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912807::DEBUG::2013-11-20 02:44:42,547::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912807::DEBUG::2013-11-20 02:44:42,547::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912807::DEBUG::2013-11-20 02:44:42,548::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912807::DEBUG::2013-11-20 02:44:42,548::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912807::DEBUG::2013-11-20 02:44:42,549::task::495::TaskManager.Task::(_debug) Task 2d1a5657-268b-4fc0-a4e4-6b9e07ca9baf: ref 0 aborting False >Thread-15912807::INFO::2013-11-20 02:44:42,550::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:44:42,777::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0701455 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:44:44,792::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:44:45,336::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0703028 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:44:47,350::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:44:47,859::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0696081 s, 14.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:44:49,873::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:44:50,399::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0743392 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:44:52,414::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:44:52,908::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0543772 s, 18.8 MB/s\n'; <rc> = 0 >Thread-15912812::DEBUG::2013-11-20 02:44:54,646::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912812::INFO::2013-11-20 02:44:54,647::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912812::DEBUG::2013-11-20 02:44:54,648::task::495::TaskManager.Task::(_debug) Task 8ac5f586-60d3-458f-bd99-fe9a63dea8d1: moving from state init -> state preparing >Thread-15912812::DEBUG::2013-11-20 02:44:54,649::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912812::DEBUG::2013-11-20 02:44:54,649::task::495::TaskManager.Task::(_debug) Task 8ac5f586-60d3-458f-bd99-fe9a63dea8d1: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912812::DEBUG::2013-11-20 02:44:54,650::task::495::TaskManager.Task::(_debug) Task 8ac5f586-60d3-458f-bd99-fe9a63dea8d1: moving from state preparing -> state finished >Thread-15912812::DEBUG::2013-11-20 02:44:54,650::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912812::DEBUG::2013-11-20 02:44:54,651::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912812::DEBUG::2013-11-20 02:44:54,651::task::495::TaskManager.Task::(_debug) Task 8ac5f586-60d3-458f-bd99-fe9a63dea8d1: ref 0 aborting False >Thread-15912812::INFO::2013-11-20 02:44:54,652::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912813::DEBUG::2013-11-20 02:44:54,656::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912813::INFO::2013-11-20 02:44:54,657::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912813::DEBUG::2013-11-20 02:44:54,658::task::495::TaskManager.Task::(_debug) Task 15661c7e-0cd9-460d-94f3-71e7faf16f77: moving from state init -> state preparing >Thread-15912813::DEBUG::2013-11-20 02:44:54,659::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`8f162de4-5dc9-4ba8-adcb-1387e45aa826`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912813::DEBUG::2013-11-20 02:44:54,660::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912813::DEBUG::2013-11-20 02:44:54,660::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912813::DEBUG::2013-11-20 02:44:54,661::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`8f162de4-5dc9-4ba8-adcb-1387e45aa826`::Granted request >Thread-15912813::DEBUG::2013-11-20 02:44:54,662::task::495::TaskManager.Task::(_debug) Task 15661c7e-0cd9-460d-94f3-71e7faf16f77: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912813::DEBUG::2013-11-20 02:44:54,662::task::495::TaskManager.Task::(_debug) Task 15661c7e-0cd9-460d-94f3-71e7faf16f77: ref 1 aborting False >Thread-15912813::WARNING::2013-11-20 02:44:54,664::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912813::DEBUG::2013-11-20 02:44:54,665::task::495::TaskManager.Task::(_debug) Task 15661c7e-0cd9-460d-94f3-71e7faf16f77: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912813::DEBUG::2013-11-20 02:44:54,666::task::495::TaskManager.Task::(_debug) Task 15661c7e-0cd9-460d-94f3-71e7faf16f77: moving from state preparing -> state finished >Thread-15912813::DEBUG::2013-11-20 02:44:54,666::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912813::DEBUG::2013-11-20 02:44:54,667::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912813::DEBUG::2013-11-20 02:44:54,668::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912813::DEBUG::2013-11-20 02:44:54,668::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912813::DEBUG::2013-11-20 02:44:54,669::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912813::DEBUG::2013-11-20 02:44:54,669::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912813::DEBUG::2013-11-20 02:44:54,670::task::495::TaskManager.Task::(_debug) Task 15661c7e-0cd9-460d-94f3-71e7faf16f77: ref 0 aborting False >Thread-15912813::INFO::2013-11-20 02:44:54,671::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:44:54,918::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:44:55,367::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0484985 s, 21.1 MB/s\n'; <rc> = 0 >Thread-15912814::INFO::2013-11-20 02:44:55,419::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912814::DEBUG::2013-11-20 02:44:55,419::task::495::TaskManager.Task::(_debug) Task ebf84b1c-dbec-4378-bc66-2d9ca93884d7: moving from state init -> state preparing >Thread-15912814::DEBUG::2013-11-20 02:44:55,420::task::495::TaskManager.Task::(_debug) Task ebf84b1c-dbec-4378-bc66-2d9ca93884d7: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00062108039856', 'lastCheck': 1384915490.5634999, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00124907493591', 'lastCheck': 1384915490.1269619, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00130486488342', 'lastCheck': 1384915490.5643971, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000457048416138', 'lastCheck': 1384915489.0234511, 'valid': True, 'code': 0}} >Thread-15912814::DEBUG::2013-11-20 02:44:55,421::task::495::TaskManager.Task::(_debug) Task ebf84b1c-dbec-4378-bc66-2d9ca93884d7: moving from state preparing -> state finished >Thread-15912814::DEBUG::2013-11-20 02:44:55,422::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912814::DEBUG::2013-11-20 02:44:55,422::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912814::DEBUG::2013-11-20 02:44:55,423::task::495::TaskManager.Task::(_debug) Task ebf84b1c-dbec-4378-bc66-2d9ca93884d7: ref 0 aborting False >Thread-15912814::INFO::2013-11-20 02:44:55,423::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00062108039856', 'lastCheck': 1384915490.5634999, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00124907493591', 'lastCheck': 1384915490.1269619, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00130486488342', 'lastCheck': 1384915490.5643971, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000457048416138', 'lastCheck': 1384915489.0234511, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:44:57,376::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:44:57,849::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0445948 s, 23.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:44:59,862::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:45:00,361::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0698657 s, 14.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:45:02,376::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 02:45:02,653::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:45:02,654::task::495::TaskManager.Task::(_debug) Task fd65ae80-5fe9-43ed-b4c0-5fd3a4351d68: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:45:02,657::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`99a75870-310e-4ce9-8800-24783d13817c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:45:02,657::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:45:02,658::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:45:02,659::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`99a75870-310e-4ce9-8800-24783d13817c`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:45:02,660::task::495::TaskManager.Task::(_debug) Task fd65ae80-5fe9-43ed-b4c0-5fd3a4351d68: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:45:02,661::task::495::TaskManager.Task::(_debug) Task fd65ae80-5fe9-43ed-b4c0-5fd3a4351d68: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:45:02,661::task::495::TaskManager.Task::(_debug) Task fd65ae80-5fe9-43ed-b4c0-5fd3a4351d68: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:45:02,662::task::495::TaskManager.Task::(_debug) Task fd65ae80-5fe9-43ed-b4c0-5fd3a4351d68: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:45:02,663::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:45:02,663::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:45:02,664::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:45:02,665::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:45:02,665::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:45:02,666::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:45:02,666::task::495::TaskManager.Task::(_debug) Task fd65ae80-5fe9-43ed-b4c0-5fd3a4351d68: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:45:02,667::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 02:45:02,915::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.066473 s, 15.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:45:04,925::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:45:05,416::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.054498 s, 18.8 MB/s\n'; <rc> = 0 >Thread-15912819::DEBUG::2013-11-20 02:45:05,753::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912819::INFO::2013-11-20 02:45:05,754::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912819::DEBUG::2013-11-20 02:45:05,755::task::495::TaskManager.Task::(_debug) Task 6fe300fc-19c5-4bb6-91d5-379e67321ba0: moving from state init -> state preparing >Thread-15912819::DEBUG::2013-11-20 02:45:05,755::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912819::DEBUG::2013-11-20 02:45:05,756::task::495::TaskManager.Task::(_debug) Task 6fe300fc-19c5-4bb6-91d5-379e67321ba0: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912819::DEBUG::2013-11-20 02:45:05,756::task::495::TaskManager.Task::(_debug) Task 6fe300fc-19c5-4bb6-91d5-379e67321ba0: moving from state preparing -> state finished >Thread-15912819::DEBUG::2013-11-20 02:45:05,756::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912819::DEBUG::2013-11-20 02:45:05,757::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912819::DEBUG::2013-11-20 02:45:05,757::task::495::TaskManager.Task::(_debug) Task 6fe300fc-19c5-4bb6-91d5-379e67321ba0: ref 0 aborting False >Thread-15912819::INFO::2013-11-20 02:45:05,757::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912820::DEBUG::2013-11-20 02:45:05,762::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912820::INFO::2013-11-20 02:45:05,763::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912820::DEBUG::2013-11-20 02:45:05,763::task::495::TaskManager.Task::(_debug) Task 2ca042a2-9a17-4291-a173-b52405bdbbbc: moving from state init -> state preparing >Thread-15912820::DEBUG::2013-11-20 02:45:05,765::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a64aad69-a16b-4448-a3c3-5b4365d0c809`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912820::DEBUG::2013-11-20 02:45:05,765::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912820::DEBUG::2013-11-20 02:45:05,766::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912820::DEBUG::2013-11-20 02:45:05,766::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a64aad69-a16b-4448-a3c3-5b4365d0c809`::Granted request >Thread-15912820::DEBUG::2013-11-20 02:45:05,767::task::495::TaskManager.Task::(_debug) Task 2ca042a2-9a17-4291-a173-b52405bdbbbc: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912820::DEBUG::2013-11-20 02:45:05,768::task::495::TaskManager.Task::(_debug) Task 2ca042a2-9a17-4291-a173-b52405bdbbbc: ref 1 aborting False >Thread-15912820::WARNING::2013-11-20 02:45:05,770::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912820::DEBUG::2013-11-20 02:45:05,770::task::495::TaskManager.Task::(_debug) Task 2ca042a2-9a17-4291-a173-b52405bdbbbc: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912820::DEBUG::2013-11-20 02:45:05,771::task::495::TaskManager.Task::(_debug) Task 2ca042a2-9a17-4291-a173-b52405bdbbbc: moving from state preparing -> state finished >Thread-15912820::DEBUG::2013-11-20 02:45:05,772::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912820::DEBUG::2013-11-20 02:45:05,772::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912820::DEBUG::2013-11-20 02:45:05,773::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912820::DEBUG::2013-11-20 02:45:05,773::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912820::DEBUG::2013-11-20 02:45:05,774::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912820::DEBUG::2013-11-20 02:45:05,774::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912820::DEBUG::2013-11-20 02:45:05,775::task::495::TaskManager.Task::(_debug) Task 2ca042a2-9a17-4291-a173-b52405bdbbbc: ref 0 aborting False >Thread-15912820::INFO::2013-11-20 02:45:05,775::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:45:07,426::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:45:07,937::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0666421 s, 15.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:45:09,952::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:45:10,459::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0742397 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15912822::INFO::2013-11-20 02:45:11,241::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912822::DEBUG::2013-11-20 02:45:11,242::task::495::TaskManager.Task::(_debug) Task 4b3bb88b-8373-42ce-a762-493d268f5a8f: moving from state init -> state preparing >Thread-15912822::DEBUG::2013-11-20 02:45:11,243::task::495::TaskManager.Task::(_debug) Task 4b3bb88b-8373-42ce-a762-493d268f5a8f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000240802764893', 'lastCheck': 1384915510.6686139, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00186586380005', 'lastCheck': 1384915510.182543, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.001296043396', 'lastCheck': 1384915510.6710401, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000378847122192', 'lastCheck': 1384915509.1792741, 'valid': True, 'code': 0}} >Thread-15912822::DEBUG::2013-11-20 02:45:11,244::task::495::TaskManager.Task::(_debug) Task 4b3bb88b-8373-42ce-a762-493d268f5a8f: moving from state preparing -> state finished >Thread-15912822::DEBUG::2013-11-20 02:45:11,244::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912822::DEBUG::2013-11-20 02:45:11,245::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912822::DEBUG::2013-11-20 02:45:11,245::task::495::TaskManager.Task::(_debug) Task 4b3bb88b-8373-42ce-a762-493d268f5a8f: ref 0 aborting False >Thread-15912822::INFO::2013-11-20 02:45:11,246::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000240802764893', 'lastCheck': 1384915510.6686139, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00186586380005', 'lastCheck': 1384915510.182543, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.001296043396', 'lastCheck': 1384915510.6710401, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000378847122192', 'lastCheck': 1384915509.1792741, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:45:12,474::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:45:12,998::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0745649 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:45:15,012::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:45:15,501::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.073274 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15912825::DEBUG::2013-11-20 02:45:17,103::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912825::INFO::2013-11-20 02:45:17,104::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912825::DEBUG::2013-11-20 02:45:17,105::task::495::TaskManager.Task::(_debug) Task b6c0c97c-554f-46ac-8e3e-adc7c74f49ca: moving from state init -> state preparing >Thread-15912825::DEBUG::2013-11-20 02:45:17,106::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912825::DEBUG::2013-11-20 02:45:17,106::task::495::TaskManager.Task::(_debug) Task b6c0c97c-554f-46ac-8e3e-adc7c74f49ca: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912825::DEBUG::2013-11-20 02:45:17,107::task::495::TaskManager.Task::(_debug) Task b6c0c97c-554f-46ac-8e3e-adc7c74f49ca: moving from state preparing -> state finished >Thread-15912825::DEBUG::2013-11-20 02:45:17,108::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912825::DEBUG::2013-11-20 02:45:17,108::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912825::DEBUG::2013-11-20 02:45:17,109::task::495::TaskManager.Task::(_debug) Task b6c0c97c-554f-46ac-8e3e-adc7c74f49ca: ref 0 aborting False >Thread-15912825::INFO::2013-11-20 02:45:17,109::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912826::DEBUG::2013-11-20 02:45:17,114::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912826::INFO::2013-11-20 02:45:17,114::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912826::DEBUG::2013-11-20 02:45:17,115::task::495::TaskManager.Task::(_debug) Task 56604f22-46e8-4335-b442-5d6316512ce5: moving from state init -> state preparing >Thread-15912826::DEBUG::2013-11-20 02:45:17,116::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`8241a27d-0ded-4024-8143-033b8d086c78`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912826::DEBUG::2013-11-20 02:45:17,117::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912826::DEBUG::2013-11-20 02:45:17,118::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912826::DEBUG::2013-11-20 02:45:17,118::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`8241a27d-0ded-4024-8143-033b8d086c78`::Granted request >Thread-15912826::DEBUG::2013-11-20 02:45:17,119::task::495::TaskManager.Task::(_debug) Task 56604f22-46e8-4335-b442-5d6316512ce5: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912826::DEBUG::2013-11-20 02:45:17,120::task::495::TaskManager.Task::(_debug) Task 56604f22-46e8-4335-b442-5d6316512ce5: ref 1 aborting False >Thread-15912826::WARNING::2013-11-20 02:45:17,122::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912826::DEBUG::2013-11-20 02:45:17,122::task::495::TaskManager.Task::(_debug) Task 56604f22-46e8-4335-b442-5d6316512ce5: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912826::DEBUG::2013-11-20 02:45:17,123::task::495::TaskManager.Task::(_debug) Task 56604f22-46e8-4335-b442-5d6316512ce5: moving from state preparing -> state finished >Thread-15912826::DEBUG::2013-11-20 02:45:17,124::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912826::DEBUG::2013-11-20 02:45:17,124::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912826::DEBUG::2013-11-20 02:45:17,125::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912826::DEBUG::2013-11-20 02:45:17,126::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912826::DEBUG::2013-11-20 02:45:17,126::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912826::DEBUG::2013-11-20 02:45:17,127::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912826::DEBUG::2013-11-20 02:45:17,127::task::495::TaskManager.Task::(_debug) Task 56604f22-46e8-4335-b442-5d6316512ce5: ref 0 aborting False >Thread-15912826::INFO::2013-11-20 02:45:17,128::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:45:17,524::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:45:18,111::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0616165 s, 16.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:45:20,133::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:45:20,637::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0538786 s, 19.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:45:22,648::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:45:23,141::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0675249 s, 15.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:45:25,156::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:45:25,695::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.07556 s, 13.6 MB/s\n'; <rc> = 0 >Thread-15912830::INFO::2013-11-20 02:45:27,387::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912830::DEBUG::2013-11-20 02:45:27,388::task::495::TaskManager.Task::(_debug) Task fcc56de3-1a38-4504-9296-95c2d91b5acd: moving from state init -> state preparing >Thread-15912830::DEBUG::2013-11-20 02:45:27,389::task::495::TaskManager.Task::(_debug) Task fcc56de3-1a38-4504-9296-95c2d91b5acd: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000333070755005', 'lastCheck': 1384915520.758769, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00108504295349', 'lastCheck': 1384915520.4330201, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000983953475952', 'lastCheck': 1384915520.7590971, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000549077987671', 'lastCheck': 1384915519.200763, 'valid': True, 'code': 0}} >Thread-15912830::DEBUG::2013-11-20 02:45:27,389::task::495::TaskManager.Task::(_debug) Task fcc56de3-1a38-4504-9296-95c2d91b5acd: moving from state preparing -> state finished >Thread-15912830::DEBUG::2013-11-20 02:45:27,389::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912830::DEBUG::2013-11-20 02:45:27,390::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912830::DEBUG::2013-11-20 02:45:27,390::task::495::TaskManager.Task::(_debug) Task fcc56de3-1a38-4504-9296-95c2d91b5acd: ref 0 aborting False >Thread-15912830::INFO::2013-11-20 02:45:27,390::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000333070755005', 'lastCheck': 1384915520.758769, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00108504295349', 'lastCheck': 1384915520.4330201, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000983953475952', 'lastCheck': 1384915520.7590971, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000549077987671', 'lastCheck': 1384915519.200763, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:45:27,711::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:45:28,292::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0788565 s, 13.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:45:30,309::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912832::DEBUG::2013-11-20 02:45:30,678::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912832::INFO::2013-11-20 02:45:30,680::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912832::DEBUG::2013-11-20 02:45:30,680::task::495::TaskManager.Task::(_debug) Task 8032906d-6447-4863-8ad2-38653c12ca6a: moving from state init -> state preparing >Thread-15912832::DEBUG::2013-11-20 02:45:30,681::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912832::DEBUG::2013-11-20 02:45:30,682::task::495::TaskManager.Task::(_debug) Task 8032906d-6447-4863-8ad2-38653c12ca6a: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912832::DEBUG::2013-11-20 02:45:30,683::task::495::TaskManager.Task::(_debug) Task 8032906d-6447-4863-8ad2-38653c12ca6a: moving from state preparing -> state finished >Thread-15912832::DEBUG::2013-11-20 02:45:30,684::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912832::DEBUG::2013-11-20 02:45:30,684::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912832::DEBUG::2013-11-20 02:45:30,685::task::495::TaskManager.Task::(_debug) Task 8032906d-6447-4863-8ad2-38653c12ca6a: ref 0 aborting False >Thread-15912832::INFO::2013-11-20 02:45:30,686::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912833::DEBUG::2013-11-20 02:45:30,691::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912833::INFO::2013-11-20 02:45:30,693::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912833::DEBUG::2013-11-20 02:45:30,693::task::495::TaskManager.Task::(_debug) Task 191b9b58-ffe8-4f7f-b1f2-47cd5d630f86: moving from state init -> state preparing >Thread-15912833::DEBUG::2013-11-20 02:45:30,695::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`36bac967-ee27-4b1f-8b38-987c57194668`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912833::DEBUG::2013-11-20 02:45:30,696::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912833::DEBUG::2013-11-20 02:45:30,697::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912833::DEBUG::2013-11-20 02:45:30,698::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`36bac967-ee27-4b1f-8b38-987c57194668`::Granted request >Thread-15912833::DEBUG::2013-11-20 02:45:30,699::task::495::TaskManager.Task::(_debug) Task 191b9b58-ffe8-4f7f-b1f2-47cd5d630f86: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912833::DEBUG::2013-11-20 02:45:30,700::task::495::TaskManager.Task::(_debug) Task 191b9b58-ffe8-4f7f-b1f2-47cd5d630f86: ref 1 aborting False >Thread-15912833::WARNING::2013-11-20 02:45:30,706::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912833::DEBUG::2013-11-20 02:45:30,710::task::495::TaskManager.Task::(_debug) Task 191b9b58-ffe8-4f7f-b1f2-47cd5d630f86: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912833::DEBUG::2013-11-20 02:45:30,715::task::495::TaskManager.Task::(_debug) Task 191b9b58-ffe8-4f7f-b1f2-47cd5d630f86: moving from state preparing -> state finished >Thread-15912833::DEBUG::2013-11-20 02:45:30,716::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912833::DEBUG::2013-11-20 02:45:30,716::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912833::DEBUG::2013-11-20 02:45:30,717::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912833::DEBUG::2013-11-20 02:45:30,718::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912833::DEBUG::2013-11-20 02:45:30,718::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912833::DEBUG::2013-11-20 02:45:30,718::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912833::DEBUG::2013-11-20 02:45:30,719::task::495::TaskManager.Task::(_debug) Task 191b9b58-ffe8-4f7f-b1f2-47cd5d630f86: ref 0 aborting False >Thread-15912833::INFO::2013-11-20 02:45:30,720::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:45:30,847::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0739534 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:45:32,861::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:45:33,371::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0732209 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:45:35,387::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:45:35,830::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0467744 s, 21.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:45:37,841::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:45:38,364::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0726019 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:45:40,379::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:45:40,895::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0723493 s, 14.2 MB/s\n'; <rc> = 0 >Thread-15912837::DEBUG::2013-11-20 02:45:41,217::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912837::INFO::2013-11-20 02:45:41,219::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912837::DEBUG::2013-11-20 02:45:41,219::task::495::TaskManager.Task::(_debug) Task 971cc26c-f17f-4e4b-9bd3-6e4b07d3c897: moving from state init -> state preparing >Thread-15912837::DEBUG::2013-11-20 02:45:41,220::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912837::DEBUG::2013-11-20 02:45:41,220::task::495::TaskManager.Task::(_debug) Task 971cc26c-f17f-4e4b-9bd3-6e4b07d3c897: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912837::DEBUG::2013-11-20 02:45:41,220::task::495::TaskManager.Task::(_debug) Task 971cc26c-f17f-4e4b-9bd3-6e4b07d3c897: moving from state preparing -> state finished >Thread-15912837::DEBUG::2013-11-20 02:45:41,221::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912837::DEBUG::2013-11-20 02:45:41,221::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912837::DEBUG::2013-11-20 02:45:41,221::task::495::TaskManager.Task::(_debug) Task 971cc26c-f17f-4e4b-9bd3-6e4b07d3c897: ref 0 aborting False >Thread-15912837::INFO::2013-11-20 02:45:41,222::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912838::DEBUG::2013-11-20 02:45:41,226::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912838::INFO::2013-11-20 02:45:41,227::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912838::DEBUG::2013-11-20 02:45:41,227::task::495::TaskManager.Task::(_debug) Task 60bab87f-d26c-49ed-abd4-c76dbca39428: moving from state init -> state preparing >Thread-15912838::DEBUG::2013-11-20 02:45:41,228::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c3089f57-078b-46c4-9d64-874da9006894`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912838::DEBUG::2013-11-20 02:45:41,229::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912838::DEBUG::2013-11-20 02:45:41,229::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912838::DEBUG::2013-11-20 02:45:41,230::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c3089f57-078b-46c4-9d64-874da9006894`::Granted request >Thread-15912838::DEBUG::2013-11-20 02:45:41,230::task::495::TaskManager.Task::(_debug) Task 60bab87f-d26c-49ed-abd4-c76dbca39428: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912838::DEBUG::2013-11-20 02:45:41,231::task::495::TaskManager.Task::(_debug) Task 60bab87f-d26c-49ed-abd4-c76dbca39428: ref 1 aborting False >Thread-15912838::WARNING::2013-11-20 02:45:41,233::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912838::DEBUG::2013-11-20 02:45:41,234::task::495::TaskManager.Task::(_debug) Task 60bab87f-d26c-49ed-abd4-c76dbca39428: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912838::DEBUG::2013-11-20 02:45:41,235::task::495::TaskManager.Task::(_debug) Task 60bab87f-d26c-49ed-abd4-c76dbca39428: moving from state preparing -> state finished >Thread-15912838::DEBUG::2013-11-20 02:45:41,236::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912838::DEBUG::2013-11-20 02:45:41,236::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912838::DEBUG::2013-11-20 02:45:41,237::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912838::DEBUG::2013-11-20 02:45:41,238::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912838::DEBUG::2013-11-20 02:45:41,238::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912838::DEBUG::2013-11-20 02:45:41,239::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912838::DEBUG::2013-11-20 02:45:41,239::task::495::TaskManager.Task::(_debug) Task 60bab87f-d26c-49ed-abd4-c76dbca39428: ref 0 aborting False >Thread-15912838::INFO::2013-11-20 02:45:41,240::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:45:42,910::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:45:43,346::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0469231 s, 21.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:45:45,357::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912840::INFO::2013-11-20 02:45:45,637::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912840::DEBUG::2013-11-20 02:45:45,638::task::495::TaskManager.Task::(_debug) Task 41db289e-df1d-4ee1-9075-2386718bef6c: moving from state init -> state preparing >Thread-15912840::DEBUG::2013-11-20 02:45:45,639::task::495::TaskManager.Task::(_debug) Task 41db289e-df1d-4ee1-9075-2386718bef6c: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000832080841064', 'lastCheck': 1384915541.0748219, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00128793716431', 'lastCheck': 1384915540.629492, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00155401229858', 'lastCheck': 1384915541.0769689, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000463008880615', 'lastCheck': 1384915539.6233411, 'valid': True, 'code': 0}} >Thread-15912840::DEBUG::2013-11-20 02:45:45,640::task::495::TaskManager.Task::(_debug) Task 41db289e-df1d-4ee1-9075-2386718bef6c: moving from state preparing -> state finished >Thread-15912840::DEBUG::2013-11-20 02:45:45,641::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912840::DEBUG::2013-11-20 02:45:45,641::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912840::DEBUG::2013-11-20 02:45:45,642::task::495::TaskManager.Task::(_debug) Task 41db289e-df1d-4ee1-9075-2386718bef6c: ref 0 aborting False >Thread-15912840::INFO::2013-11-20 02:45:45,642::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000832080841064', 'lastCheck': 1384915541.0748219, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00128793716431', 'lastCheck': 1384915540.629492, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00155401229858', 'lastCheck': 1384915541.0769689, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000463008880615', 'lastCheck': 1384915539.6233411, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:45:45,894::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0748896 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:45:47,909::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:45:48,460::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0734297 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:45:50,475::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:45:50,994::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0728742 s, 14.1 MB/s\n'; <rc> = 0 >Thread-15912843::DEBUG::2013-11-20 02:45:52,949::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912843::INFO::2013-11-20 02:45:52,950::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912843::DEBUG::2013-11-20 02:45:52,951::task::495::TaskManager.Task::(_debug) Task 9528f3ef-fc42-4afd-ad88-8c266d7b767e: moving from state init -> state preparing >Thread-15912843::DEBUG::2013-11-20 02:45:52,952::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912843::DEBUG::2013-11-20 02:45:52,952::task::495::TaskManager.Task::(_debug) Task 9528f3ef-fc42-4afd-ad88-8c266d7b767e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912843::DEBUG::2013-11-20 02:45:52,953::task::495::TaskManager.Task::(_debug) Task 9528f3ef-fc42-4afd-ad88-8c266d7b767e: moving from state preparing -> state finished >Thread-15912843::DEBUG::2013-11-20 02:45:52,953::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912843::DEBUG::2013-11-20 02:45:52,954::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912843::DEBUG::2013-11-20 02:45:52,954::task::495::TaskManager.Task::(_debug) Task 9528f3ef-fc42-4afd-ad88-8c266d7b767e: ref 0 aborting False >Thread-15912843::INFO::2013-11-20 02:45:52,955::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912844::DEBUG::2013-11-20 02:45:52,959::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912844::INFO::2013-11-20 02:45:52,960::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912844::DEBUG::2013-11-20 02:45:52,961::task::495::TaskManager.Task::(_debug) Task d55cb8b0-1a3a-4c36-85c4-abc21d84e545: moving from state init -> state preparing >Thread-15912844::DEBUG::2013-11-20 02:45:52,962::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`0dd5a9ac-465d-4ab4-b7c1-a82de7661ee3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912844::DEBUG::2013-11-20 02:45:52,963::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912844::DEBUG::2013-11-20 02:45:52,963::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912844::DEBUG::2013-11-20 02:45:52,964::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`0dd5a9ac-465d-4ab4-b7c1-a82de7661ee3`::Granted request >Thread-15912844::DEBUG::2013-11-20 02:45:52,965::task::495::TaskManager.Task::(_debug) Task d55cb8b0-1a3a-4c36-85c4-abc21d84e545: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912844::DEBUG::2013-11-20 02:45:52,965::task::495::TaskManager.Task::(_debug) Task d55cb8b0-1a3a-4c36-85c4-abc21d84e545: ref 1 aborting False >Thread-15912844::WARNING::2013-11-20 02:45:52,967::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912844::DEBUG::2013-11-20 02:45:52,968::task::495::TaskManager.Task::(_debug) Task d55cb8b0-1a3a-4c36-85c4-abc21d84e545: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912844::DEBUG::2013-11-20 02:45:52,969::task::495::TaskManager.Task::(_debug) Task d55cb8b0-1a3a-4c36-85c4-abc21d84e545: moving from state preparing -> state finished >Thread-15912844::DEBUG::2013-11-20 02:45:52,970::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912844::DEBUG::2013-11-20 02:45:52,970::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912844::DEBUG::2013-11-20 02:45:52,971::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912844::DEBUG::2013-11-20 02:45:52,971::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912844::DEBUG::2013-11-20 02:45:52,972::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912844::DEBUG::2013-11-20 02:45:52,973::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912844::DEBUG::2013-11-20 02:45:52,973::task::495::TaskManager.Task::(_debug) Task d55cb8b0-1a3a-4c36-85c4-abc21d84e545: ref 0 aborting False >Thread-15912844::INFO::2013-11-20 02:45:52,974::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:45:53,010::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:45:53,546::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0734219 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:45:55,561::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:45:56,108::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.075483 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:45:58,122::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:45:58,665::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0729437 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:46:00,680::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:46:01,203::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0734643 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:46:03,219::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 02:46:03,442::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:46:03,444::task::495::TaskManager.Task::(_debug) Task b8ebbdd8-e1d8-4a45-b9b2-8371a1dab171: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:46:03,446::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`8db79504-b786-4f7b-991d-a769b8018b65`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:46:03,446::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:46:03,447::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:46:03,448::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`8db79504-b786-4f7b-991d-a769b8018b65`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:46:03,449::task::495::TaskManager.Task::(_debug) Task b8ebbdd8-e1d8-4a45-b9b2-8371a1dab171: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:46:03,450::task::495::TaskManager.Task::(_debug) Task b8ebbdd8-e1d8-4a45-b9b2-8371a1dab171: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:46:03,450::task::495::TaskManager.Task::(_debug) Task b8ebbdd8-e1d8-4a45-b9b2-8371a1dab171: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:46:03,451::task::495::TaskManager.Task::(_debug) Task b8ebbdd8-e1d8-4a45-b9b2-8371a1dab171: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:46:03,452::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:46:03,452::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:46:03,453::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:46:03,453::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:46:03,454::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:46:03,454::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:46:03,455::task::495::TaskManager.Task::(_debug) Task b8ebbdd8-e1d8-4a45-b9b2-8371a1dab171: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:46:03,455::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 02:46:03,742::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0746276 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15912848::INFO::2013-11-20 02:46:04,428::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912848::DEBUG::2013-11-20 02:46:04,429::task::495::TaskManager.Task::(_debug) Task ca35a9b1-983c-4a05-b562-115bfc9a531c: moving from state init -> state preparing >Thread-15912848::DEBUG::2013-11-20 02:46:04,429::task::495::TaskManager.Task::(_debug) Task ca35a9b1-983c-4a05-b562-115bfc9a531c: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000766038894653', 'lastCheck': 1384915561.37098, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00126099586487', 'lastCheck': 1384915560.9136021, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00150203704834', 'lastCheck': 1384915561.3631539, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000483989715576', 'lastCheck': 1384915559.821532, 'valid': True, 'code': 0}} >Thread-15912848::DEBUG::2013-11-20 02:46:04,430::task::495::TaskManager.Task::(_debug) Task ca35a9b1-983c-4a05-b562-115bfc9a531c: moving from state preparing -> state finished >Thread-15912848::DEBUG::2013-11-20 02:46:04,431::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912848::DEBUG::2013-11-20 02:46:04,431::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912848::DEBUG::2013-11-20 02:46:04,432::task::495::TaskManager.Task::(_debug) Task ca35a9b1-983c-4a05-b562-115bfc9a531c: ref 0 aborting False >Thread-15912848::INFO::2013-11-20 02:46:04,432::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000766038894653', 'lastCheck': 1384915561.37098, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00126099586487', 'lastCheck': 1384915560.9136021, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00150203704834', 'lastCheck': 1384915561.3631539, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000483989715576', 'lastCheck': 1384915559.821532, 'valid': True, 'code': 0}} >Thread-15912850::DEBUG::2013-11-20 02:46:04,848::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912850::INFO::2013-11-20 02:46:04,850::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912850::DEBUG::2013-11-20 02:46:04,850::task::495::TaskManager.Task::(_debug) Task 01f5600c-57ee-488b-a51a-512b251e1c4f: moving from state init -> state preparing >Thread-15912850::DEBUG::2013-11-20 02:46:04,851::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912850::DEBUG::2013-11-20 02:46:04,851::task::495::TaskManager.Task::(_debug) Task 01f5600c-57ee-488b-a51a-512b251e1c4f: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912850::DEBUG::2013-11-20 02:46:04,852::task::495::TaskManager.Task::(_debug) Task 01f5600c-57ee-488b-a51a-512b251e1c4f: moving from state preparing -> state finished >Thread-15912850::DEBUG::2013-11-20 02:46:04,853::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912850::DEBUG::2013-11-20 02:46:04,853::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912850::DEBUG::2013-11-20 02:46:04,854::task::495::TaskManager.Task::(_debug) Task 01f5600c-57ee-488b-a51a-512b251e1c4f: ref 0 aborting False >Thread-15912850::INFO::2013-11-20 02:46:04,854::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912851::DEBUG::2013-11-20 02:46:04,859::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912851::INFO::2013-11-20 02:46:04,860::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912851::DEBUG::2013-11-20 02:46:04,860::task::495::TaskManager.Task::(_debug) Task acfdb1e3-0b98-4212-b701-4097fb79c903: moving from state init -> state preparing >Thread-15912851::DEBUG::2013-11-20 02:46:04,867::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`7367a4a0-17e6-4675-8bd0-41a782e44ca4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912851::DEBUG::2013-11-20 02:46:04,868::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912851::DEBUG::2013-11-20 02:46:04,869::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912851::DEBUG::2013-11-20 02:46:04,869::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`7367a4a0-17e6-4675-8bd0-41a782e44ca4`::Granted request >Thread-15912851::DEBUG::2013-11-20 02:46:04,870::task::495::TaskManager.Task::(_debug) Task acfdb1e3-0b98-4212-b701-4097fb79c903: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912851::DEBUG::2013-11-20 02:46:04,871::task::495::TaskManager.Task::(_debug) Task acfdb1e3-0b98-4212-b701-4097fb79c903: ref 1 aborting False >Thread-15912851::WARNING::2013-11-20 02:46:04,872::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912851::DEBUG::2013-11-20 02:46:04,873::task::495::TaskManager.Task::(_debug) Task acfdb1e3-0b98-4212-b701-4097fb79c903: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912851::DEBUG::2013-11-20 02:46:04,874::task::495::TaskManager.Task::(_debug) Task acfdb1e3-0b98-4212-b701-4097fb79c903: moving from state preparing -> state finished >Thread-15912851::DEBUG::2013-11-20 02:46:04,875::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912851::DEBUG::2013-11-20 02:46:04,875::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912851::DEBUG::2013-11-20 02:46:04,876::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912851::DEBUG::2013-11-20 02:46:04,876::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912851::DEBUG::2013-11-20 02:46:04,877::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912851::DEBUG::2013-11-20 02:46:04,877::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912851::DEBUG::2013-11-20 02:46:04,878::task::495::TaskManager.Task::(_debug) Task acfdb1e3-0b98-4212-b701-4097fb79c903: ref 0 aborting False >Thread-15912851::INFO::2013-11-20 02:46:04,879::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:46:05,757::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:46:06,233::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0428606 s, 23.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:46:08,243::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:46:08,816::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.109477 s, 9.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:46:10,831::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:46:11,351::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0702245 s, 14.6 MB/s\n'; <rc> = 0 >Thread-42838::DEBUG::2013-11-20 02:46:11,475::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 02:46:12,143::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:46:13,366::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:46:13,903::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0728261 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:46:15,918::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:46:16,486::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0773797 s, 13.2 MB/s\n'; <rc> = 0 >Thread-15912855::DEBUG::2013-11-20 02:46:18,288::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912855::INFO::2013-11-20 02:46:18,289::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912855::DEBUG::2013-11-20 02:46:18,290::task::495::TaskManager.Task::(_debug) Task af3b0cbe-cab7-45e2-b0ac-e15b3d701386: moving from state init -> state preparing >Thread-15912855::DEBUG::2013-11-20 02:46:18,290::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912855::DEBUG::2013-11-20 02:46:18,291::task::495::TaskManager.Task::(_debug) Task af3b0cbe-cab7-45e2-b0ac-e15b3d701386: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912855::DEBUG::2013-11-20 02:46:18,291::task::495::TaskManager.Task::(_debug) Task af3b0cbe-cab7-45e2-b0ac-e15b3d701386: moving from state preparing -> state finished >Thread-15912855::DEBUG::2013-11-20 02:46:18,292::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912855::DEBUG::2013-11-20 02:46:18,292::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912855::DEBUG::2013-11-20 02:46:18,293::task::495::TaskManager.Task::(_debug) Task af3b0cbe-cab7-45e2-b0ac-e15b3d701386: ref 0 aborting False >Thread-15912855::INFO::2013-11-20 02:46:18,293::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912856::DEBUG::2013-11-20 02:46:18,298::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912856::INFO::2013-11-20 02:46:18,298::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912856::DEBUG::2013-11-20 02:46:18,299::task::495::TaskManager.Task::(_debug) Task 9e3da743-565c-48df-a268-05759809d474: moving from state init -> state preparing >Thread-15912856::DEBUG::2013-11-20 02:46:18,300::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9e07a869-63df-48c1-b000-3f76261fa9e5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912856::DEBUG::2013-11-20 02:46:18,300::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912856::DEBUG::2013-11-20 02:46:18,301::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912856::DEBUG::2013-11-20 02:46:18,301::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9e07a869-63df-48c1-b000-3f76261fa9e5`::Granted request >Thread-15912856::DEBUG::2013-11-20 02:46:18,302::task::495::TaskManager.Task::(_debug) Task 9e3da743-565c-48df-a268-05759809d474: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912856::DEBUG::2013-11-20 02:46:18,303::task::495::TaskManager.Task::(_debug) Task 9e3da743-565c-48df-a268-05759809d474: ref 1 aborting False >Thread-15912856::WARNING::2013-11-20 02:46:18,304::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912856::DEBUG::2013-11-20 02:46:18,305::task::495::TaskManager.Task::(_debug) Task 9e3da743-565c-48df-a268-05759809d474: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115941048320', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115941048320', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912856::DEBUG::2013-11-20 02:46:18,306::task::495::TaskManager.Task::(_debug) Task 9e3da743-565c-48df-a268-05759809d474: moving from state preparing -> state finished >Thread-15912856::DEBUG::2013-11-20 02:46:18,306::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912856::DEBUG::2013-11-20 02:46:18,307::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912856::DEBUG::2013-11-20 02:46:18,307::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912856::DEBUG::2013-11-20 02:46:18,308::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912856::DEBUG::2013-11-20 02:46:18,308::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912856::DEBUG::2013-11-20 02:46:18,309::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912856::DEBUG::2013-11-20 02:46:18,309::task::495::TaskManager.Task::(_debug) Task 9e3da743-565c-48df-a268-05759809d474: ref 0 aborting False >Thread-15912856::INFO::2013-11-20 02:46:18,310::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115941048320', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115941048320', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:46:18,501::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:46:19,045::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0707807 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:46:21,059::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:46:21,623::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.10777 s, 9.5 MB/s\n'; <rc> = 0 >Thread-15912858::INFO::2013-11-20 02:46:23,121::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912858::DEBUG::2013-11-20 02:46:23,122::task::495::TaskManager.Task::(_debug) Task 4f854ef0-10c3-437e-83de-d3f79fa61235: moving from state init -> state preparing >Thread-15912858::DEBUG::2013-11-20 02:46:23,123::task::495::TaskManager.Task::(_debug) Task 4f854ef0-10c3-437e-83de-d3f79fa61235: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000284194946289', 'lastCheck': 1384915582.282527, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00133800506592', 'lastCheck': 1384915581.2967811, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00146818161011', 'lastCheck': 1384915581.741168, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000514030456543', 'lastCheck': 1384915580.153661, 'valid': True, 'code': 0}} >Thread-15912858::DEBUG::2013-11-20 02:46:23,124::task::495::TaskManager.Task::(_debug) Task 4f854ef0-10c3-437e-83de-d3f79fa61235: moving from state preparing -> state finished >Thread-15912858::DEBUG::2013-11-20 02:46:23,124::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912858::DEBUG::2013-11-20 02:46:23,125::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912858::DEBUG::2013-11-20 02:46:23,125::task::495::TaskManager.Task::(_debug) Task 4f854ef0-10c3-437e-83de-d3f79fa61235: ref 0 aborting False >Thread-15912858::INFO::2013-11-20 02:46:23,126::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000284194946289', 'lastCheck': 1384915582.282527, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00133800506592', 'lastCheck': 1384915581.2967811, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00146818161011', 'lastCheck': 1384915581.741168, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000514030456543', 'lastCheck': 1384915580.153661, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:46:23,636::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:46:24,193::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0757027 s, 13.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:46:26,209::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:46:26,739::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0743503 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:46:28,754::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912861::DEBUG::2013-11-20 02:46:29,220::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912861::INFO::2013-11-20 02:46:29,224::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912861::DEBUG::2013-11-20 02:46:29,226::task::495::TaskManager.Task::(_debug) Task 5349e46f-92ac-4e79-a9b5-7730e1e1d854: moving from state init -> state preparing >Thread-15912861::DEBUG::2013-11-20 02:46:29,228::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912861::DEBUG::2013-11-20 02:46:29,230::task::495::TaskManager.Task::(_debug) Task 5349e46f-92ac-4e79-a9b5-7730e1e1d854: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912861::DEBUG::2013-11-20 02:46:29,231::task::495::TaskManager.Task::(_debug) Task 5349e46f-92ac-4e79-a9b5-7730e1e1d854: moving from state preparing -> state finished >Thread-15912861::DEBUG::2013-11-20 02:46:29,234::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912861::DEBUG::2013-11-20 02:46:29,236::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912861::DEBUG::2013-11-20 02:46:29,237::task::495::TaskManager.Task::(_debug) Task 5349e46f-92ac-4e79-a9b5-7730e1e1d854: ref 0 aborting False >Thread-15912861::INFO::2013-11-20 02:46:29,240::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912862::DEBUG::2013-11-20 02:46:29,247::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912862::INFO::2013-11-20 02:46:29,248::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912862::DEBUG::2013-11-20 02:46:29,249::task::495::TaskManager.Task::(_debug) Task 295ed114-d44a-485c-8ce1-381b93573bab: moving from state init -> state preparing >Thread-15912862::DEBUG::2013-11-20 02:46:29,252::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`81ebe5b3-f246-4b3d-be75-bb925bb7f279`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912862::DEBUG::2013-11-20 02:46:29,253::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912862::DEBUG::2013-11-20 02:46:29,256::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912862::DEBUG::2013-11-20 02:46:29,257::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`81ebe5b3-f246-4b3d-be75-bb925bb7f279`::Granted request >Thread-15912862::DEBUG::2013-11-20 02:46:29,259::task::495::TaskManager.Task::(_debug) Task 295ed114-d44a-485c-8ce1-381b93573bab: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912862::DEBUG::2013-11-20 02:46:29,260::task::495::TaskManager.Task::(_debug) Task 295ed114-d44a-485c-8ce1-381b93573bab: ref 1 aborting False >Thread-15912862::WARNING::2013-11-20 02:46:29,262::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912862::DEBUG::2013-11-20 02:46:29,264::task::495::TaskManager.Task::(_debug) Task 295ed114-d44a-485c-8ce1-381b93573bab: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115941048320', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115941048320', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912862::DEBUG::2013-11-20 02:46:29,264::task::495::TaskManager.Task::(_debug) Task 295ed114-d44a-485c-8ce1-381b93573bab: moving from state preparing -> state finished >Thread-15912862::DEBUG::2013-11-20 02:46:29,265::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912862::DEBUG::2013-11-20 02:46:29,266::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912862::DEBUG::2013-11-20 02:46:29,268::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912862::DEBUG::2013-11-20 02:46:29,270::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912862::DEBUG::2013-11-20 02:46:29,270::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912862::DEBUG::2013-11-20 02:46:29,272::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912862::DEBUG::2013-11-20 02:46:29,274::task::495::TaskManager.Task::(_debug) Task 295ed114-d44a-485c-8ce1-381b93573bab: ref 0 aborting False >Thread-15912862::INFO::2013-11-20 02:46:29,275::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115941048320', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115941048320', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:46:29,311::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0726379 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:46:31,330::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:46:31,845::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0729434 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:46:33,860::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:46:34,318::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0516183 s, 19.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:46:36,334::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:46:36,876::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0648102 s, 15.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:46:38,890::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:46:39,397::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0873119 s, 11.7 MB/s\n'; <rc> = 0 >Thread-15912866::DEBUG::2013-11-20 02:46:40,796::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912866::INFO::2013-11-20 02:46:40,798::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912866::DEBUG::2013-11-20 02:46:40,798::task::495::TaskManager.Task::(_debug) Task 76cb0ec9-b492-4599-85be-796d73351f71: moving from state init -> state preparing >Thread-15912866::DEBUG::2013-11-20 02:46:40,799::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912866::DEBUG::2013-11-20 02:46:40,800::task::495::TaskManager.Task::(_debug) Task 76cb0ec9-b492-4599-85be-796d73351f71: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912866::DEBUG::2013-11-20 02:46:40,801::task::495::TaskManager.Task::(_debug) Task 76cb0ec9-b492-4599-85be-796d73351f71: moving from state preparing -> state finished >Thread-15912866::DEBUG::2013-11-20 02:46:40,801::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912866::DEBUG::2013-11-20 02:46:40,802::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912866::DEBUG::2013-11-20 02:46:40,802::task::495::TaskManager.Task::(_debug) Task 76cb0ec9-b492-4599-85be-796d73351f71: ref 0 aborting False >Thread-15912866::INFO::2013-11-20 02:46:40,803::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912867::DEBUG::2013-11-20 02:46:40,808::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912867::INFO::2013-11-20 02:46:40,808::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912867::DEBUG::2013-11-20 02:46:40,809::task::495::TaskManager.Task::(_debug) Task fe194583-ac28-46e1-b7d4-6bf719f6431d: moving from state init -> state preparing >Thread-15912867::DEBUG::2013-11-20 02:46:40,810::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bd817009-7aa5-47c6-8fae-31b55b4b2f98`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912867::DEBUG::2013-11-20 02:46:40,811::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912867::DEBUG::2013-11-20 02:46:40,812::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912867::DEBUG::2013-11-20 02:46:40,812::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bd817009-7aa5-47c6-8fae-31b55b4b2f98`::Granted request >Thread-15912867::DEBUG::2013-11-20 02:46:40,813::task::495::TaskManager.Task::(_debug) Task fe194583-ac28-46e1-b7d4-6bf719f6431d: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912867::DEBUG::2013-11-20 02:46:40,814::task::495::TaskManager.Task::(_debug) Task fe194583-ac28-46e1-b7d4-6bf719f6431d: ref 1 aborting False >Thread-15912867::WARNING::2013-11-20 02:46:40,816::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912867::DEBUG::2013-11-20 02:46:40,816::task::495::TaskManager.Task::(_debug) Task fe194583-ac28-46e1-b7d4-6bf719f6431d: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115924271104', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115923222528', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912867::DEBUG::2013-11-20 02:46:40,817::task::495::TaskManager.Task::(_debug) Task fe194583-ac28-46e1-b7d4-6bf719f6431d: moving from state preparing -> state finished >Thread-15912867::DEBUG::2013-11-20 02:46:40,818::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912867::DEBUG::2013-11-20 02:46:40,818::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912867::DEBUG::2013-11-20 02:46:40,819::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912867::DEBUG::2013-11-20 02:46:40,820::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912867::DEBUG::2013-11-20 02:46:40,820::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912867::DEBUG::2013-11-20 02:46:40,821::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912867::DEBUG::2013-11-20 02:46:40,821::task::495::TaskManager.Task::(_debug) Task fe194583-ac28-46e1-b7d4-6bf719f6431d: ref 0 aborting False >Thread-15912867::INFO::2013-11-20 02:46:40,822::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115924271104', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115923222528', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912868::INFO::2013-11-20 02:46:41,338::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912868::DEBUG::2013-11-20 02:46:41,339::task::495::TaskManager.Task::(_debug) Task fdfffb56-5876-4bdd-8b96-f7fccc60c4d2: moving from state init -> state preparing >Thread-15912868::DEBUG::2013-11-20 02:46:41,340::task::495::TaskManager.Task::(_debug) Task fdfffb56-5876-4bdd-8b96-f7fccc60c4d2: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000371932983398', 'lastCheck': 1384915592.5432799, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00159811973572', 'lastCheck': 1384915591.5499549, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00140309333801', 'lastCheck': 1384915591.9769869, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000420093536377', 'lastCheck': 1384915600.5503819, 'valid': True, 'code': 0}} >Thread-15912868::DEBUG::2013-11-20 02:46:41,340::task::495::TaskManager.Task::(_debug) Task fdfffb56-5876-4bdd-8b96-f7fccc60c4d2: moving from state preparing -> state finished >Thread-15912868::DEBUG::2013-11-20 02:46:41,341::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912868::DEBUG::2013-11-20 02:46:41,341::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912868::DEBUG::2013-11-20 02:46:41,342::task::495::TaskManager.Task::(_debug) Task fdfffb56-5876-4bdd-8b96-f7fccc60c4d2: ref 0 aborting False >Thread-15912868::INFO::2013-11-20 02:46:41,343::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000371932983398', 'lastCheck': 1384915592.5432799, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00159811973572', 'lastCheck': 1384915591.5499549, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00140309333801', 'lastCheck': 1384915591.9769869, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000420093536377', 'lastCheck': 1384915600.5503819, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:46:41,412::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:46:41,960::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0820122 s, 12.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:46:43,976::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:46:44,493::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0755298 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:46:46,508::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:46:46,940::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0533042 s, 19.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:46:48,953::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:46:49,492::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0744508 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:46:51,508::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:46:52,069::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0762855 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:46:54,084::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912873::DEBUG::2013-11-20 02:46:54,354::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912873::INFO::2013-11-20 02:46:54,361::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912873::DEBUG::2013-11-20 02:46:54,361::task::495::TaskManager.Task::(_debug) Task a51244df-c9b1-4a7b-8508-0cb0e78b5f3b: moving from state init -> state preparing >Thread-15912873::DEBUG::2013-11-20 02:46:54,362::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912873::DEBUG::2013-11-20 02:46:54,362::task::495::TaskManager.Task::(_debug) Task a51244df-c9b1-4a7b-8508-0cb0e78b5f3b: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912873::DEBUG::2013-11-20 02:46:54,362::task::495::TaskManager.Task::(_debug) Task a51244df-c9b1-4a7b-8508-0cb0e78b5f3b: moving from state preparing -> state finished >Thread-15912873::DEBUG::2013-11-20 02:46:54,363::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912873::DEBUG::2013-11-20 02:46:54,363::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912873::DEBUG::2013-11-20 02:46:54,363::task::495::TaskManager.Task::(_debug) Task a51244df-c9b1-4a7b-8508-0cb0e78b5f3b: ref 0 aborting False >Thread-15912873::INFO::2013-11-20 02:46:54,364::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912874::DEBUG::2013-11-20 02:46:54,368::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912874::INFO::2013-11-20 02:46:54,369::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912874::DEBUG::2013-11-20 02:46:54,370::task::495::TaskManager.Task::(_debug) Task 9756e5e4-5ce8-48b4-8989-7e26e4955b53: moving from state init -> state preparing >Thread-15912874::DEBUG::2013-11-20 02:46:54,371::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`89e7b645-937e-496f-9f97-3ddd7d30f7dc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912874::DEBUG::2013-11-20 02:46:54,372::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912874::DEBUG::2013-11-20 02:46:54,373::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912874::DEBUG::2013-11-20 02:46:54,373::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`89e7b645-937e-496f-9f97-3ddd7d30f7dc`::Granted request >Thread-15912874::DEBUG::2013-11-20 02:46:54,375::task::495::TaskManager.Task::(_debug) Task 9756e5e4-5ce8-48b4-8989-7e26e4955b53: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912874::DEBUG::2013-11-20 02:46:54,375::task::495::TaskManager.Task::(_debug) Task 9756e5e4-5ce8-48b4-8989-7e26e4955b53: ref 1 aborting False >Thread-15912874::WARNING::2013-11-20 02:46:54,377::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912874::DEBUG::2013-11-20 02:46:54,378::task::495::TaskManager.Task::(_debug) Task 9756e5e4-5ce8-48b4-8989-7e26e4955b53: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115825704960', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115822559232', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912874::DEBUG::2013-11-20 02:46:54,379::task::495::TaskManager.Task::(_debug) Task 9756e5e4-5ce8-48b4-8989-7e26e4955b53: moving from state preparing -> state finished >Thread-15912874::DEBUG::2013-11-20 02:46:54,380::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912874::DEBUG::2013-11-20 02:46:54,380::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912874::DEBUG::2013-11-20 02:46:54,381::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912874::DEBUG::2013-11-20 02:46:54,382::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912874::DEBUG::2013-11-20 02:46:54,382::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912874::DEBUG::2013-11-20 02:46:54,383::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912874::DEBUG::2013-11-20 02:46:54,383::task::495::TaskManager.Task::(_debug) Task 9756e5e4-5ce8-48b4-8989-7e26e4955b53: ref 0 aborting False >Thread-15912874::INFO::2013-11-20 02:46:54,384::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115825704960', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115822559232', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:46:54,625::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0735197 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:46:56,641::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:46:57,186::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0738327 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:46:59,202::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:46:59,625::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.045713 s, 22.4 MB/s\n'; <rc> = 0 >Thread-15912876::INFO::2013-11-20 02:46:59,894::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912876::DEBUG::2013-11-20 02:46:59,896::task::495::TaskManager.Task::(_debug) Task c2a85e7a-d0fa-409f-8430-71598aed3432: moving from state init -> state preparing >Thread-15912876::DEBUG::2013-11-20 02:46:59,897::task::495::TaskManager.Task::(_debug) Task c2a85e7a-d0fa-409f-8430-71598aed3432: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000380039215088', 'lastCheck': 1384915612.758204, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00153183937073', 'lastCheck': 1384915611.761709, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '1.17933297157', 'lastCheck': 1384915613.3739481, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000397920608521', 'lastCheck': 1384915610.692369, 'valid': True, 'code': 0}} >Thread-15912876::DEBUG::2013-11-20 02:46:59,898::task::495::TaskManager.Task::(_debug) Task c2a85e7a-d0fa-409f-8430-71598aed3432: moving from state preparing -> state finished >Thread-15912876::DEBUG::2013-11-20 02:46:59,899::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912876::DEBUG::2013-11-20 02:46:59,899::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912876::DEBUG::2013-11-20 02:46:59,900::task::495::TaskManager.Task::(_debug) Task c2a85e7a-d0fa-409f-8430-71598aed3432: ref 0 aborting False >Thread-15912876::INFO::2013-11-20 02:46:59,900::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000380039215088', 'lastCheck': 1384915612.758204, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00153183937073', 'lastCheck': 1384915611.761709, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '1.17933297157', 'lastCheck': 1384915613.3739481, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000397920608521', 'lastCheck': 1384915610.692369, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:47:01,635::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:02,135::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0729208 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:47:04,150::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 02:47:04,409::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:47:04,410::task::495::TaskManager.Task::(_debug) Task 06f9e858-8170-4cf6-80c8-eaef06d471f6: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:47:04,411::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c7e41c64-53cd-498f-bb6e-e268712df33d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:47:04,412::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:47:04,412::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:47:04,413::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c7e41c64-53cd-498f-bb6e-e268712df33d`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:47:04,413::task::495::TaskManager.Task::(_debug) Task 06f9e858-8170-4cf6-80c8-eaef06d471f6: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:47:04,414::task::495::TaskManager.Task::(_debug) Task 06f9e858-8170-4cf6-80c8-eaef06d471f6: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:47:04,414::task::495::TaskManager.Task::(_debug) Task 06f9e858-8170-4cf6-80c8-eaef06d471f6: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:47:04,415::task::495::TaskManager.Task::(_debug) Task 06f9e858-8170-4cf6-80c8-eaef06d471f6: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:47:04,415::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:47:04,415::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:47:04,416::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:47:04,416::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:47:04,416::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:47:04,417::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:47:04,417::task::495::TaskManager.Task::(_debug) Task 06f9e858-8170-4cf6-80c8-eaef06d471f6: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:47:04,417::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 02:47:04,703::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0743489 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15912878::DEBUG::2013-11-20 02:47:05,774::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912878::INFO::2013-11-20 02:47:05,775::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912878::DEBUG::2013-11-20 02:47:05,776::task::495::TaskManager.Task::(_debug) Task 0b62fdff-2762-415f-9e1d-53eed2147812: moving from state init -> state preparing >Thread-15912878::DEBUG::2013-11-20 02:47:05,777::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912878::DEBUG::2013-11-20 02:47:05,777::task::495::TaskManager.Task::(_debug) Task 0b62fdff-2762-415f-9e1d-53eed2147812: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912878::DEBUG::2013-11-20 02:47:05,778::task::495::TaskManager.Task::(_debug) Task 0b62fdff-2762-415f-9e1d-53eed2147812: moving from state preparing -> state finished >Thread-15912878::DEBUG::2013-11-20 02:47:05,779::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912878::DEBUG::2013-11-20 02:47:05,779::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912878::DEBUG::2013-11-20 02:47:05,780::task::495::TaskManager.Task::(_debug) Task 0b62fdff-2762-415f-9e1d-53eed2147812: ref 0 aborting False >Thread-15912878::INFO::2013-11-20 02:47:05,780::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912879::DEBUG::2013-11-20 02:47:05,784::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912879::INFO::2013-11-20 02:47:05,785::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912879::DEBUG::2013-11-20 02:47:05,786::task::495::TaskManager.Task::(_debug) Task bba29169-369a-42a5-958a-2030a1668038: moving from state init -> state preparing >Thread-15912879::DEBUG::2013-11-20 02:47:05,787::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b376131e-a630-4fc4-801a-d482b30bdcc6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912879::DEBUG::2013-11-20 02:47:05,788::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912879::DEBUG::2013-11-20 02:47:05,788::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912879::DEBUG::2013-11-20 02:47:05,789::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b376131e-a630-4fc4-801a-d482b30bdcc6`::Granted request >Thread-15912879::DEBUG::2013-11-20 02:47:05,790::task::495::TaskManager.Task::(_debug) Task bba29169-369a-42a5-958a-2030a1668038: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912879::DEBUG::2013-11-20 02:47:05,790::task::495::TaskManager.Task::(_debug) Task bba29169-369a-42a5-958a-2030a1668038: ref 1 aborting False >Thread-15912879::WARNING::2013-11-20 02:47:05,792::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912879::DEBUG::2013-11-20 02:47:05,793::task::495::TaskManager.Task::(_debug) Task bba29169-369a-42a5-958a-2030a1668038: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115777470464', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115777470464', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912879::DEBUG::2013-11-20 02:47:05,794::task::495::TaskManager.Task::(_debug) Task bba29169-369a-42a5-958a-2030a1668038: moving from state preparing -> state finished >Thread-15912879::DEBUG::2013-11-20 02:47:05,794::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912879::DEBUG::2013-11-20 02:47:05,795::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912879::DEBUG::2013-11-20 02:47:05,796::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912879::DEBUG::2013-11-20 02:47:05,796::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912879::DEBUG::2013-11-20 02:47:05,797::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912879::DEBUG::2013-11-20 02:47:05,797::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912879::DEBUG::2013-11-20 02:47:05,798::task::495::TaskManager.Task::(_debug) Task bba29169-369a-42a5-958a-2030a1668038: ref 0 aborting False >Thread-15912879::INFO::2013-11-20 02:47:05,798::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115777470464', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115777470464', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:47:06,718::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:07,163::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0436064 s, 23.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:47:09,176::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:09,711::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0733351 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:47:11,727::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:12,262::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0745845 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:47:14,278::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:14,830::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.071703 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:47:16,846::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:17,472::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0705565 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15912884::INFO::2013-11-20 02:47:19,097::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912884::DEBUG::2013-11-20 02:47:19,098::task::495::TaskManager.Task::(_debug) Task 242c6a79-a890-4e62-bc65-4c568979f9cd: moving from state init -> state preparing >Thread-15912884::DEBUG::2013-11-20 02:47:19,099::task::495::TaskManager.Task::(_debug) Task 242c6a79-a890-4e62-bc65-4c568979f9cd: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00434994697571', 'lastCheck': 1384915632.95312, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00172710418701', 'lastCheck': 1384915631.9526539, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00142192840576', 'lastCheck': 1384915633.4315341, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00501394271851', 'lastCheck': 1384915630.9430161, 'valid': True, 'code': 0}} >Thread-15912884::DEBUG::2013-11-20 02:47:19,100::task::495::TaskManager.Task::(_debug) Task 242c6a79-a890-4e62-bc65-4c568979f9cd: moving from state preparing -> state finished >Thread-15912884::DEBUG::2013-11-20 02:47:19,100::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912884::DEBUG::2013-11-20 02:47:19,101::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912884::DEBUG::2013-11-20 02:47:19,101::task::495::TaskManager.Task::(_debug) Task 242c6a79-a890-4e62-bc65-4c568979f9cd: ref 0 aborting False >Thread-15912884::INFO::2013-11-20 02:47:19,102::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00434994697571', 'lastCheck': 1384915632.95312, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00172710418701', 'lastCheck': 1384915631.9526539, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00142192840576', 'lastCheck': 1384915633.4315341, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00501394271851', 'lastCheck': 1384915630.9430161, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:47:19,487::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:20,015::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0746915 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15912886::DEBUG::2013-11-20 02:47:20,205::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912886::INFO::2013-11-20 02:47:20,206::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912886::DEBUG::2013-11-20 02:47:20,207::task::495::TaskManager.Task::(_debug) Task f02995b8-2ffa-4cae-bcdc-c00cb58b0111: moving from state init -> state preparing >Thread-15912886::DEBUG::2013-11-20 02:47:20,208::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912886::DEBUG::2013-11-20 02:47:20,208::task::495::TaskManager.Task::(_debug) Task f02995b8-2ffa-4cae-bcdc-c00cb58b0111: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912886::DEBUG::2013-11-20 02:47:20,209::task::495::TaskManager.Task::(_debug) Task f02995b8-2ffa-4cae-bcdc-c00cb58b0111: moving from state preparing -> state finished >Thread-15912886::DEBUG::2013-11-20 02:47:20,209::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912886::DEBUG::2013-11-20 02:47:20,210::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912886::DEBUG::2013-11-20 02:47:20,210::task::495::TaskManager.Task::(_debug) Task f02995b8-2ffa-4cae-bcdc-c00cb58b0111: ref 0 aborting False >Thread-15912886::INFO::2013-11-20 02:47:20,211::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912887::DEBUG::2013-11-20 02:47:20,215::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912887::INFO::2013-11-20 02:47:20,216::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912887::DEBUG::2013-11-20 02:47:20,217::task::495::TaskManager.Task::(_debug) Task 96ffdde3-d0f5-49a4-9f0b-08f2e9d96fef: moving from state init -> state preparing >Thread-15912887::DEBUG::2013-11-20 02:47:20,218::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5f4d666f-25a9-4e0c-b566-570db0171daa`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912887::DEBUG::2013-11-20 02:47:20,218::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912887::DEBUG::2013-11-20 02:47:20,219::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912887::DEBUG::2013-11-20 02:47:20,220::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5f4d666f-25a9-4e0c-b566-570db0171daa`::Granted request >Thread-15912887::DEBUG::2013-11-20 02:47:20,221::task::495::TaskManager.Task::(_debug) Task 96ffdde3-d0f5-49a4-9f0b-08f2e9d96fef: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912887::DEBUG::2013-11-20 02:47:20,221::task::495::TaskManager.Task::(_debug) Task 96ffdde3-d0f5-49a4-9f0b-08f2e9d96fef: ref 1 aborting False >Thread-15912887::WARNING::2013-11-20 02:47:20,223::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912887::DEBUG::2013-11-20 02:47:20,224::task::495::TaskManager.Task::(_debug) Task 96ffdde3-d0f5-49a4-9f0b-08f2e9d96fef: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115786907648', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115786907648', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912887::DEBUG::2013-11-20 02:47:20,225::task::495::TaskManager.Task::(_debug) Task 96ffdde3-d0f5-49a4-9f0b-08f2e9d96fef: moving from state preparing -> state finished >Thread-15912887::DEBUG::2013-11-20 02:47:20,226::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912887::DEBUG::2013-11-20 02:47:20,226::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912887::DEBUG::2013-11-20 02:47:20,227::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912887::DEBUG::2013-11-20 02:47:20,227::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912887::DEBUG::2013-11-20 02:47:20,228::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912887::DEBUG::2013-11-20 02:47:20,228::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912887::DEBUG::2013-11-20 02:47:20,229::task::495::TaskManager.Task::(_debug) Task 96ffdde3-d0f5-49a4-9f0b-08f2e9d96fef: ref 0 aborting False >Thread-15912887::INFO::2013-11-20 02:47:20,230::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115786907648', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115786907648', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:47:22,031::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:22,534::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.053826 s, 19.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:47:24,544::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:25,028::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0571899 s, 17.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:47:27,042::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:27,562::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.107634 s, 9.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:47:29,577::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:30,031::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0470991 s, 21.7 MB/s\n'; <rc> = 0 >Thread-15912891::DEBUG::2013-11-20 02:47:31,129::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912891::INFO::2013-11-20 02:47:31,130::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912891::DEBUG::2013-11-20 02:47:31,131::task::495::TaskManager.Task::(_debug) Task 6a232e9d-9ef7-44d2-8bb5-f8e6a10070f4: moving from state init -> state preparing >Thread-15912891::DEBUG::2013-11-20 02:47:31,132::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912891::DEBUG::2013-11-20 02:47:31,132::task::495::TaskManager.Task::(_debug) Task 6a232e9d-9ef7-44d2-8bb5-f8e6a10070f4: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912891::DEBUG::2013-11-20 02:47:31,133::task::495::TaskManager.Task::(_debug) Task 6a232e9d-9ef7-44d2-8bb5-f8e6a10070f4: moving from state preparing -> state finished >Thread-15912891::DEBUG::2013-11-20 02:47:31,134::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912891::DEBUG::2013-11-20 02:47:31,134::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912891::DEBUG::2013-11-20 02:47:31,135::task::495::TaskManager.Task::(_debug) Task 6a232e9d-9ef7-44d2-8bb5-f8e6a10070f4: ref 0 aborting False >Thread-15912891::INFO::2013-11-20 02:47:31,135::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912892::DEBUG::2013-11-20 02:47:31,140::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912892::INFO::2013-11-20 02:47:31,141::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912892::DEBUG::2013-11-20 02:47:31,142::task::495::TaskManager.Task::(_debug) Task 18b43929-e141-491a-bace-04debc8fd802: moving from state init -> state preparing >Thread-15912892::DEBUG::2013-11-20 02:47:31,143::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ca458766-2145-44ae-8f5c-5630700c0f7f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912892::DEBUG::2013-11-20 02:47:31,143::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912892::DEBUG::2013-11-20 02:47:31,144::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912892::DEBUG::2013-11-20 02:47:31,145::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ca458766-2145-44ae-8f5c-5630700c0f7f`::Granted request >Thread-15912892::DEBUG::2013-11-20 02:47:31,146::task::495::TaskManager.Task::(_debug) Task 18b43929-e141-491a-bace-04debc8fd802: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912892::DEBUG::2013-11-20 02:47:31,146::task::495::TaskManager.Task::(_debug) Task 18b43929-e141-491a-bace-04debc8fd802: ref 1 aborting False >Thread-15912892::WARNING::2013-11-20 02:47:31,149::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912892::DEBUG::2013-11-20 02:47:31,149::task::495::TaskManager.Task::(_debug) Task 18b43929-e141-491a-bace-04debc8fd802: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115785859072', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115785859072', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912892::DEBUG::2013-11-20 02:47:31,150::task::495::TaskManager.Task::(_debug) Task 18b43929-e141-491a-bace-04debc8fd802: moving from state preparing -> state finished >Thread-15912892::DEBUG::2013-11-20 02:47:31,151::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912892::DEBUG::2013-11-20 02:47:31,151::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912892::DEBUG::2013-11-20 02:47:31,152::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912892::DEBUG::2013-11-20 02:47:31,153::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912892::DEBUG::2013-11-20 02:47:31,153::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912892::DEBUG::2013-11-20 02:47:31,154::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912892::DEBUG::2013-11-20 02:47:31,154::task::495::TaskManager.Task::(_debug) Task 18b43929-e141-491a-bace-04debc8fd802: ref 0 aborting False >Thread-15912892::INFO::2013-11-20 02:47:31,155::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115785859072', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115785859072', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:47:32,041::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:32,523::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.079343 s, 12.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:47:34,540::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:35,104::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0823106 s, 12.4 MB/s\n'; <rc> = 0 >Thread-15912894::INFO::2013-11-20 02:47:35,839::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912894::DEBUG::2013-11-20 02:47:35,840::task::495::TaskManager.Task::(_debug) Task c5f15aab-44d1-4d35-821f-20f99b15cb10: moving from state init -> state preparing >Thread-15912894::DEBUG::2013-11-20 02:47:35,841::task::495::TaskManager.Task::(_debug) Task c5f15aab-44d1-4d35-821f-20f99b15cb10: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000355005264282', 'lastCheck': 1384915653.2712369, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0010039806366', 'lastCheck': 1384915652.277986, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00103306770325', 'lastCheck': 1384915653.803283, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000432968139648', 'lastCheck': 1384915651.264478, 'valid': True, 'code': 0}} >Thread-15912894::DEBUG::2013-11-20 02:47:35,842::task::495::TaskManager.Task::(_debug) Task c5f15aab-44d1-4d35-821f-20f99b15cb10: moving from state preparing -> state finished >Thread-15912894::DEBUG::2013-11-20 02:47:35,842::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912894::DEBUG::2013-11-20 02:47:35,843::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912894::DEBUG::2013-11-20 02:47:35,843::task::495::TaskManager.Task::(_debug) Task c5f15aab-44d1-4d35-821f-20f99b15cb10: ref 0 aborting False >Thread-15912894::INFO::2013-11-20 02:47:35,844::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000355005264282', 'lastCheck': 1384915653.2712369, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0010039806366', 'lastCheck': 1384915652.277986, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00103306770325', 'lastCheck': 1384915653.803283, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000432968139648', 'lastCheck': 1384915651.264478, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:47:37,116::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:37,687::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0718353 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:47:39,703::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:40,271::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0583764 s, 17.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:47:42,282::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:42,758::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0736055 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:47:44,772::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:45,305::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0748272 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15912898::DEBUG::2013-11-20 02:47:47,271::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912898::INFO::2013-11-20 02:47:47,272::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912898::DEBUG::2013-11-20 02:47:47,273::task::495::TaskManager.Task::(_debug) Task 82c53c6e-9ee9-433d-8e17-bf6604fe0dfc: moving from state init -> state preparing >Thread-15912898::DEBUG::2013-11-20 02:47:47,274::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912898::DEBUG::2013-11-20 02:47:47,274::task::495::TaskManager.Task::(_debug) Task 82c53c6e-9ee9-433d-8e17-bf6604fe0dfc: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912898::DEBUG::2013-11-20 02:47:47,275::task::495::TaskManager.Task::(_debug) Task 82c53c6e-9ee9-433d-8e17-bf6604fe0dfc: moving from state preparing -> state finished >Thread-15912898::DEBUG::2013-11-20 02:47:47,276::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912898::DEBUG::2013-11-20 02:47:47,276::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912898::DEBUG::2013-11-20 02:47:47,277::task::495::TaskManager.Task::(_debug) Task 82c53c6e-9ee9-433d-8e17-bf6604fe0dfc: ref 0 aborting False >Thread-15912898::INFO::2013-11-20 02:47:47,277::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912899::DEBUG::2013-11-20 02:47:47,281::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912899::INFO::2013-11-20 02:47:47,282::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912899::DEBUG::2013-11-20 02:47:47,283::task::495::TaskManager.Task::(_debug) Task 1ab95e21-4ab8-445d-a39f-1f18b7386cdc: moving from state init -> state preparing >Thread-15912899::DEBUG::2013-11-20 02:47:47,284::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`233643ba-8c17-4474-8431-f067d909ccb8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912899::DEBUG::2013-11-20 02:47:47,284::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912899::DEBUG::2013-11-20 02:47:47,285::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912899::DEBUG::2013-11-20 02:47:47,286::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`233643ba-8c17-4474-8431-f067d909ccb8`::Granted request >Thread-15912899::DEBUG::2013-11-20 02:47:47,287::task::495::TaskManager.Task::(_debug) Task 1ab95e21-4ab8-445d-a39f-1f18b7386cdc: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912899::DEBUG::2013-11-20 02:47:47,287::task::495::TaskManager.Task::(_debug) Task 1ab95e21-4ab8-445d-a39f-1f18b7386cdc: ref 1 aborting False >Thread-15912899::WARNING::2013-11-20 02:47:47,289::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912899::DEBUG::2013-11-20 02:47:47,290::task::495::TaskManager.Task::(_debug) Task 1ab95e21-4ab8-445d-a39f-1f18b7386cdc: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115774324736', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115768033280', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912899::DEBUG::2013-11-20 02:47:47,291::task::495::TaskManager.Task::(_debug) Task 1ab95e21-4ab8-445d-a39f-1f18b7386cdc: moving from state preparing -> state finished >Thread-15912899::DEBUG::2013-11-20 02:47:47,291::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912899::DEBUG::2013-11-20 02:47:47,292::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912899::DEBUG::2013-11-20 02:47:47,293::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912899::DEBUG::2013-11-20 02:47:47,293::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912899::DEBUG::2013-11-20 02:47:47,294::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912899::DEBUG::2013-11-20 02:47:47,294::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912899::DEBUG::2013-11-20 02:47:47,295::task::495::TaskManager.Task::(_debug) Task 1ab95e21-4ab8-445d-a39f-1f18b7386cdc: ref 0 aborting False >Thread-15912899::INFO::2013-11-20 02:47:47,295::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115774324736', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115768033280', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:47:47,320::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:47,869::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.069993 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:47:49,885::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:50,383::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0745916 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:47:52,396::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:52,938::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0825444 s, 12.4 MB/s\n'; <rc> = 0 >Thread-15912902::INFO::2013-11-20 02:47:54,087::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912902::DEBUG::2013-11-20 02:47:54,087::task::495::TaskManager.Task::(_debug) Task 835c7ddc-46b8-45d8-ad69-ad94b2adacd3: moving from state init -> state preparing >Thread-15912902::DEBUG::2013-11-20 02:47:54,088::task::495::TaskManager.Task::(_debug) Task 835c7ddc-46b8-45d8-ad69-ad94b2adacd3: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000506162643433', 'lastCheck': 1384915673.6298251, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144195556641', 'lastCheck': 1384915672.634712, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00142002105713', 'lastCheck': 1384915663.9306581, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000601053237915', 'lastCheck': 1384915671.5508289, 'valid': True, 'code': 0}} >Thread-15912902::DEBUG::2013-11-20 02:47:54,088::task::495::TaskManager.Task::(_debug) Task 835c7ddc-46b8-45d8-ad69-ad94b2adacd3: moving from state preparing -> state finished >Thread-15912902::DEBUG::2013-11-20 02:47:54,088::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912902::DEBUG::2013-11-20 02:47:54,089::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912902::DEBUG::2013-11-20 02:47:54,089::task::495::TaskManager.Task::(_debug) Task 835c7ddc-46b8-45d8-ad69-ad94b2adacd3: ref 0 aborting False >Thread-15912902::INFO::2013-11-20 02:47:54,090::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000506162643433', 'lastCheck': 1384915673.6298251, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144195556641', 'lastCheck': 1384915672.634712, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00142002105713', 'lastCheck': 1384915663.9306581, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000601053237915', 'lastCheck': 1384915671.5508289, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:47:54,953::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:55,467::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0774528 s, 13.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:47:57,482::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:47:57,977::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0462891 s, 22.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:47:59,988::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:00,484::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0589922 s, 17.4 MB/s\n'; <rc> = 0 >Thread-15912904::DEBUG::2013-11-20 02:48:00,887::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912904::INFO::2013-11-20 02:48:00,888::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912904::DEBUG::2013-11-20 02:48:00,888::task::495::TaskManager.Task::(_debug) Task a538fad0-f296-4630-b29e-2586f9f9f828: moving from state init -> state preparing >Thread-15912904::DEBUG::2013-11-20 02:48:00,889::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912904::DEBUG::2013-11-20 02:48:00,889::task::495::TaskManager.Task::(_debug) Task a538fad0-f296-4630-b29e-2586f9f9f828: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912904::DEBUG::2013-11-20 02:48:00,889::task::495::TaskManager.Task::(_debug) Task a538fad0-f296-4630-b29e-2586f9f9f828: moving from state preparing -> state finished >Thread-15912904::DEBUG::2013-11-20 02:48:00,890::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912904::DEBUG::2013-11-20 02:48:00,890::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912904::DEBUG::2013-11-20 02:48:00,890::task::495::TaskManager.Task::(_debug) Task a538fad0-f296-4630-b29e-2586f9f9f828: ref 0 aborting False >Thread-15912904::INFO::2013-11-20 02:48:00,891::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912905::DEBUG::2013-11-20 02:48:00,895::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912905::INFO::2013-11-20 02:48:00,896::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912905::DEBUG::2013-11-20 02:48:00,896::task::495::TaskManager.Task::(_debug) Task 8bbea363-570e-49df-8870-00a63372e6dd: moving from state init -> state preparing >Thread-15912905::DEBUG::2013-11-20 02:48:00,897::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`637aee6f-b9d4-494f-9cd5-a5e6431aa574`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912905::DEBUG::2013-11-20 02:48:00,898::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912905::DEBUG::2013-11-20 02:48:00,899::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912905::DEBUG::2013-11-20 02:48:00,899::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`637aee6f-b9d4-494f-9cd5-a5e6431aa574`::Granted request >Thread-15912905::DEBUG::2013-11-20 02:48:00,900::task::495::TaskManager.Task::(_debug) Task 8bbea363-570e-49df-8870-00a63372e6dd: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912905::DEBUG::2013-11-20 02:48:00,901::task::495::TaskManager.Task::(_debug) Task 8bbea363-570e-49df-8870-00a63372e6dd: ref 1 aborting False >Thread-15912905::WARNING::2013-11-20 02:48:00,903::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912905::DEBUG::2013-11-20 02:48:00,904::task::495::TaskManager.Task::(_debug) Task 8bbea363-570e-49df-8870-00a63372e6dd: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115768033280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115768033280', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912905::DEBUG::2013-11-20 02:48:00,904::task::495::TaskManager.Task::(_debug) Task 8bbea363-570e-49df-8870-00a63372e6dd: moving from state preparing -> state finished >Thread-15912905::DEBUG::2013-11-20 02:48:00,905::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912905::DEBUG::2013-11-20 02:48:00,906::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912905::DEBUG::2013-11-20 02:48:00,906::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912905::DEBUG::2013-11-20 02:48:00,907::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912905::DEBUG::2013-11-20 02:48:00,907::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912905::DEBUG::2013-11-20 02:48:00,908::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912905::DEBUG::2013-11-20 02:48:00,908::task::495::TaskManager.Task::(_debug) Task 8bbea363-570e-49df-8870-00a63372e6dd: ref 0 aborting False >Thread-15912905::INFO::2013-11-20 02:48:00,909::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115768033280', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115768033280', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:48:02,494::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:03,001::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0749928 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:48:05,016::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 02:48:05,206::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:48:05,207::task::495::TaskManager.Task::(_debug) Task b0155bf9-3620-43b0-bf64-848332995c9f: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:48:05,209::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e35b3af3-cca0-42a4-9923-a523df63c811`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:48:05,209::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:48:05,210::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:48:05,210::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e35b3af3-cca0-42a4-9923-a523df63c811`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:48:05,211::task::495::TaskManager.Task::(_debug) Task b0155bf9-3620-43b0-bf64-848332995c9f: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:48:05,212::task::495::TaskManager.Task::(_debug) Task b0155bf9-3620-43b0-bf64-848332995c9f: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:48:05,213::task::495::TaskManager.Task::(_debug) Task b0155bf9-3620-43b0-bf64-848332995c9f: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:48:05,214::task::495::TaskManager.Task::(_debug) Task b0155bf9-3620-43b0-bf64-848332995c9f: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:48:05,214::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:48:05,214::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:48:05,215::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:48:05,215::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:48:05,216::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:48:05,216::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:48:05,217::task::495::TaskManager.Task::(_debug) Task b0155bf9-3620-43b0-bf64-848332995c9f: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:48:05,217::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 02:48:05,427::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0460429 s, 22.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:48:07,437::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:07,944::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0718424 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:48:09,958::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:10,422::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0602684 s, 17.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:48:12,434::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:12,901::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0585298 s, 17.5 MB/s\n'; <rc> = 0 >Thread-15912910::INFO::2013-11-20 02:48:13,720::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912910::DEBUG::2013-11-20 02:48:13,722::task::495::TaskManager.Task::(_debug) Task 9e302beb-bfbe-4f48-837f-6a7b4cdc12b9: moving from state init -> state preparing >Thread-15912910::DEBUG::2013-11-20 02:48:13,724::task::495::TaskManager.Task::(_debug) Task 9e302beb-bfbe-4f48-837f-6a7b4cdc12b9: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000957012176514', 'lastCheck': 1384915683.707859, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00185990333557', 'lastCheck': 1384915692.7397521, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0121049880981', 'lastCheck': 1384915684.2129991, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000515937805176', 'lastCheck': 1384915691.718925, 'valid': True, 'code': 0}} >Thread-15912910::DEBUG::2013-11-20 02:48:13,725::task::495::TaskManager.Task::(_debug) Task 9e302beb-bfbe-4f48-837f-6a7b4cdc12b9: moving from state preparing -> state finished >Thread-15912910::DEBUG::2013-11-20 02:48:13,726::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912910::DEBUG::2013-11-20 02:48:13,727::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912910::DEBUG::2013-11-20 02:48:13,727::task::495::TaskManager.Task::(_debug) Task 9e302beb-bfbe-4f48-837f-6a7b4cdc12b9: ref 0 aborting False >Thread-15912910::INFO::2013-11-20 02:48:13,728::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000957012176514', 'lastCheck': 1384915683.707859, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00185990333557', 'lastCheck': 1384915692.7397521, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0121049880981', 'lastCheck': 1384915684.2129991, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000515937805176', 'lastCheck': 1384915691.718925, 'valid': True, 'code': 0}} >Thread-15912912::DEBUG::2013-11-20 02:48:13,939::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912912::INFO::2013-11-20 02:48:13,940::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912912::DEBUG::2013-11-20 02:48:13,941::task::495::TaskManager.Task::(_debug) Task b3490019-88a8-44e2-a888-d1f45e4ce219: moving from state init -> state preparing >Thread-15912912::DEBUG::2013-11-20 02:48:13,942::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912912::DEBUG::2013-11-20 02:48:13,942::task::495::TaskManager.Task::(_debug) Task b3490019-88a8-44e2-a888-d1f45e4ce219: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912912::DEBUG::2013-11-20 02:48:13,943::task::495::TaskManager.Task::(_debug) Task b3490019-88a8-44e2-a888-d1f45e4ce219: moving from state preparing -> state finished >Thread-15912912::DEBUG::2013-11-20 02:48:13,944::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912912::DEBUG::2013-11-20 02:48:13,944::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912912::DEBUG::2013-11-20 02:48:13,945::task::495::TaskManager.Task::(_debug) Task b3490019-88a8-44e2-a888-d1f45e4ce219: ref 0 aborting False >Thread-15912912::INFO::2013-11-20 02:48:13,946::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912913::DEBUG::2013-11-20 02:48:13,951::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912913::INFO::2013-11-20 02:48:13,952::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912913::DEBUG::2013-11-20 02:48:13,952::task::495::TaskManager.Task::(_debug) Task 7f78aaa5-50fd-4d38-80c2-3f7bd94cfde5: moving from state init -> state preparing >Thread-15912913::DEBUG::2013-11-20 02:48:13,954::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ff693d3a-65cd-4b91-9119-d702a61bf76b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912913::DEBUG::2013-11-20 02:48:13,954::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912913::DEBUG::2013-11-20 02:48:13,955::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912913::DEBUG::2013-11-20 02:48:13,956::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ff693d3a-65cd-4b91-9119-d702a61bf76b`::Granted request >Thread-15912913::DEBUG::2013-11-20 02:48:13,957::task::495::TaskManager.Task::(_debug) Task 7f78aaa5-50fd-4d38-80c2-3f7bd94cfde5: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912913::DEBUG::2013-11-20 02:48:13,958::task::495::TaskManager.Task::(_debug) Task 7f78aaa5-50fd-4d38-80c2-3f7bd94cfde5: ref 1 aborting False >Thread-15912913::WARNING::2013-11-20 02:48:13,960::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912913::DEBUG::2013-11-20 02:48:13,961::task::495::TaskManager.Task::(_debug) Task 7f78aaa5-50fd-4d38-80c2-3f7bd94cfde5: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912913::DEBUG::2013-11-20 02:48:13,962::task::495::TaskManager.Task::(_debug) Task 7f78aaa5-50fd-4d38-80c2-3f7bd94cfde5: moving from state preparing -> state finished >Thread-15912913::DEBUG::2013-11-20 02:48:13,963::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912913::DEBUG::2013-11-20 02:48:13,963::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912913::DEBUG::2013-11-20 02:48:13,964::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912913::DEBUG::2013-11-20 02:48:13,965::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912913::DEBUG::2013-11-20 02:48:13,965::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912913::DEBUG::2013-11-20 02:48:13,966::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912913::DEBUG::2013-11-20 02:48:13,967::task::495::TaskManager.Task::(_debug) Task 7f78aaa5-50fd-4d38-80c2-3f7bd94cfde5: ref 0 aborting False >Thread-15912913::INFO::2013-11-20 02:48:13,967::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115765936128', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115763838976', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:48:14,916::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:15,430::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0645707 s, 15.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:48:17,442::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:17,958::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0694972 s, 14.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:48:19,973::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:20,444::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0438032 s, 23.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:48:22,454::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:22,987::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0728875 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:48:25,002::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:25,517::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0769763 s, 13.3 MB/s\n'; <rc> = 0 >Thread-15912916::DEBUG::2013-11-20 02:48:26,911::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912916::INFO::2013-11-20 02:48:26,912::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912916::DEBUG::2013-11-20 02:48:26,912::task::495::TaskManager.Task::(_debug) Task 88c979f3-ec82-49ca-8c42-f7b94cb4b99a: moving from state init -> state preparing >Thread-15912916::DEBUG::2013-11-20 02:48:26,913::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912916::DEBUG::2013-11-20 02:48:26,913::task::495::TaskManager.Task::(_debug) Task 88c979f3-ec82-49ca-8c42-f7b94cb4b99a: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912916::DEBUG::2013-11-20 02:48:26,913::task::495::TaskManager.Task::(_debug) Task 88c979f3-ec82-49ca-8c42-f7b94cb4b99a: moving from state preparing -> state finished >Thread-15912916::DEBUG::2013-11-20 02:48:26,914::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912916::DEBUG::2013-11-20 02:48:26,914::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912916::DEBUG::2013-11-20 02:48:26,914::task::495::TaskManager.Task::(_debug) Task 88c979f3-ec82-49ca-8c42-f7b94cb4b99a: ref 0 aborting False >Thread-15912916::INFO::2013-11-20 02:48:26,915::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912917::DEBUG::2013-11-20 02:48:26,918::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912917::INFO::2013-11-20 02:48:26,919::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912917::DEBUG::2013-11-20 02:48:26,919::task::495::TaskManager.Task::(_debug) Task 7354f00c-a1be-4a16-b7bb-32eb806d4d66: moving from state init -> state preparing >Thread-15912917::DEBUG::2013-11-20 02:48:26,920::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ba7bec4f-d4e3-446c-99a1-234455b888cc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912917::DEBUG::2013-11-20 02:48:26,920::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912917::DEBUG::2013-11-20 02:48:26,921::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912917::DEBUG::2013-11-20 02:48:26,921::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ba7bec4f-d4e3-446c-99a1-234455b888cc`::Granted request >Thread-15912917::DEBUG::2013-11-20 02:48:26,926::task::495::TaskManager.Task::(_debug) Task 7354f00c-a1be-4a16-b7bb-32eb806d4d66: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912917::DEBUG::2013-11-20 02:48:26,926::task::495::TaskManager.Task::(_debug) Task 7354f00c-a1be-4a16-b7bb-32eb806d4d66: ref 1 aborting False >Thread-15912917::WARNING::2013-11-20 02:48:26,927::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912917::DEBUG::2013-11-20 02:48:26,928::task::495::TaskManager.Task::(_debug) Task 7354f00c-a1be-4a16-b7bb-32eb806d4d66: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115761741824', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115764887552', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912917::DEBUG::2013-11-20 02:48:26,928::task::495::TaskManager.Task::(_debug) Task 7354f00c-a1be-4a16-b7bb-32eb806d4d66: moving from state preparing -> state finished >Thread-15912917::DEBUG::2013-11-20 02:48:26,929::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912917::DEBUG::2013-11-20 02:48:26,929::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912917::DEBUG::2013-11-20 02:48:26,929::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912917::DEBUG::2013-11-20 02:48:26,930::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912917::DEBUG::2013-11-20 02:48:26,930::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912917::DEBUG::2013-11-20 02:48:26,930::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912917::DEBUG::2013-11-20 02:48:26,931::task::495::TaskManager.Task::(_debug) Task 7354f00c-a1be-4a16-b7bb-32eb806d4d66: ref 0 aborting False >Thread-15912917::INFO::2013-11-20 02:48:26,931::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115761741824', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115764887552', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:48:27,533::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:28,076::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0712208 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:48:30,091::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:30,513::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0453064 s, 22.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:48:32,523::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:33,024::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0708747 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15912920::INFO::2013-11-20 02:48:33,043::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912920::DEBUG::2013-11-20 02:48:33,044::task::495::TaskManager.Task::(_debug) Task eccaea5c-8065-48e2-a598-444bc06a5d09: moving from state init -> state preparing >Thread-15912920::DEBUG::2013-11-20 02:48:33,045::task::495::TaskManager.Task::(_debug) Task eccaea5c-8065-48e2-a598-444bc06a5d09: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000381946563721', 'lastCheck': 1384915703.741992, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00163292884827', 'lastCheck': 1384915712.780148, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00176095962524', 'lastCheck': 1384915704.251327, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000546932220459', 'lastCheck': 1384915711.796504, 'valid': True, 'code': 0}} >Thread-15912920::DEBUG::2013-11-20 02:48:33,046::task::495::TaskManager.Task::(_debug) Task eccaea5c-8065-48e2-a598-444bc06a5d09: moving from state preparing -> state finished >Thread-15912920::DEBUG::2013-11-20 02:48:33,046::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912920::DEBUG::2013-11-20 02:48:33,047::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912920::DEBUG::2013-11-20 02:48:33,047::task::495::TaskManager.Task::(_debug) Task eccaea5c-8065-48e2-a598-444bc06a5d09: ref 0 aborting False >Thread-15912920::INFO::2013-11-20 02:48:33,048::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000381946563721', 'lastCheck': 1384915703.741992, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00163292884827', 'lastCheck': 1384915712.780148, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00176095962524', 'lastCheck': 1384915704.251327, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000546932220459', 'lastCheck': 1384915711.796504, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:48:35,042::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:35,573::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0729577 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:48:37,588::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:38,061::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0475533 s, 21.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:48:40,072::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:40,561::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0675272 s, 15.2 MB/s\n'; <rc> = 0 >Thread-42840::DEBUG::2013-11-20 02:48:41,808::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 02:48:42,430::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:48:42,577::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:43,111::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0739316 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:48:45,126::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:45,566::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0457288 s, 22.4 MB/s\n'; <rc> = 0 >Thread-15912925::DEBUG::2013-11-20 02:48:47,094::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912925::INFO::2013-11-20 02:48:47,095::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912925::DEBUG::2013-11-20 02:48:47,096::task::495::TaskManager.Task::(_debug) Task 18591aa5-7403-478f-b3c4-32bc3dc3a620: moving from state init -> state preparing >Thread-15912925::DEBUG::2013-11-20 02:48:47,097::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912925::DEBUG::2013-11-20 02:48:47,097::task::495::TaskManager.Task::(_debug) Task 18591aa5-7403-478f-b3c4-32bc3dc3a620: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912925::DEBUG::2013-11-20 02:48:47,098::task::495::TaskManager.Task::(_debug) Task 18591aa5-7403-478f-b3c4-32bc3dc3a620: moving from state preparing -> state finished >Thread-15912925::DEBUG::2013-11-20 02:48:47,098::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912925::DEBUG::2013-11-20 02:48:47,099::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912925::DEBUG::2013-11-20 02:48:47,099::task::495::TaskManager.Task::(_debug) Task 18591aa5-7403-478f-b3c4-32bc3dc3a620: ref 0 aborting False >Thread-15912925::INFO::2013-11-20 02:48:47,100::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912926::DEBUG::2013-11-20 02:48:47,104::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912926::INFO::2013-11-20 02:48:47,105::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912926::DEBUG::2013-11-20 02:48:47,105::task::495::TaskManager.Task::(_debug) Task d6c20981-d3ac-4e3c-9518-1584340dd6ce: moving from state init -> state preparing >Thread-15912926::DEBUG::2013-11-20 02:48:47,107::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3aa8c541-dc85-4240-9991-65358ba51633`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912926::DEBUG::2013-11-20 02:48:47,107::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912926::DEBUG::2013-11-20 02:48:47,107::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912926::DEBUG::2013-11-20 02:48:47,108::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3aa8c541-dc85-4240-9991-65358ba51633`::Granted request >Thread-15912926::DEBUG::2013-11-20 02:48:47,108::task::495::TaskManager.Task::(_debug) Task d6c20981-d3ac-4e3c-9518-1584340dd6ce: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912926::DEBUG::2013-11-20 02:48:47,109::task::495::TaskManager.Task::(_debug) Task d6c20981-d3ac-4e3c-9518-1584340dd6ce: ref 1 aborting False >Thread-15912926::WARNING::2013-11-20 02:48:47,110::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912926::DEBUG::2013-11-20 02:48:47,110::task::495::TaskManager.Task::(_debug) Task d6c20981-d3ac-4e3c-9518-1584340dd6ce: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912926::DEBUG::2013-11-20 02:48:47,111::task::495::TaskManager.Task::(_debug) Task d6c20981-d3ac-4e3c-9518-1584340dd6ce: moving from state preparing -> state finished >Thread-15912926::DEBUG::2013-11-20 02:48:47,111::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912926::DEBUG::2013-11-20 02:48:47,112::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912926::DEBUG::2013-11-20 02:48:47,112::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912926::DEBUG::2013-11-20 02:48:47,112::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912926::DEBUG::2013-11-20 02:48:47,113::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912926::DEBUG::2013-11-20 02:48:47,113::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912926::DEBUG::2013-11-20 02:48:47,113::task::495::TaskManager.Task::(_debug) Task d6c20981-d3ac-4e3c-9518-1584340dd6ce: ref 0 aborting False >Thread-15912926::INFO::2013-11-20 02:48:47,114::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:48:47,576::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:48,081::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0523581 s, 19.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:48:50,097::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:50,625::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0695586 s, 14.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:48:52,638::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912928::INFO::2013-11-20 02:48:52,913::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912928::DEBUG::2013-11-20 02:48:52,915::task::495::TaskManager.Task::(_debug) Task 9e158a02-a890-43ab-8a48-2d561ced011e: moving from state init -> state preparing >Thread-15912928::DEBUG::2013-11-20 02:48:52,916::task::495::TaskManager.Task::(_debug) Task 9e158a02-a890-43ab-8a48-2d561ced011e: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000432014465332', 'lastCheck': 1384915724.0141771, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150203704834', 'lastCheck': 1384915722.8505931, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.002277135849', 'lastCheck': 1384915724.3242149, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000415802001953', 'lastCheck': 1384915732.4435279, 'valid': True, 'code': 0}} >Thread-15912928::DEBUG::2013-11-20 02:48:52,917::task::495::TaskManager.Task::(_debug) Task 9e158a02-a890-43ab-8a48-2d561ced011e: moving from state preparing -> state finished >Thread-15912928::DEBUG::2013-11-20 02:48:52,918::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912928::DEBUG::2013-11-20 02:48:52,918::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912928::DEBUG::2013-11-20 02:48:52,919::task::495::TaskManager.Task::(_debug) Task 9e158a02-a890-43ab-8a48-2d561ced011e: ref 0 aborting False >Thread-15912928::INFO::2013-11-20 02:48:52,919::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000432014465332', 'lastCheck': 1384915724.0141771, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150203704834', 'lastCheck': 1384915722.8505931, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.002277135849', 'lastCheck': 1384915724.3242149, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000415802001953', 'lastCheck': 1384915732.4435279, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:48:53,207::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0738321 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:48:55,222::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:55,653::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0456371 s, 22.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:48:57,663::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:48:58,215::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.074524 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:49:00,231::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:00,768::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0751032 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:49:02,783::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:03,323::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.072361 s, 14.2 MB/s\n'; <rc> = 0 >Thread-15912933::DEBUG::2013-11-20 02:49:04,438::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912933::INFO::2013-11-20 02:49:04,440::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912933::DEBUG::2013-11-20 02:49:04,441::task::495::TaskManager.Task::(_debug) Task 59e09a00-f76e-4bcd-b34f-5f55e3b8297f: moving from state init -> state preparing >Thread-15912933::DEBUG::2013-11-20 02:49:04,441::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912933::DEBUG::2013-11-20 02:49:04,442::task::495::TaskManager.Task::(_debug) Task 59e09a00-f76e-4bcd-b34f-5f55e3b8297f: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912933::DEBUG::2013-11-20 02:49:04,443::task::495::TaskManager.Task::(_debug) Task 59e09a00-f76e-4bcd-b34f-5f55e3b8297f: moving from state preparing -> state finished >Thread-15912933::DEBUG::2013-11-20 02:49:04,443::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912933::DEBUG::2013-11-20 02:49:04,444::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912933::DEBUG::2013-11-20 02:49:04,444::task::495::TaskManager.Task::(_debug) Task 59e09a00-f76e-4bcd-b34f-5f55e3b8297f: ref 0 aborting False >Thread-15912933::INFO::2013-11-20 02:49:04,445::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912934::DEBUG::2013-11-20 02:49:04,565::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912934::INFO::2013-11-20 02:49:04,566::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912934::DEBUG::2013-11-20 02:49:04,566::task::495::TaskManager.Task::(_debug) Task c8144f8a-3f57-44c7-8dd8-62b60cdad1b3: moving from state init -> state preparing >Thread-15912934::DEBUG::2013-11-20 02:49:04,567::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f3542e8a-e2b7-4c46-b08d-f9e299c5f364`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912934::DEBUG::2013-11-20 02:49:04,567::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912934::DEBUG::2013-11-20 02:49:04,568::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912934::DEBUG::2013-11-20 02:49:04,568::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f3542e8a-e2b7-4c46-b08d-f9e299c5f364`::Granted request >Thread-15912934::DEBUG::2013-11-20 02:49:04,569::task::495::TaskManager.Task::(_debug) Task c8144f8a-3f57-44c7-8dd8-62b60cdad1b3: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912934::DEBUG::2013-11-20 02:49:04,569::task::495::TaskManager.Task::(_debug) Task c8144f8a-3f57-44c7-8dd8-62b60cdad1b3: ref 1 aborting False >Thread-15912934::WARNING::2013-11-20 02:49:04,570::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912934::DEBUG::2013-11-20 02:49:04,571::task::495::TaskManager.Task::(_debug) Task c8144f8a-3f57-44c7-8dd8-62b60cdad1b3: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912934::DEBUG::2013-11-20 02:49:04,571::task::495::TaskManager.Task::(_debug) Task c8144f8a-3f57-44c7-8dd8-62b60cdad1b3: moving from state preparing -> state finished >Thread-15912934::DEBUG::2013-11-20 02:49:04,572::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912934::DEBUG::2013-11-20 02:49:04,572::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912934::DEBUG::2013-11-20 02:49:04,573::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912934::DEBUG::2013-11-20 02:49:04,573::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912934::DEBUG::2013-11-20 02:49:04,573::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912934::DEBUG::2013-11-20 02:49:04,574::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912934::DEBUG::2013-11-20 02:49:04,574::task::495::TaskManager.Task::(_debug) Task c8144f8a-3f57-44c7-8dd8-62b60cdad1b3: ref 0 aborting False >Thread-15912934::INFO::2013-11-20 02:49:04,575::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:49:05,338::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 02:49:05,547::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:49:05,549::task::495::TaskManager.Task::(_debug) Task e6b34105-9a41-4e12-b8f1-a55370e01b44: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:49:05,551::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c712803f-1805-4b83-a27f-34c8e103a453`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:49:05,552::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:49:05,553::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:49:05,554::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c712803f-1805-4b83-a27f-34c8e103a453`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:49:05,555::task::495::TaskManager.Task::(_debug) Task e6b34105-9a41-4e12-b8f1-a55370e01b44: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:49:05,555::task::495::TaskManager.Task::(_debug) Task e6b34105-9a41-4e12-b8f1-a55370e01b44: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:49:05,556::task::495::TaskManager.Task::(_debug) Task e6b34105-9a41-4e12-b8f1-a55370e01b44: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:49:05,557::task::495::TaskManager.Task::(_debug) Task e6b34105-9a41-4e12-b8f1-a55370e01b44: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:49:05,557::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:49:05,558::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:49:05,559::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:49:05,559::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:49:05,560::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:49:05,560::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:49:05,561::task::495::TaskManager.Task::(_debug) Task e6b34105-9a41-4e12-b8f1-a55370e01b44: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:49:05,561::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 02:49:05,858::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0724051 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:49:07,872::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:08,397::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0844596 s, 12.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:49:10,412::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912936::INFO::2013-11-20 02:49:10,658::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912936::DEBUG::2013-11-20 02:49:10,659::task::495::TaskManager.Task::(_debug) Task 7d68041a-f284-4e10-a73f-562e0979b132: moving from state init -> state preparing >Thread-15912936::DEBUG::2013-11-20 02:49:10,659::task::495::TaskManager.Task::(_debug) Task 7d68041a-f284-4e10-a73f-562e0979b132: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000304937362671', 'lastCheck': 1384915744.039022, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146698951721', 'lastCheck': 1384915743.015085, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00161600112915', 'lastCheck': 1384915744.4627481, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000468969345093', 'lastCheck': 1384915742.4597249, 'valid': True, 'code': 0}} >Thread-15912936::DEBUG::2013-11-20 02:49:10,660::task::495::TaskManager.Task::(_debug) Task 7d68041a-f284-4e10-a73f-562e0979b132: moving from state preparing -> state finished >Thread-15912936::DEBUG::2013-11-20 02:49:10,661::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912936::DEBUG::2013-11-20 02:49:10,661::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912936::DEBUG::2013-11-20 02:49:10,662::task::495::TaskManager.Task::(_debug) Task 7d68041a-f284-4e10-a73f-562e0979b132: ref 0 aborting False >Thread-15912936::INFO::2013-11-20 02:49:10,662::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000304937362671', 'lastCheck': 1384915744.039022, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146698951721', 'lastCheck': 1384915743.015085, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00161600112915', 'lastCheck': 1384915744.4627481, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000468969345093', 'lastCheck': 1384915742.4597249, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:49:10,952::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0736196 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:49:12,967::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:13,485::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0725883 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:49:15,500::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:16,082::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.1084 s, 9.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:49:18,098::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:18,644::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0757122 s, 13.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:49:20,660::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:21,217::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0747138 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15912941::DEBUG::2013-11-20 02:49:22,075::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912941::INFO::2013-11-20 02:49:22,076::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912941::DEBUG::2013-11-20 02:49:22,077::task::495::TaskManager.Task::(_debug) Task cb5f460f-b180-4bcf-a164-a6cad1d6b9c2: moving from state init -> state preparing >Thread-15912941::DEBUG::2013-11-20 02:49:22,077::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912941::DEBUG::2013-11-20 02:49:22,078::task::495::TaskManager.Task::(_debug) Task cb5f460f-b180-4bcf-a164-a6cad1d6b9c2: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912941::DEBUG::2013-11-20 02:49:22,078::task::495::TaskManager.Task::(_debug) Task cb5f460f-b180-4bcf-a164-a6cad1d6b9c2: moving from state preparing -> state finished >Thread-15912941::DEBUG::2013-11-20 02:49:22,079::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912941::DEBUG::2013-11-20 02:49:22,079::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912941::DEBUG::2013-11-20 02:49:22,080::task::495::TaskManager.Task::(_debug) Task cb5f460f-b180-4bcf-a164-a6cad1d6b9c2: ref 0 aborting False >Thread-15912941::INFO::2013-11-20 02:49:22,080::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912942::DEBUG::2013-11-20 02:49:22,085::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912942::INFO::2013-11-20 02:49:22,086::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912942::DEBUG::2013-11-20 02:49:22,087::task::495::TaskManager.Task::(_debug) Task ea7ee8bf-8b11-46af-b1c2-c2f958950dc2: moving from state init -> state preparing >Thread-15912942::DEBUG::2013-11-20 02:49:22,089::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`01a04fd9-0160-4a87-9466-2da04b4a8cea`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912942::DEBUG::2013-11-20 02:49:22,089::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912942::DEBUG::2013-11-20 02:49:22,090::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912942::DEBUG::2013-11-20 02:49:22,091::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`01a04fd9-0160-4a87-9466-2da04b4a8cea`::Granted request >Thread-15912942::DEBUG::2013-11-20 02:49:22,093::task::495::TaskManager.Task::(_debug) Task ea7ee8bf-8b11-46af-b1c2-c2f958950dc2: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912942::DEBUG::2013-11-20 02:49:22,093::task::495::TaskManager.Task::(_debug) Task ea7ee8bf-8b11-46af-b1c2-c2f958950dc2: ref 1 aborting False >Thread-15912942::WARNING::2013-11-20 02:49:22,096::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912942::DEBUG::2013-11-20 02:49:22,097::task::495::TaskManager.Task::(_debug) Task ea7ee8bf-8b11-46af-b1c2-c2f958950dc2: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912942::DEBUG::2013-11-20 02:49:22,099::task::495::TaskManager.Task::(_debug) Task ea7ee8bf-8b11-46af-b1c2-c2f958950dc2: moving from state preparing -> state finished >Thread-15912942::DEBUG::2013-11-20 02:49:22,100::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912942::DEBUG::2013-11-20 02:49:22,101::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912942::DEBUG::2013-11-20 02:49:22,102::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912942::DEBUG::2013-11-20 02:49:22,103::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912942::DEBUG::2013-11-20 02:49:22,104::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912942::DEBUG::2013-11-20 02:49:22,105::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912942::DEBUG::2013-11-20 02:49:22,106::task::495::TaskManager.Task::(_debug) Task ea7ee8bf-8b11-46af-b1c2-c2f958950dc2: ref 0 aborting False >Thread-15912942::INFO::2013-11-20 02:49:22,107::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:49:23,233::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:23,763::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0751059 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:49:25,779::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:26,347::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0764128 s, 13.4 MB/s\n'; <rc> = 0 >Thread-15912944::INFO::2013-11-20 02:49:26,924::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912944::DEBUG::2013-11-20 02:49:26,925::task::495::TaskManager.Task::(_debug) Task 4f037215-8796-46ff-b2b9-68905fb15c60: moving from state init -> state preparing >Thread-15912944::DEBUG::2013-11-20 02:49:26,926::task::495::TaskManager.Task::(_debug) Task 4f037215-8796-46ff-b2b9-68905fb15c60: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000745058059692', 'lastCheck': 1384915764.456069, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0013439655304', 'lastCheck': 1384915763.471374, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00153684616089', 'lastCheck': 1384915764.893177, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00042200088501', 'lastCheck': 1384915762.8853631, 'valid': True, 'code': 0}} >Thread-15912944::DEBUG::2013-11-20 02:49:26,927::task::495::TaskManager.Task::(_debug) Task 4f037215-8796-46ff-b2b9-68905fb15c60: moving from state preparing -> state finished >Thread-15912944::DEBUG::2013-11-20 02:49:26,928::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912944::DEBUG::2013-11-20 02:49:26,928::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912944::DEBUG::2013-11-20 02:49:26,929::task::495::TaskManager.Task::(_debug) Task 4f037215-8796-46ff-b2b9-68905fb15c60: ref 0 aborting False >Thread-15912944::INFO::2013-11-20 02:49:26,929::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000745058059692', 'lastCheck': 1384915764.456069, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0013439655304', 'lastCheck': 1384915763.471374, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00153684616089', 'lastCheck': 1384915764.893177, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00042200088501', 'lastCheck': 1384915762.8853631, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:49:28,361::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:28,870::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0758551 s, 13.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:49:30,885::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:31,427::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0640628 s, 16.0 MB/s\n'; <rc> = 0 >Thread-15912947::DEBUG::2013-11-20 02:49:32,860::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912947::INFO::2013-11-20 02:49:32,861::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912947::DEBUG::2013-11-20 02:49:32,862::task::495::TaskManager.Task::(_debug) Task 55f67a0b-55bd-46c5-8bc5-27fcbcd14af1: moving from state init -> state preparing >Thread-15912947::DEBUG::2013-11-20 02:49:32,863::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912947::DEBUG::2013-11-20 02:49:32,863::task::495::TaskManager.Task::(_debug) Task 55f67a0b-55bd-46c5-8bc5-27fcbcd14af1: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912947::DEBUG::2013-11-20 02:49:32,864::task::495::TaskManager.Task::(_debug) Task 55f67a0b-55bd-46c5-8bc5-27fcbcd14af1: moving from state preparing -> state finished >Thread-15912947::DEBUG::2013-11-20 02:49:32,864::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912947::DEBUG::2013-11-20 02:49:32,864::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912947::DEBUG::2013-11-20 02:49:32,865::task::495::TaskManager.Task::(_debug) Task 55f67a0b-55bd-46c5-8bc5-27fcbcd14af1: ref 0 aborting False >Thread-15912947::INFO::2013-11-20 02:49:32,865::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912948::DEBUG::2013-11-20 02:49:32,869::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912948::INFO::2013-11-20 02:49:32,870::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912948::DEBUG::2013-11-20 02:49:32,870::task::495::TaskManager.Task::(_debug) Task ae99fd1d-4045-4811-95af-e92ff6cfb162: moving from state init -> state preparing >Thread-15912948::DEBUG::2013-11-20 02:49:32,871::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d01ca06d-d53f-42f3-a27c-527f34dda141`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912948::DEBUG::2013-11-20 02:49:32,872::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912948::DEBUG::2013-11-20 02:49:32,872::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912948::DEBUG::2013-11-20 02:49:32,873::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d01ca06d-d53f-42f3-a27c-527f34dda141`::Granted request >Thread-15912948::DEBUG::2013-11-20 02:49:32,874::task::495::TaskManager.Task::(_debug) Task ae99fd1d-4045-4811-95af-e92ff6cfb162: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912948::DEBUG::2013-11-20 02:49:32,874::task::495::TaskManager.Task::(_debug) Task ae99fd1d-4045-4811-95af-e92ff6cfb162: ref 1 aborting False >Thread-15912948::WARNING::2013-11-20 02:49:32,876::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912948::DEBUG::2013-11-20 02:49:32,876::task::495::TaskManager.Task::(_debug) Task ae99fd1d-4045-4811-95af-e92ff6cfb162: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912948::DEBUG::2013-11-20 02:49:32,877::task::495::TaskManager.Task::(_debug) Task ae99fd1d-4045-4811-95af-e92ff6cfb162: moving from state preparing -> state finished >Thread-15912948::DEBUG::2013-11-20 02:49:32,877::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912948::DEBUG::2013-11-20 02:49:32,878::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912948::DEBUG::2013-11-20 02:49:32,878::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912948::DEBUG::2013-11-20 02:49:32,879::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912948::DEBUG::2013-11-20 02:49:32,879::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912948::DEBUG::2013-11-20 02:49:32,879::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912948::DEBUG::2013-11-20 02:49:32,880::task::495::TaskManager.Task::(_debug) Task ae99fd1d-4045-4811-95af-e92ff6cfb162: ref 0 aborting False >Thread-15912948::INFO::2013-11-20 02:49:32,880::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:49:33,442::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:33,997::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0725894 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:49:36,013::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:36,544::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0728346 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:49:38,559::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:39,086::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0738477 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:49:41,100::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:41,613::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0737127 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:49:43,628::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:44,174::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0648222 s, 15.8 MB/s\n'; <rc> = 0 >Thread-15912952::INFO::2013-11-20 02:49:44,351::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912952::DEBUG::2013-11-20 02:49:44,352::task::495::TaskManager.Task::(_debug) Task e3d7ade4-548f-4bd9-a797-4d608caab67a: moving from state init -> state preparing >Thread-15912952::DEBUG::2013-11-20 02:49:44,353::task::495::TaskManager.Task::(_debug) Task e3d7ade4-548f-4bd9-a797-4d608caab67a: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000440835952759', 'lastCheck': 1384915774.6865561, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00125598907471', 'lastCheck': 1384915783.8640771, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00144410133362', 'lastCheck': 1384915775.1432559, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000496864318848', 'lastCheck': 1384915783.313942, 'valid': True, 'code': 0}} >Thread-15912952::DEBUG::2013-11-20 02:49:44,354::task::495::TaskManager.Task::(_debug) Task e3d7ade4-548f-4bd9-a797-4d608caab67a: moving from state preparing -> state finished >Thread-15912952::DEBUG::2013-11-20 02:49:44,354::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912952::DEBUG::2013-11-20 02:49:44,355::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912952::DEBUG::2013-11-20 02:49:44,356::task::495::TaskManager.Task::(_debug) Task e3d7ade4-548f-4bd9-a797-4d608caab67a: ref 0 aborting False >Thread-15912952::INFO::2013-11-20 02:49:44,356::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000440835952759', 'lastCheck': 1384915774.6865561, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00125598907471', 'lastCheck': 1384915783.8640771, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00144410133362', 'lastCheck': 1384915775.1432559, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000496864318848', 'lastCheck': 1384915783.313942, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:49:46,187::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:46,668::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0594174 s, 17.2 MB/s\n'; <rc> = 0 >Thread-15912954::DEBUG::2013-11-20 02:49:46,858::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912954::INFO::2013-11-20 02:49:46,859::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912954::DEBUG::2013-11-20 02:49:46,859::task::495::TaskManager.Task::(_debug) Task 97fee814-ea6e-49c1-8448-257bd4f8b24d: moving from state init -> state preparing >Thread-15912954::DEBUG::2013-11-20 02:49:46,861::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912954::DEBUG::2013-11-20 02:49:46,862::task::495::TaskManager.Task::(_debug) Task 97fee814-ea6e-49c1-8448-257bd4f8b24d: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912954::DEBUG::2013-11-20 02:49:46,862::task::495::TaskManager.Task::(_debug) Task 97fee814-ea6e-49c1-8448-257bd4f8b24d: moving from state preparing -> state finished >Thread-15912954::DEBUG::2013-11-20 02:49:46,863::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912954::DEBUG::2013-11-20 02:49:46,863::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912954::DEBUG::2013-11-20 02:49:46,863::task::495::TaskManager.Task::(_debug) Task 97fee814-ea6e-49c1-8448-257bd4f8b24d: ref 0 aborting False >Thread-15912954::INFO::2013-11-20 02:49:46,864::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912955::DEBUG::2013-11-20 02:49:46,867::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912955::INFO::2013-11-20 02:49:46,868::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912955::DEBUG::2013-11-20 02:49:46,868::task::495::TaskManager.Task::(_debug) Task bceaee84-dbf2-4d9e-8caa-b4f5cf155a86: moving from state init -> state preparing >Thread-15912955::DEBUG::2013-11-20 02:49:46,869::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fa1dc8b0-5669-44db-a8f0-b41921ae28af`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912955::DEBUG::2013-11-20 02:49:46,869::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912955::DEBUG::2013-11-20 02:49:46,870::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912955::DEBUG::2013-11-20 02:49:46,870::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fa1dc8b0-5669-44db-a8f0-b41921ae28af`::Granted request >Thread-15912955::DEBUG::2013-11-20 02:49:46,871::task::495::TaskManager.Task::(_debug) Task bceaee84-dbf2-4d9e-8caa-b4f5cf155a86: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912955::DEBUG::2013-11-20 02:49:46,871::task::495::TaskManager.Task::(_debug) Task bceaee84-dbf2-4d9e-8caa-b4f5cf155a86: ref 1 aborting False >Thread-15912955::WARNING::2013-11-20 02:49:46,872::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912955::DEBUG::2013-11-20 02:49:46,873::task::495::TaskManager.Task::(_debug) Task bceaee84-dbf2-4d9e-8caa-b4f5cf155a86: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912955::DEBUG::2013-11-20 02:49:46,874::task::495::TaskManager.Task::(_debug) Task bceaee84-dbf2-4d9e-8caa-b4f5cf155a86: moving from state preparing -> state finished >Thread-15912955::DEBUG::2013-11-20 02:49:46,874::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912955::DEBUG::2013-11-20 02:49:46,874::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912955::DEBUG::2013-11-20 02:49:46,875::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912955::DEBUG::2013-11-20 02:49:46,875::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912955::DEBUG::2013-11-20 02:49:46,875::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912955::DEBUG::2013-11-20 02:49:46,876::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912955::DEBUG::2013-11-20 02:49:46,876::task::495::TaskManager.Task::(_debug) Task bceaee84-dbf2-4d9e-8caa-b4f5cf155a86: ref 0 aborting False >Thread-15912955::INFO::2013-11-20 02:49:46,876::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:49:48,684::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:49,246::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100836 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:49:51,260::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:51,799::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0526211 s, 19.5 MB/s\n'; <rc> = 0 >Thread-15912956::DEBUG::2013-11-20 02:49:53,492::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912956::INFO::2013-11-20 02:49:53,494::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getIsoList, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912956::DEBUG::2013-11-20 02:49:53,495::task::495::TaskManager.Task::(_debug) Task 13e93aa2-4d9e-4e54-a24f-5b7e8d86f62d: moving from state init -> state preparing >Thread-15912956::DEBUG::2013-11-20 02:49:53,496::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`281d013a-516f-4374-800e-178204a5e851`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1032' at 'public_getIsoList' >Thread-15912956::DEBUG::2013-11-20 02:49:53,497::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912956::DEBUG::2013-11-20 02:49:53,498::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912956::DEBUG::2013-11-20 02:49:53,499::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`281d013a-516f-4374-800e-178204a5e851`::Granted request >Thread-15912956::DEBUG::2013-11-20 02:49:53,500::task::495::TaskManager.Task::(_debug) Task 13e93aa2-4d9e-4e54-a24f-5b7e8d86f62d: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912956::DEBUG::2013-11-20 02:49:53,501::task::495::TaskManager.Task::(_debug) Task 13e93aa2-4d9e-4e54-a24f-5b7e8d86f62d: ref 1 aborting False >Thread-15912956::DEBUG::2013-11-20 02:49:53,522::task::495::TaskManager.Task::(_debug) Task 13e93aa2-4d9e-4e54-a24f-5b7e8d86f62d: finished: {'isolist': ['Acronis Boot Image 31-10-2013.iso', 'Windows.2000.SP3.iso', 'ubuntu-13.04-desktop-amd64.iso', 'virtio-win-1.5.3.iso', 'rhev-tools-setup.iso', 'ubuntu-12.04.1-server-amd64.iso', 'ubuntu-13.04-server-amd64.iso', 'ubuntu-12.04-mini.iso', 'SW_DVD5_Win_Pro_8_64BIT_English_MLF_X18-16134.ISO', '.rhev-hypervisor6-6.4-20131016.0.iso', 'RHEV-toolsSetup_3.0_39.iso', 'SW_DVD5_Windows_Svr_2008w_SP2_English__x86_DC_EE_SE_X15-41116.ISO', 'SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO']} >Thread-15912956::DEBUG::2013-11-20 02:49:53,523::task::495::TaskManager.Task::(_debug) Task 13e93aa2-4d9e-4e54-a24f-5b7e8d86f62d: moving from state preparing -> state finished >Thread-15912956::DEBUG::2013-11-20 02:49:53,523::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912956::DEBUG::2013-11-20 02:49:53,524::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912956::DEBUG::2013-11-20 02:49:53,525::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912956::DEBUG::2013-11-20 02:49:53,525::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912956::DEBUG::2013-11-20 02:49:53,526::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912956::DEBUG::2013-11-20 02:49:53,526::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912956::DEBUG::2013-11-20 02:49:53,527::task::495::TaskManager.Task::(_debug) Task 13e93aa2-4d9e-4e54-a24f-5b7e8d86f62d: ref 0 aborting False >Thread-15912956::INFO::2013-11-20 02:49:53,528::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getIsoList, Return response: {'status': {'message': 'OK', 'code': 0}, 'isolist': ['Acronis Boot Image 31-10-2013.iso', 'Windows.2000.SP3.iso', 'ubuntu-13.04-desktop-amd64.iso', 'virtio-win-1.5.3.iso', 'rhev-tools-setup.iso', 'ubuntu-12.04.1-server-amd64.iso', 'ubuntu-13.04-server-amd64.iso', 'ubuntu-12.04-mini.iso', 'SW_DVD5_Win_Pro_8_64BIT_English_MLF_X18-16134.ISO', '.rhev-hypervisor6-6.4-20131016.0.iso', 'RHEV-toolsSetup_3.0_39.iso', 'SW_DVD5_Windows_Svr_2008w_SP2_English__x86_DC_EE_SE_X15-41116.ISO', 'SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO']} >Dummy-15912651::DEBUG::2013-11-20 02:49:53,814::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:54,396::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0743406 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:49:56,412::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:56,982::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0740924 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:49:58,997::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:49:59,516::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0755148 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:50:01,532::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:50:01,935::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.044599 s, 23.0 MB/s\n'; <rc> = 0 >Thread-15912960::DEBUG::2013-11-20 02:50:02,022::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912960::INFO::2013-11-20 02:50:02,024::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912960::DEBUG::2013-11-20 02:50:02,025::task::495::TaskManager.Task::(_debug) Task 5760bc15-80b7-423e-8bd2-4f3c5ac791e8: moving from state init -> state preparing >Thread-15912960::DEBUG::2013-11-20 02:50:02,035::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912960::DEBUG::2013-11-20 02:50:02,036::task::495::TaskManager.Task::(_debug) Task 5760bc15-80b7-423e-8bd2-4f3c5ac791e8: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912960::DEBUG::2013-11-20 02:50:02,037::task::495::TaskManager.Task::(_debug) Task 5760bc15-80b7-423e-8bd2-4f3c5ac791e8: moving from state preparing -> state finished >Thread-15912960::DEBUG::2013-11-20 02:50:02,037::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912960::DEBUG::2013-11-20 02:50:02,038::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912960::DEBUG::2013-11-20 02:50:02,038::task::495::TaskManager.Task::(_debug) Task 5760bc15-80b7-423e-8bd2-4f3c5ac791e8: ref 0 aborting False >Thread-15912960::INFO::2013-11-20 02:50:02,039::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912961::DEBUG::2013-11-20 02:50:02,044::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912961::INFO::2013-11-20 02:50:02,045::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912961::DEBUG::2013-11-20 02:50:02,045::task::495::TaskManager.Task::(_debug) Task 22aefeca-db44-4a4a-aba4-b26ce7321f22: moving from state init -> state preparing >Thread-15912961::DEBUG::2013-11-20 02:50:02,047::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`97196260-3678-4883-91a7-d7c45d37c351`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912961::DEBUG::2013-11-20 02:50:02,047::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912961::DEBUG::2013-11-20 02:50:02,048::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912961::DEBUG::2013-11-20 02:50:02,049::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`97196260-3678-4883-91a7-d7c45d37c351`::Granted request >Thread-15912961::DEBUG::2013-11-20 02:50:02,050::task::495::TaskManager.Task::(_debug) Task 22aefeca-db44-4a4a-aba4-b26ce7321f22: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912961::DEBUG::2013-11-20 02:50:02,051::task::495::TaskManager.Task::(_debug) Task 22aefeca-db44-4a4a-aba4-b26ce7321f22: ref 1 aborting False >Thread-15912961::WARNING::2013-11-20 02:50:02,053::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912961::DEBUG::2013-11-20 02:50:02,054::task::495::TaskManager.Task::(_debug) Task 22aefeca-db44-4a4a-aba4-b26ce7321f22: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912961::DEBUG::2013-11-20 02:50:02,055::task::495::TaskManager.Task::(_debug) Task 22aefeca-db44-4a4a-aba4-b26ce7321f22: moving from state preparing -> state finished >Thread-15912961::DEBUG::2013-11-20 02:50:02,055::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912961::DEBUG::2013-11-20 02:50:02,056::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912961::DEBUG::2013-11-20 02:50:02,057::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912961::DEBUG::2013-11-20 02:50:02,057::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912961::DEBUG::2013-11-20 02:50:02,058::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912961::DEBUG::2013-11-20 02:50:02,058::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912961::DEBUG::2013-11-20 02:50:02,059::task::495::TaskManager.Task::(_debug) Task 22aefeca-db44-4a4a-aba4-b26ce7321f22: ref 0 aborting False >Thread-15912961::INFO::2013-11-20 02:50:02,060::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115559366656', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:50:03,945::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:50:04,542::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.071034 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:50:06,557::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 02:50:06,790::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:50:06,791::task::495::TaskManager.Task::(_debug) Task 9afcd146-965f-4938-a69a-ddf5f4b89b30: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:50:06,793::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`bd2c11d9-bc4b-4bab-b630-2798f4f59188`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:50:06,793::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:50:06,794::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:50:06,794::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`bd2c11d9-bc4b-4bab-b630-2798f4f59188`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:50:06,795::task::495::TaskManager.Task::(_debug) Task 9afcd146-965f-4938-a69a-ddf5f4b89b30: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:50:06,795::task::495::TaskManager.Task::(_debug) Task 9afcd146-965f-4938-a69a-ddf5f4b89b30: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:50:06,796::task::495::TaskManager.Task::(_debug) Task 9afcd146-965f-4938-a69a-ddf5f4b89b30: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:50:06,796::task::495::TaskManager.Task::(_debug) Task 9afcd146-965f-4938-a69a-ddf5f4b89b30: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:50:06,796::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:50:06,797::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:50:06,797::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:50:06,798::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:50:06,798::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:50:06,799::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:50:06,799::task::495::TaskManager.Task::(_debug) Task 9afcd146-965f-4938-a69a-ddf5f4b89b30: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:50:06,800::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 02:50:07,117::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0736143 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15912963::INFO::2013-11-20 02:50:08,771::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912963::DEBUG::2013-11-20 02:50:08,771::task::495::TaskManager.Task::(_debug) Task 9f16b3d9-faf2-40c8-893c-2137f97a6f86: moving from state init -> state preparing >Thread-15912963::DEBUG::2013-11-20 02:50:08,772::task::495::TaskManager.Task::(_debug) Task 9f16b3d9-faf2-40c8-893c-2137f97a6f86: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000396966934204', 'lastCheck': 1384915805.24564, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146889686584', 'lastCheck': 1384915804.27742, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00161981582642', 'lastCheck': 1384915805.727365, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00048303604126', 'lastCheck': 1384915803.720016, 'valid': True, 'code': 0}} >Thread-15912963::DEBUG::2013-11-20 02:50:08,772::task::495::TaskManager.Task::(_debug) Task 9f16b3d9-faf2-40c8-893c-2137f97a6f86: moving from state preparing -> state finished >Thread-15912963::DEBUG::2013-11-20 02:50:08,773::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912963::DEBUG::2013-11-20 02:50:08,773::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912963::DEBUG::2013-11-20 02:50:08,774::task::495::TaskManager.Task::(_debug) Task 9f16b3d9-faf2-40c8-893c-2137f97a6f86: ref 0 aborting False >Thread-15912963::INFO::2013-11-20 02:50:08,774::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000396966934204', 'lastCheck': 1384915805.24564, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146889686584', 'lastCheck': 1384915804.27742, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00161981582642', 'lastCheck': 1384915805.727365, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00048303604126', 'lastCheck': 1384915803.720016, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:50:09,131::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:50:09,670::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0749571 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15912965::DEBUG::2013-11-20 02:50:09,824::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912965::INFO::2013-11-20 02:50:09,825::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getFloppyList, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912965::DEBUG::2013-11-20 02:50:09,826::task::495::TaskManager.Task::(_debug) Task 65653789-7fb0-40eb-ba55-c74ac6124e00: moving from state init -> state preparing >Thread-15912965::DEBUG::2013-11-20 02:50:09,835::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`752ec471-53ea-4b02-ba0c-d3878948c2e1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1032' at 'public_getIsoList' >Thread-15912965::DEBUG::2013-11-20 02:50:09,836::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912965::DEBUG::2013-11-20 02:50:09,837::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912965::DEBUG::2013-11-20 02:50:09,837::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`752ec471-53ea-4b02-ba0c-d3878948c2e1`::Granted request >Thread-15912965::DEBUG::2013-11-20 02:50:09,838::task::495::TaskManager.Task::(_debug) Task 65653789-7fb0-40eb-ba55-c74ac6124e00: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912965::DEBUG::2013-11-20 02:50:09,839::task::495::TaskManager.Task::(_debug) Task 65653789-7fb0-40eb-ba55-c74ac6124e00: ref 1 aborting False >Thread-15912965::DEBUG::2013-11-20 02:50:09,846::task::495::TaskManager.Task::(_debug) Task 65653789-7fb0-40eb-ba55-c74ac6124e00: finished: {'isolist': ['virtio-win-1.5.3.vfd']} >Thread-15912965::DEBUG::2013-11-20 02:50:09,846::task::495::TaskManager.Task::(_debug) Task 65653789-7fb0-40eb-ba55-c74ac6124e00: moving from state preparing -> state finished >Thread-15912965::DEBUG::2013-11-20 02:50:09,847::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912965::DEBUG::2013-11-20 02:50:09,847::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912965::DEBUG::2013-11-20 02:50:09,847::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912965::DEBUG::2013-11-20 02:50:09,848::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912965::DEBUG::2013-11-20 02:50:09,848::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912965::DEBUG::2013-11-20 02:50:09,848::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912965::DEBUG::2013-11-20 02:50:09,849::task::495::TaskManager.Task::(_debug) Task 65653789-7fb0-40eb-ba55-c74ac6124e00: ref 0 aborting False >Thread-15912965::INFO::2013-11-20 02:50:09,849::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getFloppyList, Return response: {'status': {'message': 'OK', 'code': 0}, 'isolist': ['virtio-win-1.5.3.vfd']} >Thread-15912966::DEBUG::2013-11-20 02:50:10,236::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912966::INFO::2013-11-20 02:50:10,238::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getIsoList, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912966::DEBUG::2013-11-20 02:50:10,238::task::495::TaskManager.Task::(_debug) Task b8a4e780-fa0f-4b47-9b98-335055bffd1a: moving from state init -> state preparing >Thread-15912966::DEBUG::2013-11-20 02:50:10,240::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5e9b0c2c-8f30-483a-883d-0307bcf2e3e6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1032' at 'public_getIsoList' >Thread-15912966::DEBUG::2013-11-20 02:50:10,240::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912966::DEBUG::2013-11-20 02:50:10,241::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912966::DEBUG::2013-11-20 02:50:10,243::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5e9b0c2c-8f30-483a-883d-0307bcf2e3e6`::Granted request >Thread-15912966::DEBUG::2013-11-20 02:50:10,244::task::495::TaskManager.Task::(_debug) Task b8a4e780-fa0f-4b47-9b98-335055bffd1a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912966::DEBUG::2013-11-20 02:50:10,245::task::495::TaskManager.Task::(_debug) Task b8a4e780-fa0f-4b47-9b98-335055bffd1a: ref 1 aborting False >Thread-15912966::DEBUG::2013-11-20 02:50:10,277::task::495::TaskManager.Task::(_debug) Task b8a4e780-fa0f-4b47-9b98-335055bffd1a: finished: {'isolist': ['Acronis Boot Image 31-10-2013.iso', 'Windows.2000.SP3.iso', 'ubuntu-13.04-desktop-amd64.iso', 'virtio-win-1.5.3.iso', 'rhev-tools-setup.iso', 'rhev-hypervisor6-6.4-20131016.0.iso', 'ubuntu-12.04.1-server-amd64.iso', 'ubuntu-13.04-server-amd64.iso', 'ubuntu-12.04-mini.iso', 'SW_DVD5_Win_Pro_8_64BIT_English_MLF_X18-16134.ISO', 'RHEV-toolsSetup_3.0_39.iso', 'SW_DVD5_Windows_Svr_2008w_SP2_English__x86_DC_EE_SE_X15-41116.ISO', 'SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO']} >Thread-15912966::DEBUG::2013-11-20 02:50:10,279::task::495::TaskManager.Task::(_debug) Task b8a4e780-fa0f-4b47-9b98-335055bffd1a: moving from state preparing -> state finished >Thread-15912966::DEBUG::2013-11-20 02:50:10,282::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912966::DEBUG::2013-11-20 02:50:10,283::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912966::DEBUG::2013-11-20 02:50:10,286::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912966::DEBUG::2013-11-20 02:50:10,287::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912966::DEBUG::2013-11-20 02:50:10,287::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912966::DEBUG::2013-11-20 02:50:10,287::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912966::DEBUG::2013-11-20 02:50:10,288::task::495::TaskManager.Task::(_debug) Task b8a4e780-fa0f-4b47-9b98-335055bffd1a: ref 0 aborting False >Thread-15912966::INFO::2013-11-20 02:50:10,289::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getIsoList, Return response: {'status': {'message': 'OK', 'code': 0}, 'isolist': ['Acronis Boot Image 31-10-2013.iso', 'Windows.2000.SP3.iso', 'ubuntu-13.04-desktop-amd64.iso', 'virtio-win-1.5.3.iso', 'rhev-tools-setup.iso', 'rhev-hypervisor6-6.4-20131016.0.iso', 'ubuntu-12.04.1-server-amd64.iso', 'ubuntu-13.04-server-amd64.iso', 'ubuntu-12.04-mini.iso', 'SW_DVD5_Win_Pro_8_64BIT_English_MLF_X18-16134.ISO', 'RHEV-toolsSetup_3.0_39.iso', 'SW_DVD5_Windows_Svr_2008w_SP2_English__x86_DC_EE_SE_X15-41116.ISO', 'SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO']} >Thread-15912967::DEBUG::2013-11-20 02:50:10,981::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912967::INFO::2013-11-20 02:50:10,982::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getFloppyList, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912967::DEBUG::2013-11-20 02:50:10,983::task::495::TaskManager.Task::(_debug) Task daea1a81-4d89-4c07-8a0b-716e9b5a68b0: moving from state init -> state preparing >Thread-15912967::DEBUG::2013-11-20 02:50:10,984::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ccf2345c-a27d-4736-9bd9-78c9ec3124bc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1032' at 'public_getIsoList' >Thread-15912967::DEBUG::2013-11-20 02:50:10,984::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912967::DEBUG::2013-11-20 02:50:10,985::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912967::DEBUG::2013-11-20 02:50:10,986::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ccf2345c-a27d-4736-9bd9-78c9ec3124bc`::Granted request >Thread-15912967::DEBUG::2013-11-20 02:50:10,987::task::495::TaskManager.Task::(_debug) Task daea1a81-4d89-4c07-8a0b-716e9b5a68b0: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912967::DEBUG::2013-11-20 02:50:10,987::task::495::TaskManager.Task::(_debug) Task daea1a81-4d89-4c07-8a0b-716e9b5a68b0: ref 1 aborting False >Thread-15912967::DEBUG::2013-11-20 02:50:10,991::task::495::TaskManager.Task::(_debug) Task daea1a81-4d89-4c07-8a0b-716e9b5a68b0: finished: {'isolist': ['virtio-win-1.5.3.vfd']} >Thread-15912967::DEBUG::2013-11-20 02:50:10,992::task::495::TaskManager.Task::(_debug) Task daea1a81-4d89-4c07-8a0b-716e9b5a68b0: moving from state preparing -> state finished >Thread-15912967::DEBUG::2013-11-20 02:50:10,992::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912967::DEBUG::2013-11-20 02:50:10,993::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912967::DEBUG::2013-11-20 02:50:10,993::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912967::DEBUG::2013-11-20 02:50:10,994::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912967::DEBUG::2013-11-20 02:50:10,994::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912967::DEBUG::2013-11-20 02:50:10,995::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912967::DEBUG::2013-11-20 02:50:10,995::task::495::TaskManager.Task::(_debug) Task daea1a81-4d89-4c07-8a0b-716e9b5a68b0: ref 0 aborting False >Thread-15912967::INFO::2013-11-20 02:50:10,996::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getFloppyList, Return response: {'status': {'message': 'OK', 'code': 0}, 'isolist': ['virtio-win-1.5.3.vfd']} >Dummy-15912651::DEBUG::2013-11-20 02:50:11,692::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:50:12,208::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0720437 s, 14.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:50:14,224::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:50:14,756::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0719609 s, 14.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:50:16,772::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:50:17,321::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0802047 s, 12.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:50:19,336::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912971::DEBUG::2013-11-20 02:50:19,826::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912971::INFO::2013-11-20 02:50:19,829::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912971::DEBUG::2013-11-20 02:50:19,829::task::495::TaskManager.Task::(_debug) Task 9a8b9ac0-7ea9-445a-9b1f-267cbbeae45e: moving from state init -> state preparing >Thread-15912971::DEBUG::2013-11-20 02:50:19,832::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912971::DEBUG::2013-11-20 02:50:19,832::task::495::TaskManager.Task::(_debug) Task 9a8b9ac0-7ea9-445a-9b1f-267cbbeae45e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912971::DEBUG::2013-11-20 02:50:19,834::task::495::TaskManager.Task::(_debug) Task 9a8b9ac0-7ea9-445a-9b1f-267cbbeae45e: moving from state preparing -> state finished >Thread-15912971::DEBUG::2013-11-20 02:50:19,835::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912971::DEBUG::2013-11-20 02:50:19,836::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912971::DEBUG::2013-11-20 02:50:19,837::task::495::TaskManager.Task::(_debug) Task 9a8b9ac0-7ea9-445a-9b1f-267cbbeae45e: ref 0 aborting False >Thread-15912971::INFO::2013-11-20 02:50:19,838::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912972::DEBUG::2013-11-20 02:50:19,844::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912972::INFO::2013-11-20 02:50:19,846::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912972::DEBUG::2013-11-20 02:50:19,847::task::495::TaskManager.Task::(_debug) Task 8804d2d0-4766-4a21-bde0-b8472c77d9e5: moving from state init -> state preparing >Thread-15912972::DEBUG::2013-11-20 02:50:19,849::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a38af112-98a9-4dc3-aa43-f1981c6fa186`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912972::DEBUG::2013-11-20 02:50:19,850::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912972::DEBUG::2013-11-20 02:50:19,853::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912972::DEBUG::2013-11-20 02:50:19,854::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a38af112-98a9-4dc3-aa43-f1981c6fa186`::Granted request >Thread-15912972::DEBUG::2013-11-20 02:50:19,857::task::495::TaskManager.Task::(_debug) Task 8804d2d0-4766-4a21-bde0-b8472c77d9e5: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912972::DEBUG::2013-11-20 02:50:19,859::task::495::TaskManager.Task::(_debug) Task 8804d2d0-4766-4a21-bde0-b8472c77d9e5: ref 1 aborting False >Dummy-15912651::DEBUG::2013-11-20 02:50:19,859::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0530862 s, 19.3 MB/s\n'; <rc> = 0 >Thread-15912972::WARNING::2013-11-20 02:50:19,862::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912972::DEBUG::2013-11-20 02:50:19,874::task::495::TaskManager.Task::(_debug) Task 8804d2d0-4766-4a21-bde0-b8472c77d9e5: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115370622976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115370622976', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912972::DEBUG::2013-11-20 02:50:19,875::task::495::TaskManager.Task::(_debug) Task 8804d2d0-4766-4a21-bde0-b8472c77d9e5: moving from state preparing -> state finished >Thread-15912972::DEBUG::2013-11-20 02:50:19,875::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912972::DEBUG::2013-11-20 02:50:19,876::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912972::DEBUG::2013-11-20 02:50:19,877::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912972::DEBUG::2013-11-20 02:50:19,877::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912972::DEBUG::2013-11-20 02:50:19,878::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912972::DEBUG::2013-11-20 02:50:19,878::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912972::DEBUG::2013-11-20 02:50:19,879::task::495::TaskManager.Task::(_debug) Task 8804d2d0-4766-4a21-bde0-b8472c77d9e5: ref 0 aborting False >Thread-15912972::INFO::2013-11-20 02:50:19,880::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115370622976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115370622976', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:50:21,876::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:50:22,396::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0599446 s, 17.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:50:24,406::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:50:24,950::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.072663 s, 14.1 MB/s\n'; <rc> = 0 >Thread-15912974::INFO::2013-11-20 02:50:25,619::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912974::DEBUG::2013-11-20 02:50:25,619::task::495::TaskManager.Task::(_debug) Task 87321db5-55df-459d-b569-3c3669ec18db: moving from state init -> state preparing >Thread-15912974::DEBUG::2013-11-20 02:50:25,620::task::495::TaskManager.Task::(_debug) Task 87321db5-55df-459d-b569-3c3669ec18db: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000339031219482', 'lastCheck': 1384915815.42944, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152492523193', 'lastCheck': 1384915824.633805, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0139360427856', 'lastCheck': 1384915815.9381449, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000513076782227', 'lastCheck': 1384915824.1183629, 'valid': True, 'code': 0}} >Thread-15912974::DEBUG::2013-11-20 02:50:25,620::task::495::TaskManager.Task::(_debug) Task 87321db5-55df-459d-b569-3c3669ec18db: moving from state preparing -> state finished >Thread-15912974::DEBUG::2013-11-20 02:50:25,621::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912974::DEBUG::2013-11-20 02:50:25,621::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912974::DEBUG::2013-11-20 02:50:25,621::task::495::TaskManager.Task::(_debug) Task 87321db5-55df-459d-b569-3c3669ec18db: ref 0 aborting False >Thread-15912974::INFO::2013-11-20 02:50:25,622::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000339031219482', 'lastCheck': 1384915815.42944, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152492523193', 'lastCheck': 1384915824.633805, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0139360427856', 'lastCheck': 1384915815.9381449, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000513076782227', 'lastCheck': 1384915824.1183629, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:50:26,965::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:50:27,483::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0742179 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:50:29,497::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:50:29,968::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0461171 s, 22.2 MB/s\n'; <rc> = 0 >Thread-15912976::DEBUG::2013-11-20 02:50:31,677::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912976::INFO::2013-11-20 02:50:31,678::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912976::DEBUG::2013-11-20 02:50:31,679::task::495::TaskManager.Task::(_debug) Task bd6fd4b4-c2e6-41a5-9c6c-94e4be15d9e8: moving from state init -> state preparing >Thread-15912976::DEBUG::2013-11-20 02:50:31,680::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912976::DEBUG::2013-11-20 02:50:31,680::task::495::TaskManager.Task::(_debug) Task bd6fd4b4-c2e6-41a5-9c6c-94e4be15d9e8: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912976::DEBUG::2013-11-20 02:50:31,681::task::495::TaskManager.Task::(_debug) Task bd6fd4b4-c2e6-41a5-9c6c-94e4be15d9e8: moving from state preparing -> state finished >Thread-15912976::DEBUG::2013-11-20 02:50:31,682::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912976::DEBUG::2013-11-20 02:50:31,682::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912976::DEBUG::2013-11-20 02:50:31,683::task::495::TaskManager.Task::(_debug) Task bd6fd4b4-c2e6-41a5-9c6c-94e4be15d9e8: ref 0 aborting False >Thread-15912976::INFO::2013-11-20 02:50:31,683::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912977::DEBUG::2013-11-20 02:50:31,688::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912977::INFO::2013-11-20 02:50:31,689::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912977::DEBUG::2013-11-20 02:50:31,689::task::495::TaskManager.Task::(_debug) Task 87772d03-2325-4216-b3d7-bfd08f57a520: moving from state init -> state preparing >Thread-15912977::DEBUG::2013-11-20 02:50:31,691::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f5bcf253-0519-4a75-b481-1670409bddeb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912977::DEBUG::2013-11-20 02:50:31,691::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912977::DEBUG::2013-11-20 02:50:31,692::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912977::DEBUG::2013-11-20 02:50:31,693::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f5bcf253-0519-4a75-b481-1670409bddeb`::Granted request >Thread-15912977::DEBUG::2013-11-20 02:50:31,694::task::495::TaskManager.Task::(_debug) Task 87772d03-2325-4216-b3d7-bfd08f57a520: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912977::DEBUG::2013-11-20 02:50:31,694::task::495::TaskManager.Task::(_debug) Task 87772d03-2325-4216-b3d7-bfd08f57a520: ref 1 aborting False >Thread-15912977::WARNING::2013-11-20 02:50:31,696::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912977::DEBUG::2013-11-20 02:50:31,697::task::495::TaskManager.Task::(_debug) Task 87772d03-2325-4216-b3d7-bfd08f57a520: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115370622976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115370622976', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912977::DEBUG::2013-11-20 02:50:31,698::task::495::TaskManager.Task::(_debug) Task 87772d03-2325-4216-b3d7-bfd08f57a520: moving from state preparing -> state finished >Thread-15912977::DEBUG::2013-11-20 02:50:31,699::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912977::DEBUG::2013-11-20 02:50:31,699::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912977::DEBUG::2013-11-20 02:50:31,700::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912977::DEBUG::2013-11-20 02:50:31,700::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912977::DEBUG::2013-11-20 02:50:31,701::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912977::DEBUG::2013-11-20 02:50:31,701::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912977::DEBUG::2013-11-20 02:50:31,702::task::495::TaskManager.Task::(_debug) Task 87772d03-2325-4216-b3d7-bfd08f57a520: ref 0 aborting False >Thread-15912977::INFO::2013-11-20 02:50:31,703::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115370622976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115370622976', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:50:31,982::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:50:32,499::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0743677 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15912978::DEBUG::2013-11-20 02:50:34,403::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912978::INFO::2013-11-20 02:50:34,404::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getIsoList, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912978::DEBUG::2013-11-20 02:50:34,405::task::495::TaskManager.Task::(_debug) Task cf3e47fa-1799-4383-a1d9-00cf86437374: moving from state init -> state preparing >Thread-15912978::DEBUG::2013-11-20 02:50:34,406::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`53623afa-fae5-45c8-a485-0ec742963590`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1032' at 'public_getIsoList' >Thread-15912978::DEBUG::2013-11-20 02:50:34,407::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912978::DEBUG::2013-11-20 02:50:34,407::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912978::DEBUG::2013-11-20 02:50:34,408::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`53623afa-fae5-45c8-a485-0ec742963590`::Granted request >Thread-15912978::DEBUG::2013-11-20 02:50:34,409::task::495::TaskManager.Task::(_debug) Task cf3e47fa-1799-4383-a1d9-00cf86437374: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912978::DEBUG::2013-11-20 02:50:34,410::task::495::TaskManager.Task::(_debug) Task cf3e47fa-1799-4383-a1d9-00cf86437374: ref 1 aborting False >Thread-15912978::DEBUG::2013-11-20 02:50:34,425::task::495::TaskManager.Task::(_debug) Task cf3e47fa-1799-4383-a1d9-00cf86437374: finished: {'isolist': ['Acronis Boot Image 31-10-2013.iso', 'Windows.2000.SP3.iso', 'ubuntu-13.04-desktop-amd64.iso', 'virtio-win-1.5.3.iso', 'rhev-tools-setup.iso', 'rhev-hypervisor6-6.4-20131016.0.iso', 'ubuntu-12.04.1-server-amd64.iso', 'ubuntu-13.04-server-amd64.iso', 'ubuntu-12.04-mini.iso', 'SW_DVD5_Win_Pro_8_64BIT_English_MLF_X18-16134.ISO', 'RHEV-toolsSetup_3.0_39.iso', 'SW_DVD5_Windows_Svr_2008w_SP2_English__x86_DC_EE_SE_X15-41116.ISO', 'SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO']} >Thread-15912978::DEBUG::2013-11-20 02:50:34,426::task::495::TaskManager.Task::(_debug) Task cf3e47fa-1799-4383-a1d9-00cf86437374: moving from state preparing -> state finished >Thread-15912978::DEBUG::2013-11-20 02:50:34,427::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912978::DEBUG::2013-11-20 02:50:34,427::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912978::DEBUG::2013-11-20 02:50:34,428::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912978::DEBUG::2013-11-20 02:50:34,429::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912978::DEBUG::2013-11-20 02:50:34,429::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912978::DEBUG::2013-11-20 02:50:34,430::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912978::DEBUG::2013-11-20 02:50:34,431::task::495::TaskManager.Task::(_debug) Task cf3e47fa-1799-4383-a1d9-00cf86437374: ref 0 aborting False >Thread-15912978::INFO::2013-11-20 02:50:34,431::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getIsoList, Return response: {'status': {'message': 'OK', 'code': 0}, 'isolist': ['Acronis Boot Image 31-10-2013.iso', 'Windows.2000.SP3.iso', 'ubuntu-13.04-desktop-amd64.iso', 'virtio-win-1.5.3.iso', 'rhev-tools-setup.iso', 'rhev-hypervisor6-6.4-20131016.0.iso', 'ubuntu-12.04.1-server-amd64.iso', 'ubuntu-13.04-server-amd64.iso', 'ubuntu-12.04-mini.iso', 'SW_DVD5_Win_Pro_8_64BIT_English_MLF_X18-16134.ISO', 'RHEV-toolsSetup_3.0_39.iso', 'SW_DVD5_Windows_Svr_2008w_SP2_English__x86_DC_EE_SE_X15-41116.ISO', 'SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO']} >Dummy-15912651::DEBUG::2013-11-20 02:50:34,514::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:50:35,066::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0708826 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15912980::DEBUG::2013-11-20 02:50:35,774::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912980::INFO::2013-11-20 02:50:35,775::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getFloppyList, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912980::DEBUG::2013-11-20 02:50:35,776::task::495::TaskManager.Task::(_debug) Task 7607c3ad-96c2-43ea-9099-1987174732d9: moving from state init -> state preparing >Thread-15912980::DEBUG::2013-11-20 02:50:35,777::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`4e473fb2-0927-442c-a084-3ca323568f03`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1032' at 'public_getIsoList' >Thread-15912980::DEBUG::2013-11-20 02:50:35,777::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912980::DEBUG::2013-11-20 02:50:35,778::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912980::DEBUG::2013-11-20 02:50:35,779::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`4e473fb2-0927-442c-a084-3ca323568f03`::Granted request >Thread-15912980::DEBUG::2013-11-20 02:50:35,780::task::495::TaskManager.Task::(_debug) Task 7607c3ad-96c2-43ea-9099-1987174732d9: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912980::DEBUG::2013-11-20 02:50:35,780::task::495::TaskManager.Task::(_debug) Task 7607c3ad-96c2-43ea-9099-1987174732d9: ref 1 aborting False >Thread-15912980::DEBUG::2013-11-20 02:50:35,785::task::495::TaskManager.Task::(_debug) Task 7607c3ad-96c2-43ea-9099-1987174732d9: finished: {'isolist': ['virtio-win-1.5.3.vfd']} >Thread-15912980::DEBUG::2013-11-20 02:50:35,785::task::495::TaskManager.Task::(_debug) Task 7607c3ad-96c2-43ea-9099-1987174732d9: moving from state preparing -> state finished >Thread-15912980::DEBUG::2013-11-20 02:50:35,786::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912980::DEBUG::2013-11-20 02:50:35,787::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912980::DEBUG::2013-11-20 02:50:35,787::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912980::DEBUG::2013-11-20 02:50:35,788::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912980::DEBUG::2013-11-20 02:50:35,788::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912980::DEBUG::2013-11-20 02:50:35,789::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912980::DEBUG::2013-11-20 02:50:35,790::task::495::TaskManager.Task::(_debug) Task 7607c3ad-96c2-43ea-9099-1987174732d9: ref 0 aborting False >Thread-15912980::INFO::2013-11-20 02:50:35,790::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getFloppyList, Return response: {'status': {'message': 'OK', 'code': 0}, 'isolist': ['virtio-win-1.5.3.vfd']} >Dummy-15912651::DEBUG::2013-11-20 02:50:37,082::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:50:37,605::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0699288 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:50:39,619::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:50:40,164::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0729455 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:50:42,177::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:50:42,727::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0751999 s, 13.6 MB/s\n'; <rc> = 0 >Thread-15912983::DEBUG::2013-11-20 02:50:43,051::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912983::INFO::2013-11-20 02:50:43,052::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912983::DEBUG::2013-11-20 02:50:43,053::task::495::TaskManager.Task::(_debug) Task a10376d1-62ef-4c7f-9dcb-363a43e7a936: moving from state init -> state preparing >Thread-15912983::DEBUG::2013-11-20 02:50:43,054::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912983::DEBUG::2013-11-20 02:50:43,054::task::495::TaskManager.Task::(_debug) Task a10376d1-62ef-4c7f-9dcb-363a43e7a936: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912983::DEBUG::2013-11-20 02:50:43,055::task::495::TaskManager.Task::(_debug) Task a10376d1-62ef-4c7f-9dcb-363a43e7a936: moving from state preparing -> state finished >Thread-15912983::DEBUG::2013-11-20 02:50:43,056::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912983::DEBUG::2013-11-20 02:50:43,056::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912983::DEBUG::2013-11-20 02:50:43,057::task::495::TaskManager.Task::(_debug) Task a10376d1-62ef-4c7f-9dcb-363a43e7a936: ref 0 aborting False >Thread-15912983::INFO::2013-11-20 02:50:43,057::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912984::DEBUG::2013-11-20 02:50:43,064::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912984::INFO::2013-11-20 02:50:43,065::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912984::DEBUG::2013-11-20 02:50:43,066::task::495::TaskManager.Task::(_debug) Task 41784cff-1825-46a1-b9a0-4013c1cc5cd5: moving from state init -> state preparing >Thread-15912984::DEBUG::2013-11-20 02:50:43,067::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5f87567b-4013-4f2e-bd3e-65b54cc8b920`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912984::DEBUG::2013-11-20 02:50:43,068::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912984::DEBUG::2013-11-20 02:50:43,069::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912984::DEBUG::2013-11-20 02:50:43,069::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5f87567b-4013-4f2e-bd3e-65b54cc8b920`::Granted request >Thread-15912984::DEBUG::2013-11-20 02:50:43,070::task::495::TaskManager.Task::(_debug) Task 41784cff-1825-46a1-b9a0-4013c1cc5cd5: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912984::DEBUG::2013-11-20 02:50:43,071::task::495::TaskManager.Task::(_debug) Task 41784cff-1825-46a1-b9a0-4013c1cc5cd5: ref 1 aborting False >Thread-15912984::WARNING::2013-11-20 02:50:43,073::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912984::DEBUG::2013-11-20 02:50:43,074::task::495::TaskManager.Task::(_debug) Task 41784cff-1825-46a1-b9a0-4013c1cc5cd5: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115370622976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115370622976', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912984::DEBUG::2013-11-20 02:50:43,074::task::495::TaskManager.Task::(_debug) Task 41784cff-1825-46a1-b9a0-4013c1cc5cd5: moving from state preparing -> state finished >Thread-15912984::DEBUG::2013-11-20 02:50:43,075::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912984::DEBUG::2013-11-20 02:50:43,076::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912984::DEBUG::2013-11-20 02:50:43,076::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912984::DEBUG::2013-11-20 02:50:43,077::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912984::DEBUG::2013-11-20 02:50:43,077::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912984::DEBUG::2013-11-20 02:50:43,078::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912984::DEBUG::2013-11-20 02:50:43,079::task::495::TaskManager.Task::(_debug) Task 41784cff-1825-46a1-b9a0-4013c1cc5cd5: ref 0 aborting False >Thread-15912984::INFO::2013-11-20 02:50:43,079::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115370622976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115370622976', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:50:44,742::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:50:45,272::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0746065 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:50:47,287::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:50:47,839::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0784803 s, 13.0 MB/s\n'; <rc> = 0 >Thread-15912986::INFO::2013-11-20 02:50:49,204::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912986::DEBUG::2013-11-20 02:50:49,205::task::495::TaskManager.Task::(_debug) Task 70d49232-bb31-45be-9d21-74d96e903f90: moving from state init -> state preparing >Thread-15912986::DEBUG::2013-11-20 02:50:49,206::task::495::TaskManager.Task::(_debug) Task 70d49232-bb31-45be-9d21-74d96e903f90: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000391960144043', 'lastCheck': 1384915845.971807, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0014820098877', 'lastCheck': 1384915844.9842529, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00178289413452', 'lastCheck': 1384915846.414712, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000411033630371', 'lastCheck': 1384915844.407243, 'valid': True, 'code': 0}} >Thread-15912986::DEBUG::2013-11-20 02:50:49,207::task::495::TaskManager.Task::(_debug) Task 70d49232-bb31-45be-9d21-74d96e903f90: moving from state preparing -> state finished >Thread-15912986::DEBUG::2013-11-20 02:50:49,208::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912986::DEBUG::2013-11-20 02:50:49,208::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912986::DEBUG::2013-11-20 02:50:49,209::task::495::TaskManager.Task::(_debug) Task 70d49232-bb31-45be-9d21-74d96e903f90: ref 0 aborting False >Thread-15912986::INFO::2013-11-20 02:50:49,210::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000391960144043', 'lastCheck': 1384915845.971807, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0014820098877', 'lastCheck': 1384915844.9842529, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00178289413452', 'lastCheck': 1384915846.414712, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000411033630371', 'lastCheck': 1384915844.407243, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:50:49,855::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:50:50,373::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.082751 s, 12.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:50:52,388::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:50:52,972::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0748192 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15912988::DEBUG::2013-11-20 02:50:53,831::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912988::INFO::2013-11-20 02:50:53,832::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912988::DEBUG::2013-11-20 02:50:53,833::task::495::TaskManager.Task::(_debug) Task ebcebeb3-823d-4d13-b0c4-bfe7dddc969e: moving from state init -> state preparing >Thread-15912988::DEBUG::2013-11-20 02:50:53,834::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912988::DEBUG::2013-11-20 02:50:53,834::task::495::TaskManager.Task::(_debug) Task ebcebeb3-823d-4d13-b0c4-bfe7dddc969e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912988::DEBUG::2013-11-20 02:50:53,835::task::495::TaskManager.Task::(_debug) Task ebcebeb3-823d-4d13-b0c4-bfe7dddc969e: moving from state preparing -> state finished >Thread-15912988::DEBUG::2013-11-20 02:50:53,836::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912988::DEBUG::2013-11-20 02:50:53,836::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912988::DEBUG::2013-11-20 02:50:53,837::task::495::TaskManager.Task::(_debug) Task ebcebeb3-823d-4d13-b0c4-bfe7dddc969e: ref 0 aborting False >Thread-15912988::INFO::2013-11-20 02:50:53,837::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912989::DEBUG::2013-11-20 02:50:53,846::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912989::INFO::2013-11-20 02:50:53,847::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912989::DEBUG::2013-11-20 02:50:53,847::task::495::TaskManager.Task::(_debug) Task d5ebe618-1278-4104-a464-ac09e2f38f88: moving from state init -> state preparing >Thread-15912989::DEBUG::2013-11-20 02:50:53,849::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6216bfd8-6efc-47af-bfe0-d81571a53e0b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912989::DEBUG::2013-11-20 02:50:53,850::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912989::DEBUG::2013-11-20 02:50:53,851::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912989::DEBUG::2013-11-20 02:50:53,852::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6216bfd8-6efc-47af-bfe0-d81571a53e0b`::Granted request >Thread-15912989::DEBUG::2013-11-20 02:50:53,854::task::495::TaskManager.Task::(_debug) Task d5ebe618-1278-4104-a464-ac09e2f38f88: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912989::DEBUG::2013-11-20 02:50:53,855::task::495::TaskManager.Task::(_debug) Task d5ebe618-1278-4104-a464-ac09e2f38f88: ref 1 aborting False >Thread-15912989::WARNING::2013-11-20 02:50:53,858::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912989::DEBUG::2013-11-20 02:50:53,860::task::495::TaskManager.Task::(_debug) Task d5ebe618-1278-4104-a464-ac09e2f38f88: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115370622976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115370622976', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912989::DEBUG::2013-11-20 02:50:53,861::task::495::TaskManager.Task::(_debug) Task d5ebe618-1278-4104-a464-ac09e2f38f88: moving from state preparing -> state finished >Thread-15912989::DEBUG::2013-11-20 02:50:53,862::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912989::DEBUG::2013-11-20 02:50:53,862::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912989::DEBUG::2013-11-20 02:50:53,864::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912989::DEBUG::2013-11-20 02:50:53,864::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912989::DEBUG::2013-11-20 02:50:53,865::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912989::DEBUG::2013-11-20 02:50:53,866::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912989::DEBUG::2013-11-20 02:50:53,867::task::495::TaskManager.Task::(_debug) Task d5ebe618-1278-4104-a464-ac09e2f38f88: ref 0 aborting False >Thread-15912989::INFO::2013-11-20 02:50:53,868::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115370622976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115370622976', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:50:54,986::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:50:55,529::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0734923 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:50:57,545::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:50:58,096::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.07443 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:51:00,110::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:00,643::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0733602 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:51:02,658::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:03,125::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0450394 s, 22.7 MB/s\n'; <rc> = 0 >Thread-15912993::DEBUG::2013-11-20 02:51:04,438::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912993::INFO::2013-11-20 02:51:04,440::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912993::DEBUG::2013-11-20 02:51:04,441::task::495::TaskManager.Task::(_debug) Task 6530d2f3-8579-445e-976d-c69e2a2010fc: moving from state init -> state preparing >Thread-15912993::DEBUG::2013-11-20 02:51:04,442::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912993::DEBUG::2013-11-20 02:51:04,442::task::495::TaskManager.Task::(_debug) Task 6530d2f3-8579-445e-976d-c69e2a2010fc: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912993::DEBUG::2013-11-20 02:51:04,443::task::495::TaskManager.Task::(_debug) Task 6530d2f3-8579-445e-976d-c69e2a2010fc: moving from state preparing -> state finished >Thread-15912993::DEBUG::2013-11-20 02:51:04,444::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912993::DEBUG::2013-11-20 02:51:04,444::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912993::DEBUG::2013-11-20 02:51:04,445::task::495::TaskManager.Task::(_debug) Task 6530d2f3-8579-445e-976d-c69e2a2010fc: ref 0 aborting False >Thread-15912993::INFO::2013-11-20 02:51:04,445::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912994::DEBUG::2013-11-20 02:51:04,450::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912994::INFO::2013-11-20 02:51:04,451::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912994::DEBUG::2013-11-20 02:51:04,451::task::495::TaskManager.Task::(_debug) Task f4babad3-9f64-4f4a-ae02-0d9b1f18bf29: moving from state init -> state preparing >Thread-15912994::DEBUG::2013-11-20 02:51:04,453::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`4e2f7a5d-4971-46cd-b77d-38b893f7e004`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15912994::DEBUG::2013-11-20 02:51:04,453::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15912994::DEBUG::2013-11-20 02:51:04,454::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15912994::DEBUG::2013-11-20 02:51:04,455::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`4e2f7a5d-4971-46cd-b77d-38b893f7e004`::Granted request >Thread-15912994::DEBUG::2013-11-20 02:51:04,456::task::495::TaskManager.Task::(_debug) Task f4babad3-9f64-4f4a-ae02-0d9b1f18bf29: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15912994::DEBUG::2013-11-20 02:51:04,456::task::495::TaskManager.Task::(_debug) Task f4babad3-9f64-4f4a-ae02-0d9b1f18bf29: ref 1 aborting False >Thread-15912994::WARNING::2013-11-20 02:51:04,458::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15912994::DEBUG::2013-11-20 02:51:04,459::task::495::TaskManager.Task::(_debug) Task f4babad3-9f64-4f4a-ae02-0d9b1f18bf29: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115370622976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115370622976', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912994::DEBUG::2013-11-20 02:51:04,460::task::495::TaskManager.Task::(_debug) Task f4babad3-9f64-4f4a-ae02-0d9b1f18bf29: moving from state preparing -> state finished >Thread-15912994::DEBUG::2013-11-20 02:51:04,461::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15912994::DEBUG::2013-11-20 02:51:04,461::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912994::DEBUG::2013-11-20 02:51:04,462::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15912994::DEBUG::2013-11-20 02:51:04,463::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912994::DEBUG::2013-11-20 02:51:04,463::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912994::DEBUG::2013-11-20 02:51:04,464::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912994::DEBUG::2013-11-20 02:51:04,465::task::495::TaskManager.Task::(_debug) Task f4babad3-9f64-4f4a-ae02-0d9b1f18bf29: ref 0 aborting False >Thread-15912994::INFO::2013-11-20 02:51:04,465::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115370622976', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115370622976', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:51:05,150::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:05,660::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0857552 s, 11.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:51:07,675::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 02:51:07,900::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:51:07,905::task::495::TaskManager.Task::(_debug) Task f1819e31-dfba-490e-8db0-be8973913c41: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:51:07,911::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a706504c-289f-4187-8297-d10f4133b2c5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912996::INFO::2013-11-20 02:51:07,914::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15912392::DEBUG::2013-11-20 02:51:07,915::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912996::DEBUG::2013-11-20 02:51:07,915::task::495::TaskManager.Task::(_debug) Task 00e30dac-2a20-4d0b-a73d-12b5666162ae: moving from state init -> state preparing >Thread-15912996::DEBUG::2013-11-20 02:51:07,916::task::495::TaskManager.Task::(_debug) Task 00e30dac-2a20-4d0b-a73d-12b5666162ae: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000488996505737', 'lastCheck': 1384915866.358283, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0215401649475', 'lastCheck': 1384915865.3875771, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00170207023621', 'lastCheck': 1384915866.8855109, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000532865524292', 'lastCheck': 1384915864.881032, 'valid': True, 'code': 0}} >Thread-15912392::DEBUG::2013-11-20 02:51:07,917::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912996::DEBUG::2013-11-20 02:51:07,918::task::495::TaskManager.Task::(_debug) Task 00e30dac-2a20-4d0b-a73d-12b5666162ae: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:51:07,918::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a706504c-289f-4187-8297-d10f4133b2c5`::Granted request >Thread-15912996::DEBUG::2013-11-20 02:51:07,919::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912996::DEBUG::2013-11-20 02:51:07,920::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:51:07,921::task::495::TaskManager.Task::(_debug) Task f1819e31-dfba-490e-8db0-be8973913c41: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912996::DEBUG::2013-11-20 02:51:07,921::task::495::TaskManager.Task::(_debug) Task 00e30dac-2a20-4d0b-a73d-12b5666162ae: ref 0 aborting False >Thread-15912392::DEBUG::2013-11-20 02:51:07,922::task::495::TaskManager.Task::(_debug) Task f1819e31-dfba-490e-8db0-be8973913c41: ref 1 aborting False >Thread-15912996::INFO::2013-11-20 02:51:07,922::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000488996505737', 'lastCheck': 1384915866.358283, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0215401649475', 'lastCheck': 1384915865.3875771, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00170207023621', 'lastCheck': 1384915866.8855109, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000532865524292', 'lastCheck': 1384915864.881032, 'valid': True, 'code': 0}} >Thread-15912392::DEBUG::2013-11-20 02:51:07,923::task::495::TaskManager.Task::(_debug) Task f1819e31-dfba-490e-8db0-be8973913c41: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:51:07,924::task::495::TaskManager.Task::(_debug) Task f1819e31-dfba-490e-8db0-be8973913c41: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:51:07,925::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:51:07,928::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:51:07,929::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:51:07,930::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:51:07,931::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:51:07,931::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:51:07,932::task::495::TaskManager.Task::(_debug) Task f1819e31-dfba-490e-8db0-be8973913c41: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:51:07,932::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 02:51:08,185::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0725852 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:51:10,200::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:10,684::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0754084 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:51:12,700::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:13,235::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0748208 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15912999::DEBUG::2013-11-20 02:51:14,642::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912999::INFO::2013-11-20 02:51:14,643::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912999::DEBUG::2013-11-20 02:51:14,644::task::495::TaskManager.Task::(_debug) Task 232309b9-e3ad-44e0-80c4-8ecc073091ba: moving from state init -> state preparing >Thread-15912999::DEBUG::2013-11-20 02:51:14,645::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912999::DEBUG::2013-11-20 02:51:14,645::task::495::TaskManager.Task::(_debug) Task 232309b9-e3ad-44e0-80c4-8ecc073091ba: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912999::DEBUG::2013-11-20 02:51:14,646::task::495::TaskManager.Task::(_debug) Task 232309b9-e3ad-44e0-80c4-8ecc073091ba: moving from state preparing -> state finished >Thread-15912999::DEBUG::2013-11-20 02:51:14,646::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912999::DEBUG::2013-11-20 02:51:14,647::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912999::DEBUG::2013-11-20 02:51:14,647::task::495::TaskManager.Task::(_debug) Task 232309b9-e3ad-44e0-80c4-8ecc073091ba: ref 0 aborting False >Thread-15912999::INFO::2013-11-20 02:51:14,648::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913000::DEBUG::2013-11-20 02:51:14,652::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913000::INFO::2013-11-20 02:51:14,653::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913000::DEBUG::2013-11-20 02:51:14,653::task::495::TaskManager.Task::(_debug) Task 2c428e33-8e88-4d56-afe1-2aba56953da7: moving from state init -> state preparing >Thread-15913000::DEBUG::2013-11-20 02:51:14,654::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`447de0e3-628b-4e82-a788-a1445a5e76d9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913000::DEBUG::2013-11-20 02:51:14,655::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913000::DEBUG::2013-11-20 02:51:14,655::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913000::DEBUG::2013-11-20 02:51:14,656::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`447de0e3-628b-4e82-a788-a1445a5e76d9`::Granted request >Thread-15913000::DEBUG::2013-11-20 02:51:14,657::task::495::TaskManager.Task::(_debug) Task 2c428e33-8e88-4d56-afe1-2aba56953da7: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913000::DEBUG::2013-11-20 02:51:14,658::task::495::TaskManager.Task::(_debug) Task 2c428e33-8e88-4d56-afe1-2aba56953da7: ref 1 aborting False >Thread-15913000::WARNING::2013-11-20 02:51:14,659::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913000::DEBUG::2013-11-20 02:51:14,661::task::495::TaskManager.Task::(_debug) Task 2c428e33-8e88-4d56-afe1-2aba56953da7: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115387400192', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115387400192', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913000::DEBUG::2013-11-20 02:51:14,662::task::495::TaskManager.Task::(_debug) Task 2c428e33-8e88-4d56-afe1-2aba56953da7: moving from state preparing -> state finished >Thread-15913000::DEBUG::2013-11-20 02:51:14,663::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913000::DEBUG::2013-11-20 02:51:14,664::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913000::DEBUG::2013-11-20 02:51:14,665::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913000::DEBUG::2013-11-20 02:51:14,666::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913000::DEBUG::2013-11-20 02:51:14,666::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913000::DEBUG::2013-11-20 02:51:14,667::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913000::DEBUG::2013-11-20 02:51:14,667::task::495::TaskManager.Task::(_debug) Task 2c428e33-8e88-4d56-afe1-2aba56953da7: ref 0 aborting False >Thread-15913000::INFO::2013-11-20 02:51:14,668::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115387400192', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115387400192', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:51:15,249::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:15,809::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.073965 s, 13.8 MB/s\n'; <rc> = 0 >Thread-42838::DEBUG::2013-11-20 02:51:16,491::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 02:51:17,141::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:51:17,825::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:18,256::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.044855 s, 22.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:51:20,266::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:20,765::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0781857 s, 13.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:51:22,781::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:23,248::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0450195 s, 22.7 MB/s\n'; <rc> = 0 >Thread-15913004::INFO::2013-11-20 02:51:23,708::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913004::DEBUG::2013-11-20 02:51:23,708::task::495::TaskManager.Task::(_debug) Task d71a8633-92e6-4262-ad93-d1b86bc6a312: moving from state init -> state preparing >Thread-15913004::DEBUG::2013-11-20 02:51:23,709::task::495::TaskManager.Task::(_debug) Task d71a8633-92e6-4262-ad93-d1b86bc6a312: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000319004058838', 'lastCheck': 1384915877.144207, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00219392776489', 'lastCheck': 1384915875.499285, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00141215324402', 'lastCheck': 1384915876.9372849, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000462055206299', 'lastCheck': 1384915874.9258659, 'valid': True, 'code': 0}} >Thread-15913004::DEBUG::2013-11-20 02:51:23,709::task::495::TaskManager.Task::(_debug) Task d71a8633-92e6-4262-ad93-d1b86bc6a312: moving from state preparing -> state finished >Thread-15913004::DEBUG::2013-11-20 02:51:23,710::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913004::DEBUG::2013-11-20 02:51:23,710::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913004::DEBUG::2013-11-20 02:51:23,710::task::495::TaskManager.Task::(_debug) Task d71a8633-92e6-4262-ad93-d1b86bc6a312: ref 0 aborting False >Thread-15913004::INFO::2013-11-20 02:51:23,711::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000319004058838', 'lastCheck': 1384915877.144207, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00219392776489', 'lastCheck': 1384915875.499285, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00141215324402', 'lastCheck': 1384915876.9372849, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000462055206299', 'lastCheck': 1384915874.9258659, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:51:25,258::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913006::DEBUG::2013-11-20 02:51:25,402::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913006::INFO::2013-11-20 02:51:25,404::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913006::DEBUG::2013-11-20 02:51:25,414::task::495::TaskManager.Task::(_debug) Task 508b6f43-1c7c-492e-9dee-6cbac9c10c7e: moving from state init -> state preparing >Thread-15913006::DEBUG::2013-11-20 02:51:25,415::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913006::DEBUG::2013-11-20 02:51:25,415::task::495::TaskManager.Task::(_debug) Task 508b6f43-1c7c-492e-9dee-6cbac9c10c7e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913006::DEBUG::2013-11-20 02:51:25,416::task::495::TaskManager.Task::(_debug) Task 508b6f43-1c7c-492e-9dee-6cbac9c10c7e: moving from state preparing -> state finished >Thread-15913006::DEBUG::2013-11-20 02:51:25,416::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913006::DEBUG::2013-11-20 02:51:25,416::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913006::DEBUG::2013-11-20 02:51:25,417::task::495::TaskManager.Task::(_debug) Task 508b6f43-1c7c-492e-9dee-6cbac9c10c7e: ref 0 aborting False >Thread-15913006::INFO::2013-11-20 02:51:25,417::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913007::DEBUG::2013-11-20 02:51:25,421::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913007::INFO::2013-11-20 02:51:25,422::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913007::DEBUG::2013-11-20 02:51:25,423::task::495::TaskManager.Task::(_debug) Task 74e269ed-14c4-458b-8617-6c4f199d97fc: moving from state init -> state preparing >Thread-15913007::DEBUG::2013-11-20 02:51:25,424::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e32f8dca-806c-4822-a68b-80c000bf1b36`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913007::DEBUG::2013-11-20 02:51:25,425::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913007::DEBUG::2013-11-20 02:51:25,426::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913007::DEBUG::2013-11-20 02:51:25,426::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e32f8dca-806c-4822-a68b-80c000bf1b36`::Granted request >Thread-15913007::DEBUG::2013-11-20 02:51:25,428::task::495::TaskManager.Task::(_debug) Task 74e269ed-14c4-458b-8617-6c4f199d97fc: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913007::DEBUG::2013-11-20 02:51:25,428::task::495::TaskManager.Task::(_debug) Task 74e269ed-14c4-458b-8617-6c4f199d97fc: ref 1 aborting False >Thread-15913007::WARNING::2013-11-20 02:51:25,430::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913007::DEBUG::2013-11-20 02:51:25,431::task::495::TaskManager.Task::(_debug) Task 74e269ed-14c4-458b-8617-6c4f199d97fc: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913007::DEBUG::2013-11-20 02:51:25,432::task::495::TaskManager.Task::(_debug) Task 74e269ed-14c4-458b-8617-6c4f199d97fc: moving from state preparing -> state finished >Thread-15913007::DEBUG::2013-11-20 02:51:25,432::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913007::DEBUG::2013-11-20 02:51:25,433::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913007::DEBUG::2013-11-20 02:51:25,434::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913007::DEBUG::2013-11-20 02:51:25,435::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913007::DEBUG::2013-11-20 02:51:25,435::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913007::DEBUG::2013-11-20 02:51:25,435::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913007::DEBUG::2013-11-20 02:51:25,436::task::495::TaskManager.Task::(_debug) Task 74e269ed-14c4-458b-8617-6c4f199d97fc: ref 0 aborting False >Thread-15913007::INFO::2013-11-20 02:51:25,437::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:51:25,591::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0374406 s, 27.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:51:27,601::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:28,118::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0752216 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:51:30,133::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:30,673::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0702479 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:51:32,688::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:33,227::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0779434 s, 13.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:51:35,238::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:35,696::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.045861 s, 22.3 MB/s\n'; <rc> = 0 >Thread-15913011::DEBUG::2013-11-20 02:51:36,172::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913011::INFO::2013-11-20 02:51:36,173::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913011::DEBUG::2013-11-20 02:51:36,174::task::495::TaskManager.Task::(_debug) Task 039bd3db-2603-4503-b22b-22a40fbb0770: moving from state init -> state preparing >Thread-15913011::DEBUG::2013-11-20 02:51:36,175::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913011::DEBUG::2013-11-20 02:51:36,175::task::495::TaskManager.Task::(_debug) Task 039bd3db-2603-4503-b22b-22a40fbb0770: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913011::DEBUG::2013-11-20 02:51:36,176::task::495::TaskManager.Task::(_debug) Task 039bd3db-2603-4503-b22b-22a40fbb0770: moving from state preparing -> state finished >Thread-15913011::DEBUG::2013-11-20 02:51:36,176::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913011::DEBUG::2013-11-20 02:51:36,177::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913011::DEBUG::2013-11-20 02:51:36,177::task::495::TaskManager.Task::(_debug) Task 039bd3db-2603-4503-b22b-22a40fbb0770: ref 0 aborting False >Thread-15913011::INFO::2013-11-20 02:51:36,178::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913012::DEBUG::2013-11-20 02:51:36,182::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913012::INFO::2013-11-20 02:51:36,183::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913012::DEBUG::2013-11-20 02:51:36,184::task::495::TaskManager.Task::(_debug) Task 113f9d53-a7d4-4c13-8128-9b3933245d25: moving from state init -> state preparing >Thread-15913012::DEBUG::2013-11-20 02:51:36,185::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c3ccb082-cea4-4735-87d3-286fba701a53`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913012::DEBUG::2013-11-20 02:51:36,186::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913012::DEBUG::2013-11-20 02:51:36,186::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913012::DEBUG::2013-11-20 02:51:36,187::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c3ccb082-cea4-4735-87d3-286fba701a53`::Granted request >Thread-15913012::DEBUG::2013-11-20 02:51:36,188::task::495::TaskManager.Task::(_debug) Task 113f9d53-a7d4-4c13-8128-9b3933245d25: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913012::DEBUG::2013-11-20 02:51:36,188::task::495::TaskManager.Task::(_debug) Task 113f9d53-a7d4-4c13-8128-9b3933245d25: ref 1 aborting False >Thread-15913012::WARNING::2013-11-20 02:51:36,190::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913012::DEBUG::2013-11-20 02:51:36,191::task::495::TaskManager.Task::(_debug) Task 113f9d53-a7d4-4c13-8128-9b3933245d25: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115405225984', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913012::DEBUG::2013-11-20 02:51:36,192::task::495::TaskManager.Task::(_debug) Task 113f9d53-a7d4-4c13-8128-9b3933245d25: moving from state preparing -> state finished >Thread-15913012::DEBUG::2013-11-20 02:51:36,193::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913012::DEBUG::2013-11-20 02:51:36,193::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913012::DEBUG::2013-11-20 02:51:36,194::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913012::DEBUG::2013-11-20 02:51:36,194::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913012::DEBUG::2013-11-20 02:51:36,195::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913012::DEBUG::2013-11-20 02:51:36,195::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913012::DEBUG::2013-11-20 02:51:36,196::task::495::TaskManager.Task::(_debug) Task 113f9d53-a7d4-4c13-8128-9b3933245d25: ref 0 aborting False >Thread-15913012::INFO::2013-11-20 02:51:36,197::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115405225984', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:51:37,707::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:38,160::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0456979 s, 22.4 MB/s\n'; <rc> = 0 >Thread-15913014::INFO::2013-11-20 02:51:39,420::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913014::DEBUG::2013-11-20 02:51:39,421::task::495::TaskManager.Task::(_debug) Task 2a0a2b40-3b8f-474c-a522-74a212838566: moving from state init -> state preparing >Thread-15913014::DEBUG::2013-11-20 02:51:39,421::task::495::TaskManager.Task::(_debug) Task 2a0a2b40-3b8f-474c-a522-74a212838566: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000358104705811', 'lastCheck': 1384915897.432658, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147008895874', 'lastCheck': 1384915895.9054141, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00116896629333', 'lastCheck': 1384915897.0758171, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000489950180054', 'lastCheck': 1384915895.0517759, 'valid': True, 'code': 0}} >Thread-15913014::DEBUG::2013-11-20 02:51:39,422::task::495::TaskManager.Task::(_debug) Task 2a0a2b40-3b8f-474c-a522-74a212838566: moving from state preparing -> state finished >Thread-15913014::DEBUG::2013-11-20 02:51:39,422::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913014::DEBUG::2013-11-20 02:51:39,422::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913014::DEBUG::2013-11-20 02:51:39,423::task::495::TaskManager.Task::(_debug) Task 2a0a2b40-3b8f-474c-a522-74a212838566: ref 0 aborting False >Thread-15913014::INFO::2013-11-20 02:51:39,423::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000358104705811', 'lastCheck': 1384915897.432658, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147008895874', 'lastCheck': 1384915895.9054141, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00116896629333', 'lastCheck': 1384915897.0758171, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000489950180054', 'lastCheck': 1384915895.0517759, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:51:40,170::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:40,696::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0591702 s, 17.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:51:42,715::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:43,239::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0598479 s, 17.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:51:45,253::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:45,770::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0638774 s, 16.0 MB/s\n'; <rc> = 0 >Thread-15913018::DEBUG::2013-11-20 02:51:46,743::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913018::INFO::2013-11-20 02:51:46,744::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913018::DEBUG::2013-11-20 02:51:46,745::task::495::TaskManager.Task::(_debug) Task cad36d53-9112-43af-b285-ae4a9f885737: moving from state init -> state preparing >Thread-15913018::DEBUG::2013-11-20 02:51:46,746::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913018::DEBUG::2013-11-20 02:51:46,746::task::495::TaskManager.Task::(_debug) Task cad36d53-9112-43af-b285-ae4a9f885737: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913018::DEBUG::2013-11-20 02:51:46,747::task::495::TaskManager.Task::(_debug) Task cad36d53-9112-43af-b285-ae4a9f885737: moving from state preparing -> state finished >Thread-15913018::DEBUG::2013-11-20 02:51:46,748::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913018::DEBUG::2013-11-20 02:51:46,748::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913018::DEBUG::2013-11-20 02:51:46,749::task::495::TaskManager.Task::(_debug) Task cad36d53-9112-43af-b285-ae4a9f885737: ref 0 aborting False >Thread-15913018::INFO::2013-11-20 02:51:46,749::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913019::DEBUG::2013-11-20 02:51:46,754::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913019::INFO::2013-11-20 02:51:46,755::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913019::DEBUG::2013-11-20 02:51:46,756::task::495::TaskManager.Task::(_debug) Task 39c464d8-c62a-4810-8ad2-1fb5e592a7de: moving from state init -> state preparing >Thread-15913019::DEBUG::2013-11-20 02:51:46,758::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`754ed8b4-afac-4c9e-9952-3cde9c9ddc63`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913019::DEBUG::2013-11-20 02:51:46,758::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913019::DEBUG::2013-11-20 02:51:46,759::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913019::DEBUG::2013-11-20 02:51:46,760::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`754ed8b4-afac-4c9e-9952-3cde9c9ddc63`::Granted request >Thread-15913019::DEBUG::2013-11-20 02:51:46,761::task::495::TaskManager.Task::(_debug) Task 39c464d8-c62a-4810-8ad2-1fb5e592a7de: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913019::DEBUG::2013-11-20 02:51:46,761::task::495::TaskManager.Task::(_debug) Task 39c464d8-c62a-4810-8ad2-1fb5e592a7de: ref 1 aborting False >Thread-15913019::WARNING::2013-11-20 02:51:46,764::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913019::DEBUG::2013-11-20 02:51:46,765::task::495::TaskManager.Task::(_debug) Task 39c464d8-c62a-4810-8ad2-1fb5e592a7de: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115405225984', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115405225984', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913019::DEBUG::2013-11-20 02:51:46,767::task::495::TaskManager.Task::(_debug) Task 39c464d8-c62a-4810-8ad2-1fb5e592a7de: moving from state preparing -> state finished >Thread-15913019::DEBUG::2013-11-20 02:51:46,768::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913019::DEBUG::2013-11-20 02:51:46,769::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913019::DEBUG::2013-11-20 02:51:46,769::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913019::DEBUG::2013-11-20 02:51:46,770::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913019::DEBUG::2013-11-20 02:51:46,771::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913019::DEBUG::2013-11-20 02:51:46,771::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913019::DEBUG::2013-11-20 02:51:46,772::task::495::TaskManager.Task::(_debug) Task 39c464d8-c62a-4810-8ad2-1fb5e592a7de: ref 0 aborting False >Thread-15913019::INFO::2013-11-20 02:51:46,773::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115405225984', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115405225984', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:51:47,780::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:48,266::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0699758 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:51:50,281::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:50,739::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.051175 s, 20.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:51:52,749::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:53,326::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0810723 s, 12.6 MB/s\n'; <rc> = 0 >Thread-15913022::INFO::2013-11-20 02:51:55,188::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913022::DEBUG::2013-11-20 02:51:55,189::task::495::TaskManager.Task::(_debug) Task 41534ae6-6ccf-44aa-8ff2-40feba14f2a9: moving from state init -> state preparing >Thread-15913022::DEBUG::2013-11-20 02:51:55,190::task::495::TaskManager.Task::(_debug) Task 41534ae6-6ccf-44aa-8ff2-40feba14f2a9: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000339031219482', 'lastCheck': 1384915907.4881401, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00815320014954', 'lastCheck': 1384915905.965147, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00165319442749', 'lastCheck': 1384915907.0908949, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000507831573486', 'lastCheck': 1384915915.0746379, 'valid': True, 'code': 0}} >Thread-15913022::DEBUG::2013-11-20 02:51:55,191::task::495::TaskManager.Task::(_debug) Task 41534ae6-6ccf-44aa-8ff2-40feba14f2a9: moving from state preparing -> state finished >Thread-15913022::DEBUG::2013-11-20 02:51:55,191::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913022::DEBUG::2013-11-20 02:51:55,192::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913022::DEBUG::2013-11-20 02:51:55,192::task::495::TaskManager.Task::(_debug) Task 41534ae6-6ccf-44aa-8ff2-40feba14f2a9: ref 0 aborting False >Thread-15913022::INFO::2013-11-20 02:51:55,193::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000339031219482', 'lastCheck': 1384915907.4881401, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00815320014954', 'lastCheck': 1384915905.965147, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00165319442749', 'lastCheck': 1384915907.0908949, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000507831573486', 'lastCheck': 1384915915.0746379, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:51:55,342::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:55,892::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0729512 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15913024::DEBUG::2013-11-20 02:51:56,938::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913024::INFO::2013-11-20 02:51:56,940::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913024::DEBUG::2013-11-20 02:51:56,940::task::495::TaskManager.Task::(_debug) Task 09c29049-7eeb-421d-b671-52faddbe1a0e: moving from state init -> state preparing >Thread-15913024::DEBUG::2013-11-20 02:51:56,941::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913024::DEBUG::2013-11-20 02:51:56,942::task::495::TaskManager.Task::(_debug) Task 09c29049-7eeb-421d-b671-52faddbe1a0e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913024::DEBUG::2013-11-20 02:51:56,942::task::495::TaskManager.Task::(_debug) Task 09c29049-7eeb-421d-b671-52faddbe1a0e: moving from state preparing -> state finished >Thread-15913024::DEBUG::2013-11-20 02:51:56,943::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913024::DEBUG::2013-11-20 02:51:56,943::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913024::DEBUG::2013-11-20 02:51:56,944::task::495::TaskManager.Task::(_debug) Task 09c29049-7eeb-421d-b671-52faddbe1a0e: ref 0 aborting False >Thread-15913024::INFO::2013-11-20 02:51:56,944::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913025::DEBUG::2013-11-20 02:51:56,949::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913025::INFO::2013-11-20 02:51:56,950::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913025::DEBUG::2013-11-20 02:51:56,951::task::495::TaskManager.Task::(_debug) Task 68a3b8cf-7045-4349-87f2-220edc0081af: moving from state init -> state preparing >Thread-15913025::DEBUG::2013-11-20 02:51:56,952::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e3b2e296-120d-4d68-9dd3-711f18ebff10`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913025::DEBUG::2013-11-20 02:51:56,953::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913025::DEBUG::2013-11-20 02:51:56,953::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913025::DEBUG::2013-11-20 02:51:56,954::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e3b2e296-120d-4d68-9dd3-711f18ebff10`::Granted request >Thread-15913025::DEBUG::2013-11-20 02:51:56,960::task::495::TaskManager.Task::(_debug) Task 68a3b8cf-7045-4349-87f2-220edc0081af: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913025::DEBUG::2013-11-20 02:51:56,961::task::495::TaskManager.Task::(_debug) Task 68a3b8cf-7045-4349-87f2-220edc0081af: ref 1 aborting False >Thread-15913025::WARNING::2013-11-20 02:51:56,963::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913025::DEBUG::2013-11-20 02:51:56,964::task::495::TaskManager.Task::(_debug) Task 68a3b8cf-7045-4349-87f2-220edc0081af: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115405225984', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115405225984', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913025::DEBUG::2013-11-20 02:51:56,965::task::495::TaskManager.Task::(_debug) Task 68a3b8cf-7045-4349-87f2-220edc0081af: moving from state preparing -> state finished >Thread-15913025::DEBUG::2013-11-20 02:51:56,965::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913025::DEBUG::2013-11-20 02:51:56,966::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913025::DEBUG::2013-11-20 02:51:56,967::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913025::DEBUG::2013-11-20 02:51:56,967::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913025::DEBUG::2013-11-20 02:51:56,968::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913025::DEBUG::2013-11-20 02:51:56,968::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913025::DEBUG::2013-11-20 02:51:56,969::task::495::TaskManager.Task::(_debug) Task 68a3b8cf-7045-4349-87f2-220edc0081af: ref 0 aborting False >Thread-15913025::INFO::2013-11-20 02:51:56,970::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115405225984', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115405225984', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:51:57,907::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:51:58,427::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.071833 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:52:00,442::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:52:00,970::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0734917 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:52:02,984::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:52:03,493::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.072761 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:52:05,509::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:52:06,070::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0751324 s, 13.6 MB/s\n'; <rc> = 0 >Thread-15913029::DEBUG::2013-11-20 02:52:07,678::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913029::INFO::2013-11-20 02:52:07,679::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913029::DEBUG::2013-11-20 02:52:07,679::task::495::TaskManager.Task::(_debug) Task 2c3ceecd-0a99-4dcd-9489-cdb27fe8578a: moving from state init -> state preparing >Thread-15913029::DEBUG::2013-11-20 02:52:07,680::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913029::DEBUG::2013-11-20 02:52:07,680::task::495::TaskManager.Task::(_debug) Task 2c3ceecd-0a99-4dcd-9489-cdb27fe8578a: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913029::DEBUG::2013-11-20 02:52:07,681::task::495::TaskManager.Task::(_debug) Task 2c3ceecd-0a99-4dcd-9489-cdb27fe8578a: moving from state preparing -> state finished >Thread-15913029::DEBUG::2013-11-20 02:52:07,681::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913029::DEBUG::2013-11-20 02:52:07,681::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913029::DEBUG::2013-11-20 02:52:07,682::task::495::TaskManager.Task::(_debug) Task 2c3ceecd-0a99-4dcd-9489-cdb27fe8578a: ref 0 aborting False >Thread-15913029::INFO::2013-11-20 02:52:07,682::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913030::DEBUG::2013-11-20 02:52:07,685::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913030::INFO::2013-11-20 02:52:07,686::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913030::DEBUG::2013-11-20 02:52:07,686::task::495::TaskManager.Task::(_debug) Task f6dcdee7-216c-435f-be8a-49de29bcd494: moving from state init -> state preparing >Thread-15913030::DEBUG::2013-11-20 02:52:07,687::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`da59c914-1930-4e77-8424-4535732880cc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913030::DEBUG::2013-11-20 02:52:07,687::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913030::DEBUG::2013-11-20 02:52:07,688::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913030::DEBUG::2013-11-20 02:52:07,688::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`da59c914-1930-4e77-8424-4535732880cc`::Granted request >Thread-15913030::DEBUG::2013-11-20 02:52:07,689::task::495::TaskManager.Task::(_debug) Task f6dcdee7-216c-435f-be8a-49de29bcd494: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913030::DEBUG::2013-11-20 02:52:07,689::task::495::TaskManager.Task::(_debug) Task f6dcdee7-216c-435f-be8a-49de29bcd494: ref 1 aborting False >Thread-15913030::WARNING::2013-11-20 02:52:07,690::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913030::DEBUG::2013-11-20 02:52:07,691::task::495::TaskManager.Task::(_debug) Task f6dcdee7-216c-435f-be8a-49de29bcd494: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115405225984', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115405225984', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913030::DEBUG::2013-11-20 02:52:07,691::task::495::TaskManager.Task::(_debug) Task f6dcdee7-216c-435f-be8a-49de29bcd494: moving from state preparing -> state finished >Thread-15913030::DEBUG::2013-11-20 02:52:07,692::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913030::DEBUG::2013-11-20 02:52:07,692::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913030::DEBUG::2013-11-20 02:52:07,692::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913030::DEBUG::2013-11-20 02:52:07,693::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913030::DEBUG::2013-11-20 02:52:07,693::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913030::DEBUG::2013-11-20 02:52:07,693::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913030::DEBUG::2013-11-20 02:52:07,694::task::495::TaskManager.Task::(_debug) Task f6dcdee7-216c-435f-be8a-49de29bcd494: ref 0 aborting False >Thread-15913030::INFO::2013-11-20 02:52:07,694::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115405225984', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115405225984', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:52:08,086::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 02:52:08,305::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:52:08,307::task::495::TaskManager.Task::(_debug) Task e27e764f-4238-46ff-8226-ef3a5ae59a87: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:52:08,309::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b5dc8790-ea23-4363-8968-35edc81db0d0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:52:08,310::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:52:08,311::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:52:08,311::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b5dc8790-ea23-4363-8968-35edc81db0d0`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:52:08,313::task::495::TaskManager.Task::(_debug) Task e27e764f-4238-46ff-8226-ef3a5ae59a87: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:52:08,313::task::495::TaskManager.Task::(_debug) Task e27e764f-4238-46ff-8226-ef3a5ae59a87: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:52:08,314::task::495::TaskManager.Task::(_debug) Task e27e764f-4238-46ff-8226-ef3a5ae59a87: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:52:08,315::task::495::TaskManager.Task::(_debug) Task e27e764f-4238-46ff-8226-ef3a5ae59a87: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:52:08,315::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:52:08,316::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:52:08,316::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:52:08,317::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:52:08,318::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:52:08,318::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:52:08,319::task::495::TaskManager.Task::(_debug) Task e27e764f-4238-46ff-8226-ef3a5ae59a87: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:52:08,319::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 02:52:08,628::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.082776 s, 12.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:52:10,642::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913032::INFO::2013-11-20 02:52:10,873::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913032::DEBUG::2013-11-20 02:52:10,874::task::495::TaskManager.Task::(_debug) Task 96f42030-20dd-4cc6-9c71-35dffb780d3d: moving from state init -> state preparing >Thread-15913032::DEBUG::2013-11-20 02:52:10,875::task::495::TaskManager.Task::(_debug) Task 96f42030-20dd-4cc6-9c71-35dffb780d3d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000420808792114', 'lastCheck': 1384915927.7614231, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00159597396851', 'lastCheck': 1384915926.1905799, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0158338546753', 'lastCheck': 1384915927.214159, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000525951385498', 'lastCheck': 1384915925.190167, 'valid': True, 'code': 0}} >Thread-15913032::DEBUG::2013-11-20 02:52:10,876::task::495::TaskManager.Task::(_debug) Task 96f42030-20dd-4cc6-9c71-35dffb780d3d: moving from state preparing -> state finished >Thread-15913032::DEBUG::2013-11-20 02:52:10,876::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913032::DEBUG::2013-11-20 02:52:10,877::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913032::DEBUG::2013-11-20 02:52:10,878::task::495::TaskManager.Task::(_debug) Task 96f42030-20dd-4cc6-9c71-35dffb780d3d: ref 0 aborting False >Thread-15913032::INFO::2013-11-20 02:52:10,878::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000420808792114', 'lastCheck': 1384915927.7614231, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00159597396851', 'lastCheck': 1384915926.1905799, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0158338546753', 'lastCheck': 1384915927.214159, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000525951385498', 'lastCheck': 1384915925.190167, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:52:11,169::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0735897 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:52:13,184::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:52:13,741::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.10415 s, 9.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:52:15,756::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:52:16,296::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0710003 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:52:18,312::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913036::DEBUG::2013-11-20 02:52:18,521::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913036::INFO::2013-11-20 02:52:18,523::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913036::DEBUG::2013-11-20 02:52:18,523::task::495::TaskManager.Task::(_debug) Task 09ea68e9-d734-4ae3-b573-23f8e51d3e50: moving from state init -> state preparing >Thread-15913036::DEBUG::2013-11-20 02:52:18,524::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913036::DEBUG::2013-11-20 02:52:18,524::task::495::TaskManager.Task::(_debug) Task 09ea68e9-d734-4ae3-b573-23f8e51d3e50: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913036::DEBUG::2013-11-20 02:52:18,525::task::495::TaskManager.Task::(_debug) Task 09ea68e9-d734-4ae3-b573-23f8e51d3e50: moving from state preparing -> state finished >Thread-15913036::DEBUG::2013-11-20 02:52:18,525::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913036::DEBUG::2013-11-20 02:52:18,525::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913036::DEBUG::2013-11-20 02:52:18,526::task::495::TaskManager.Task::(_debug) Task 09ea68e9-d734-4ae3-b573-23f8e51d3e50: ref 0 aborting False >Thread-15913036::INFO::2013-11-20 02:52:18,526::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913037::DEBUG::2013-11-20 02:52:18,530::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913037::INFO::2013-11-20 02:52:18,531::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913037::DEBUG::2013-11-20 02:52:18,531::task::495::TaskManager.Task::(_debug) Task c13b1a2c-b627-4197-b064-1f4752aa019a: moving from state init -> state preparing >Thread-15913037::DEBUG::2013-11-20 02:52:18,533::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f497ec5c-a766-407a-8d44-827b6939f5c8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913037::DEBUG::2013-11-20 02:52:18,533::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913037::DEBUG::2013-11-20 02:52:18,534::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913037::DEBUG::2013-11-20 02:52:18,534::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f497ec5c-a766-407a-8d44-827b6939f5c8`::Granted request >Thread-15913037::DEBUG::2013-11-20 02:52:18,535::task::495::TaskManager.Task::(_debug) Task c13b1a2c-b627-4197-b064-1f4752aa019a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913037::DEBUG::2013-11-20 02:52:18,535::task::495::TaskManager.Task::(_debug) Task c13b1a2c-b627-4197-b064-1f4752aa019a: ref 1 aborting False >Thread-15913037::WARNING::2013-11-20 02:52:18,537::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913037::DEBUG::2013-11-20 02:52:18,537::task::495::TaskManager.Task::(_debug) Task c13b1a2c-b627-4197-b064-1f4752aa019a: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115405225984', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115405225984', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913037::DEBUG::2013-11-20 02:52:18,538::task::495::TaskManager.Task::(_debug) Task c13b1a2c-b627-4197-b064-1f4752aa019a: moving from state preparing -> state finished >Thread-15913037::DEBUG::2013-11-20 02:52:18,538::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913037::DEBUG::2013-11-20 02:52:18,538::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913037::DEBUG::2013-11-20 02:52:18,539::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913037::DEBUG::2013-11-20 02:52:18,539::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913037::DEBUG::2013-11-20 02:52:18,540::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913037::DEBUG::2013-11-20 02:52:18,540::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913037::DEBUG::2013-11-20 02:52:18,540::task::495::TaskManager.Task::(_debug) Task c13b1a2c-b627-4197-b064-1f4752aa019a: ref 0 aborting False >Thread-15913037::INFO::2013-11-20 02:52:18,541::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115405225984', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115405225984', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:52:18,818::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0728216 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:52:20,833::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:52:21,351::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0717271 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:52:23,366::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:52:23,894::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0768041 s, 13.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:52:25,909::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913040::INFO::2013-11-20 02:52:26,271::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913040::DEBUG::2013-11-20 02:52:26,271::task::495::TaskManager.Task::(_debug) Task 1520f0a6-49b2-42b2-97e8-5a1b8eb05e0f: moving from state init -> state preparing >Thread-15913040::DEBUG::2013-11-20 02:52:26,272::task::495::TaskManager.Task::(_debug) Task 1520f0a6-49b2-42b2-97e8-5a1b8eb05e0f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000324010848999', 'lastCheck': 1384915937.982949, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00210285186768', 'lastCheck': 1384915936.404006, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00155901908875', 'lastCheck': 1384915937.4039919, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000532150268555', 'lastCheck': 1384915945.596802, 'valid': True, 'code': 0}} >Thread-15913040::DEBUG::2013-11-20 02:52:26,273::task::495::TaskManager.Task::(_debug) Task 1520f0a6-49b2-42b2-97e8-5a1b8eb05e0f: moving from state preparing -> state finished >Thread-15913040::DEBUG::2013-11-20 02:52:26,273::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913040::DEBUG::2013-11-20 02:52:26,274::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913040::DEBUG::2013-11-20 02:52:26,274::task::495::TaskManager.Task::(_debug) Task 1520f0a6-49b2-42b2-97e8-5a1b8eb05e0f: ref 0 aborting False >Thread-15913040::INFO::2013-11-20 02:52:26,274::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000324010848999', 'lastCheck': 1384915937.982949, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00210285186768', 'lastCheck': 1384915936.404006, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00155901908875', 'lastCheck': 1384915937.4039919, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000532150268555', 'lastCheck': 1384915945.596802, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:52:26,435::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0709268 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:52:28,450::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:52:28,986::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0787868 s, 13.0 MB/s\n'; <rc> = 0 >Thread-15913042::DEBUG::2013-11-20 02:52:29,046::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913042::INFO::2013-11-20 02:52:29,047::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913042::DEBUG::2013-11-20 02:52:29,047::task::495::TaskManager.Task::(_debug) Task f45f94c2-a889-4f42-9b1f-11a5b5e742be: moving from state init -> state preparing >Thread-15913042::DEBUG::2013-11-20 02:52:29,048::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913042::DEBUG::2013-11-20 02:52:29,048::task::495::TaskManager.Task::(_debug) Task f45f94c2-a889-4f42-9b1f-11a5b5e742be: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913042::DEBUG::2013-11-20 02:52:29,048::task::495::TaskManager.Task::(_debug) Task f45f94c2-a889-4f42-9b1f-11a5b5e742be: moving from state preparing -> state finished >Thread-15913042::DEBUG::2013-11-20 02:52:29,049::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913042::DEBUG::2013-11-20 02:52:29,049::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913042::DEBUG::2013-11-20 02:52:29,049::task::495::TaskManager.Task::(_debug) Task f45f94c2-a889-4f42-9b1f-11a5b5e742be: ref 0 aborting False >Thread-15913042::INFO::2013-11-20 02:52:29,050::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913043::DEBUG::2013-11-20 02:52:29,053::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913043::INFO::2013-11-20 02:52:29,053::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913043::DEBUG::2013-11-20 02:52:29,054::task::495::TaskManager.Task::(_debug) Task 381d1af5-a4b8-4286-aae5-1d86c008dbde: moving from state init -> state preparing >Thread-15913043::DEBUG::2013-11-20 02:52:29,055::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`dd194b59-31ac-4a6f-bd2d-e1b20982c866`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913043::DEBUG::2013-11-20 02:52:29,055::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913043::DEBUG::2013-11-20 02:52:29,055::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913043::DEBUG::2013-11-20 02:52:29,056::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`dd194b59-31ac-4a6f-bd2d-e1b20982c866`::Granted request >Thread-15913043::DEBUG::2013-11-20 02:52:29,056::task::495::TaskManager.Task::(_debug) Task 381d1af5-a4b8-4286-aae5-1d86c008dbde: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913043::DEBUG::2013-11-20 02:52:29,057::task::495::TaskManager.Task::(_debug) Task 381d1af5-a4b8-4286-aae5-1d86c008dbde: ref 1 aborting False >Thread-15913043::WARNING::2013-11-20 02:52:29,058::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913043::DEBUG::2013-11-20 02:52:29,059::task::495::TaskManager.Task::(_debug) Task 381d1af5-a4b8-4286-aae5-1d86c008dbde: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115405225984', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913043::DEBUG::2013-11-20 02:52:29,059::task::495::TaskManager.Task::(_debug) Task 381d1af5-a4b8-4286-aae5-1d86c008dbde: moving from state preparing -> state finished >Thread-15913043::DEBUG::2013-11-20 02:52:29,059::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913043::DEBUG::2013-11-20 02:52:29,060::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913043::DEBUG::2013-11-20 02:52:29,060::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913043::DEBUG::2013-11-20 02:52:29,061::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913043::DEBUG::2013-11-20 02:52:29,061::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913043::DEBUG::2013-11-20 02:52:29,062::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913043::DEBUG::2013-11-20 02:52:29,062::task::495::TaskManager.Task::(_debug) Task 381d1af5-a4b8-4286-aae5-1d86c008dbde: ref 0 aborting False >Thread-15913043::INFO::2013-11-20 02:52:29,062::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115405225984', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:52:31,001::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:52:31,539::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.075363 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:52:33,554::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:52:34,102::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0738119 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:52:36,117::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:52:36,615::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0730562 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:52:38,630::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:52:39,151::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0644409 s, 15.9 MB/s\n'; <rc> = 0 >Thread-15913048::DEBUG::2013-11-20 02:52:39,349::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913048::INFO::2013-11-20 02:52:39,350::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913048::DEBUG::2013-11-20 02:52:39,351::task::495::TaskManager.Task::(_debug) Task fec0356d-de1f-4d59-9a67-29528bfd60fd: moving from state init -> state preparing >Thread-15913048::DEBUG::2013-11-20 02:52:39,352::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913048::DEBUG::2013-11-20 02:52:39,352::task::495::TaskManager.Task::(_debug) Task fec0356d-de1f-4d59-9a67-29528bfd60fd: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913048::DEBUG::2013-11-20 02:52:39,353::task::495::TaskManager.Task::(_debug) Task fec0356d-de1f-4d59-9a67-29528bfd60fd: moving from state preparing -> state finished >Thread-15913048::DEBUG::2013-11-20 02:52:39,354::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913048::DEBUG::2013-11-20 02:52:39,354::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913048::DEBUG::2013-11-20 02:52:39,355::task::495::TaskManager.Task::(_debug) Task fec0356d-de1f-4d59-9a67-29528bfd60fd: ref 0 aborting False >Thread-15913048::INFO::2013-11-20 02:52:39,356::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913049::DEBUG::2013-11-20 02:52:39,359::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913049::INFO::2013-11-20 02:52:39,360::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913049::DEBUG::2013-11-20 02:52:39,361::task::495::TaskManager.Task::(_debug) Task 1894e705-ee8a-4c04-92b7-ab2f3a369857: moving from state init -> state preparing >Thread-15913049::DEBUG::2013-11-20 02:52:39,362::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2b7cf11b-3dbf-4dee-b526-bb1f5fc920e8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913049::DEBUG::2013-11-20 02:52:39,363::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913049::DEBUG::2013-11-20 02:52:39,363::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913049::DEBUG::2013-11-20 02:52:39,364::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2b7cf11b-3dbf-4dee-b526-bb1f5fc920e8`::Granted request >Thread-15913049::DEBUG::2013-11-20 02:52:39,365::task::495::TaskManager.Task::(_debug) Task 1894e705-ee8a-4c04-92b7-ab2f3a369857: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913049::DEBUG::2013-11-20 02:52:39,366::task::495::TaskManager.Task::(_debug) Task 1894e705-ee8a-4c04-92b7-ab2f3a369857: ref 1 aborting False >Thread-15913049::WARNING::2013-11-20 02:52:39,368::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913049::DEBUG::2013-11-20 02:52:39,368::task::495::TaskManager.Task::(_debug) Task 1894e705-ee8a-4c04-92b7-ab2f3a369857: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913049::DEBUG::2013-11-20 02:52:39,369::task::495::TaskManager.Task::(_debug) Task 1894e705-ee8a-4c04-92b7-ab2f3a369857: moving from state preparing -> state finished >Thread-15913049::DEBUG::2013-11-20 02:52:39,370::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913049::DEBUG::2013-11-20 02:52:39,370::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913049::DEBUG::2013-11-20 02:52:39,371::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913049::DEBUG::2013-11-20 02:52:39,372::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913049::DEBUG::2013-11-20 02:52:39,372::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913049::DEBUG::2013-11-20 02:52:39,373::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913049::DEBUG::2013-11-20 02:52:39,373::task::495::TaskManager.Task::(_debug) Task 1894e705-ee8a-4c04-92b7-ab2f3a369857: ref 0 aborting False >Thread-15913049::INFO::2013-11-20 02:52:39,374::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:52:41,161::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:52:41,690::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0829689 s, 12.3 MB/s\n'; <rc> = 0 >Thread-15913050::INFO::2013-11-20 02:52:41,902::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913050::DEBUG::2013-11-20 02:52:41,903::task::495::TaskManager.Task::(_debug) Task dc60d0e2-29e8-49d0-a919-8887da294767: moving from state init -> state preparing >Thread-15913050::DEBUG::2013-11-20 02:52:41,904::task::495::TaskManager.Task::(_debug) Task dc60d0e2-29e8-49d0-a919-8887da294767: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000300168991089', 'lastCheck': 1384915958.324738, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00156497955322', 'lastCheck': 1384915956.805903, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00141620635986', 'lastCheck': 1384915957.8051829, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00049090385437', 'lastCheck': 1384915955.7982471, 'valid': True, 'code': 0}} >Thread-15913050::DEBUG::2013-11-20 02:52:41,905::task::495::TaskManager.Task::(_debug) Task dc60d0e2-29e8-49d0-a919-8887da294767: moving from state preparing -> state finished >Thread-15913050::DEBUG::2013-11-20 02:52:41,906::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913050::DEBUG::2013-11-20 02:52:41,906::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913050::DEBUG::2013-11-20 02:52:41,907::task::495::TaskManager.Task::(_debug) Task dc60d0e2-29e8-49d0-a919-8887da294767: ref 0 aborting False >Thread-15913050::INFO::2013-11-20 02:52:41,907::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000300168991089', 'lastCheck': 1384915958.324738, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00156497955322', 'lastCheck': 1384915956.805903, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00141620635986', 'lastCheck': 1384915957.8051829, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00049090385437', 'lastCheck': 1384915955.7982471, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:52:43,706::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:52:44,216::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0681689 s, 15.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:52:46,231::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:52:46,775::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0960525 s, 10.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:52:48,791::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:52:49,266::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0479717 s, 21.3 MB/s\n'; <rc> = 0 >Thread-15913054::DEBUG::2013-11-20 02:52:49,581::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913054::INFO::2013-11-20 02:52:49,583::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913054::DEBUG::2013-11-20 02:52:49,584::task::495::TaskManager.Task::(_debug) Task 3b869c51-1bc1-40a6-b8e2-3633e0e9fb31: moving from state init -> state preparing >Thread-15913054::DEBUG::2013-11-20 02:52:49,586::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913054::DEBUG::2013-11-20 02:52:49,587::task::495::TaskManager.Task::(_debug) Task 3b869c51-1bc1-40a6-b8e2-3633e0e9fb31: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913054::DEBUG::2013-11-20 02:52:49,588::task::495::TaskManager.Task::(_debug) Task 3b869c51-1bc1-40a6-b8e2-3633e0e9fb31: moving from state preparing -> state finished >Thread-15913054::DEBUG::2013-11-20 02:52:49,589::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913054::DEBUG::2013-11-20 02:52:49,589::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913054::DEBUG::2013-11-20 02:52:49,590::task::495::TaskManager.Task::(_debug) Task 3b869c51-1bc1-40a6-b8e2-3633e0e9fb31: ref 0 aborting False >Thread-15913054::INFO::2013-11-20 02:52:49,591::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913055::DEBUG::2013-11-20 02:52:49,595::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913055::INFO::2013-11-20 02:52:49,596::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913055::DEBUG::2013-11-20 02:52:49,597::task::495::TaskManager.Task::(_debug) Task f9f0e4f0-eae7-44d3-acb6-957e9760d0ba: moving from state init -> state preparing >Thread-15913055::DEBUG::2013-11-20 02:52:49,598::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bb3a305c-44eb-47d2-88c8-06920a3713e7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913055::DEBUG::2013-11-20 02:52:49,599::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913055::DEBUG::2013-11-20 02:52:49,600::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913055::DEBUG::2013-11-20 02:52:49,600::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bb3a305c-44eb-47d2-88c8-06920a3713e7`::Granted request >Thread-15913055::DEBUG::2013-11-20 02:52:49,601::task::495::TaskManager.Task::(_debug) Task f9f0e4f0-eae7-44d3-acb6-957e9760d0ba: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913055::DEBUG::2013-11-20 02:52:49,602::task::495::TaskManager.Task::(_debug) Task f9f0e4f0-eae7-44d3-acb6-957e9760d0ba: ref 1 aborting False >Thread-15913055::WARNING::2013-11-20 02:52:49,604::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913055::DEBUG::2013-11-20 02:52:49,605::task::495::TaskManager.Task::(_debug) Task f9f0e4f0-eae7-44d3-acb6-957e9760d0ba: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913055::DEBUG::2013-11-20 02:52:49,606::task::495::TaskManager.Task::(_debug) Task f9f0e4f0-eae7-44d3-acb6-957e9760d0ba: moving from state preparing -> state finished >Thread-15913055::DEBUG::2013-11-20 02:52:49,607::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913055::DEBUG::2013-11-20 02:52:49,607::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913055::DEBUG::2013-11-20 02:52:49,608::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913055::DEBUG::2013-11-20 02:52:49,608::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913055::DEBUG::2013-11-20 02:52:49,609::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913055::DEBUG::2013-11-20 02:52:49,609::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913055::DEBUG::2013-11-20 02:52:49,610::task::495::TaskManager.Task::(_debug) Task f9f0e4f0-eae7-44d3-acb6-957e9760d0ba: ref 0 aborting False >Thread-15913055::INFO::2013-11-20 02:52:49,611::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:52:51,282::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:52:51,833::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0744025 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:52:53,848::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:52:54,352::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0695366 s, 14.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:52:56,367::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:52:56,889::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0729321 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15913058::INFO::2013-11-20 02:52:57,126::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913058::DEBUG::2013-11-20 02:52:57,127::task::495::TaskManager.Task::(_debug) Task 1225be40-3b44-4da8-a220-4592420bae22: moving from state init -> state preparing >Thread-15913058::DEBUG::2013-11-20 02:52:57,128::task::495::TaskManager.Task::(_debug) Task 1225be40-3b44-4da8-a220-4592420bae22: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000334024429321', 'lastCheck': 1384915968.435797, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00156998634338', 'lastCheck': 1384915977.0586939, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00158905982971', 'lastCheck': 1384915967.945555, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000483989715576', 'lastCheck': 1384915976.052932, 'valid': True, 'code': 0}} >Thread-15913058::DEBUG::2013-11-20 02:52:57,129::task::495::TaskManager.Task::(_debug) Task 1225be40-3b44-4da8-a220-4592420bae22: moving from state preparing -> state finished >Thread-15913058::DEBUG::2013-11-20 02:52:57,129::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913058::DEBUG::2013-11-20 02:52:57,130::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913058::DEBUG::2013-11-20 02:52:57,130::task::495::TaskManager.Task::(_debug) Task 1225be40-3b44-4da8-a220-4592420bae22: ref 0 aborting False >Thread-15913058::INFO::2013-11-20 02:52:57,131::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000334024429321', 'lastCheck': 1384915968.435797, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00156998634338', 'lastCheck': 1384915977.0586939, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00158905982971', 'lastCheck': 1384915967.945555, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000483989715576', 'lastCheck': 1384915976.052932, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:52:58,904::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:52:59,446::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.073278 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15913060::DEBUG::2013-11-20 02:52:59,753::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913060::INFO::2013-11-20 02:52:59,754::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913060::DEBUG::2013-11-20 02:52:59,755::task::495::TaskManager.Task::(_debug) Task 0fcf85e3-ddf3-4481-9619-385dbec7cadc: moving from state init -> state preparing >Thread-15913060::DEBUG::2013-11-20 02:52:59,756::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913060::DEBUG::2013-11-20 02:52:59,756::task::495::TaskManager.Task::(_debug) Task 0fcf85e3-ddf3-4481-9619-385dbec7cadc: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913060::DEBUG::2013-11-20 02:52:59,757::task::495::TaskManager.Task::(_debug) Task 0fcf85e3-ddf3-4481-9619-385dbec7cadc: moving from state preparing -> state finished >Thread-15913060::DEBUG::2013-11-20 02:52:59,758::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913060::DEBUG::2013-11-20 02:52:59,758::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913060::DEBUG::2013-11-20 02:52:59,759::task::495::TaskManager.Task::(_debug) Task 0fcf85e3-ddf3-4481-9619-385dbec7cadc: ref 0 aborting False >Thread-15913060::INFO::2013-11-20 02:52:59,759::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913061::DEBUG::2013-11-20 02:52:59,764::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913061::INFO::2013-11-20 02:52:59,765::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913061::DEBUG::2013-11-20 02:52:59,765::task::495::TaskManager.Task::(_debug) Task e515f78e-de64-4bac-8acd-7cbb5aa373f1: moving from state init -> state preparing >Thread-15913061::DEBUG::2013-11-20 02:52:59,767::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e8606c6e-5860-46f4-8c08-e870ffdafd1e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913061::DEBUG::2013-11-20 02:52:59,767::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913061::DEBUG::2013-11-20 02:52:59,768::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913061::DEBUG::2013-11-20 02:52:59,768::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e8606c6e-5860-46f4-8c08-e870ffdafd1e`::Granted request >Thread-15913061::DEBUG::2013-11-20 02:52:59,770::task::495::TaskManager.Task::(_debug) Task e515f78e-de64-4bac-8acd-7cbb5aa373f1: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913061::DEBUG::2013-11-20 02:52:59,770::task::495::TaskManager.Task::(_debug) Task e515f78e-de64-4bac-8acd-7cbb5aa373f1: ref 1 aborting False >Thread-15913061::WARNING::2013-11-20 02:52:59,772::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913061::DEBUG::2013-11-20 02:52:59,773::task::495::TaskManager.Task::(_debug) Task e515f78e-de64-4bac-8acd-7cbb5aa373f1: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913061::DEBUG::2013-11-20 02:52:59,774::task::495::TaskManager.Task::(_debug) Task e515f78e-de64-4bac-8acd-7cbb5aa373f1: moving from state preparing -> state finished >Thread-15913061::DEBUG::2013-11-20 02:52:59,774::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913061::DEBUG::2013-11-20 02:52:59,775::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913061::DEBUG::2013-11-20 02:52:59,776::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913061::DEBUG::2013-11-20 02:52:59,776::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913061::DEBUG::2013-11-20 02:52:59,777::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913061::DEBUG::2013-11-20 02:52:59,777::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913061::DEBUG::2013-11-20 02:52:59,778::task::495::TaskManager.Task::(_debug) Task e515f78e-de64-4bac-8acd-7cbb5aa373f1: ref 0 aborting False >Thread-15913061::INFO::2013-11-20 02:52:59,779::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:53:01,461::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:01,989::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0731514 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:53:04,004::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:04,558::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0723847 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:53:06,573::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:07,103::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0515021 s, 19.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:53:09,114::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 02:53:09,360::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:53:09,362::task::495::TaskManager.Task::(_debug) Task e187b3f5-545a-460c-a987-00dddf5859f7: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:53:09,380::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`111b306a-ecd3-4df6-b1b6-0616e736e104`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:53:09,381::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:53:09,382::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:53:09,382::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`111b306a-ecd3-4df6-b1b6-0616e736e104`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:53:09,383::task::495::TaskManager.Task::(_debug) Task e187b3f5-545a-460c-a987-00dddf5859f7: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:53:09,384::task::495::TaskManager.Task::(_debug) Task e187b3f5-545a-460c-a987-00dddf5859f7: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:53:09,385::task::495::TaskManager.Task::(_debug) Task e187b3f5-545a-460c-a987-00dddf5859f7: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:53:09,385::task::495::TaskManager.Task::(_debug) Task e187b3f5-545a-460c-a987-00dddf5859f7: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:53:09,386::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:53:09,386::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:53:09,387::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:53:09,388::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:53:09,388::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:53:09,389::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:53:09,389::task::495::TaskManager.Task::(_debug) Task e187b3f5-545a-460c-a987-00dddf5859f7: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:53:09,390::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 02:53:09,662::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0585003 s, 17.5 MB/s\n'; <rc> = 0 >Thread-15913066::DEBUG::2013-11-20 02:53:10,352::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913066::INFO::2013-11-20 02:53:10,353::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913066::DEBUG::2013-11-20 02:53:10,354::task::495::TaskManager.Task::(_debug) Task 6fad9d52-2fe4-4600-b1d0-868720b24fd3: moving from state init -> state preparing >Thread-15913066::DEBUG::2013-11-20 02:53:10,355::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913066::DEBUG::2013-11-20 02:53:10,355::task::495::TaskManager.Task::(_debug) Task 6fad9d52-2fe4-4600-b1d0-868720b24fd3: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913066::DEBUG::2013-11-20 02:53:10,355::task::495::TaskManager.Task::(_debug) Task 6fad9d52-2fe4-4600-b1d0-868720b24fd3: moving from state preparing -> state finished >Thread-15913066::DEBUG::2013-11-20 02:53:10,356::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913066::DEBUG::2013-11-20 02:53:10,356::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913066::DEBUG::2013-11-20 02:53:10,356::task::495::TaskManager.Task::(_debug) Task 6fad9d52-2fe4-4600-b1d0-868720b24fd3: ref 0 aborting False >Thread-15913066::INFO::2013-11-20 02:53:10,357::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913067::DEBUG::2013-11-20 02:53:10,360::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913067::INFO::2013-11-20 02:53:10,360::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913067::DEBUG::2013-11-20 02:53:10,361::task::495::TaskManager.Task::(_debug) Task f7361d0d-dadc-417b-b307-b0d155f7da5a: moving from state init -> state preparing >Thread-15913067::DEBUG::2013-11-20 02:53:10,362::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`4759f247-e48d-4ff1-b26b-4f34988d8904`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913067::DEBUG::2013-11-20 02:53:10,362::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913067::DEBUG::2013-11-20 02:53:10,362::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913067::DEBUG::2013-11-20 02:53:10,363::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`4759f247-e48d-4ff1-b26b-4f34988d8904`::Granted request >Thread-15913067::DEBUG::2013-11-20 02:53:10,363::task::495::TaskManager.Task::(_debug) Task f7361d0d-dadc-417b-b307-b0d155f7da5a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913067::DEBUG::2013-11-20 02:53:10,364::task::495::TaskManager.Task::(_debug) Task f7361d0d-dadc-417b-b307-b0d155f7da5a: ref 1 aborting False >Thread-15913067::WARNING::2013-11-20 02:53:10,365::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913067::DEBUG::2013-11-20 02:53:10,365::task::495::TaskManager.Task::(_debug) Task f7361d0d-dadc-417b-b307-b0d155f7da5a: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913067::DEBUG::2013-11-20 02:53:10,366::task::495::TaskManager.Task::(_debug) Task f7361d0d-dadc-417b-b307-b0d155f7da5a: moving from state preparing -> state finished >Thread-15913067::DEBUG::2013-11-20 02:53:10,366::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913067::DEBUG::2013-11-20 02:53:10,366::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913067::DEBUG::2013-11-20 02:53:10,367::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913067::DEBUG::2013-11-20 02:53:10,367::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913067::DEBUG::2013-11-20 02:53:10,367::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913067::DEBUG::2013-11-20 02:53:10,368::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913067::DEBUG::2013-11-20 02:53:10,368::task::495::TaskManager.Task::(_debug) Task f7361d0d-dadc-417b-b307-b0d155f7da5a: ref 0 aborting False >Thread-15913067::INFO::2013-11-20 02:53:10,369::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:53:11,672::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:12,156::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0576131 s, 17.8 MB/s\n'; <rc> = 0 >Thread-15913068::INFO::2013-11-20 02:53:12,855::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913068::DEBUG::2013-11-20 02:53:12,856::task::495::TaskManager.Task::(_debug) Task f6268304-c25e-4ae4-bcf7-6faf16e5e4ed: moving from state init -> state preparing >Thread-15913068::DEBUG::2013-11-20 02:53:12,857::task::495::TaskManager.Task::(_debug) Task f6268304-c25e-4ae4-bcf7-6faf16e5e4ed: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000325918197632', 'lastCheck': 1384915988.8211141, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.01038813591', 'lastCheck': 1384915987.2520299, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0269019603729', 'lastCheck': 1384915988.273618, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000505208969116', 'lastCheck': 1384915986.240344, 'valid': True, 'code': 0}} >Thread-15913068::DEBUG::2013-11-20 02:53:12,858::task::495::TaskManager.Task::(_debug) Task f6268304-c25e-4ae4-bcf7-6faf16e5e4ed: moving from state preparing -> state finished >Thread-15913068::DEBUG::2013-11-20 02:53:12,858::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913068::DEBUG::2013-11-20 02:53:12,859::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913068::DEBUG::2013-11-20 02:53:12,859::task::495::TaskManager.Task::(_debug) Task f6268304-c25e-4ae4-bcf7-6faf16e5e4ed: ref 0 aborting False >Thread-15913068::INFO::2013-11-20 02:53:12,860::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000325918197632', 'lastCheck': 1384915988.8211141, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.01038813591', 'lastCheck': 1384915987.2520299, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0269019603729', 'lastCheck': 1384915988.273618, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000505208969116', 'lastCheck': 1384915986.240344, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:53:14,169::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:14,728::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0722109 s, 14.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:53:16,743::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:17,285::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0934037 s, 11.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:53:19,301::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:19,824::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0524574 s, 19.5 MB/s\n'; <rc> = 0 >Thread-15913072::DEBUG::2013-11-20 02:53:20,878::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913072::INFO::2013-11-20 02:53:20,879::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913072::DEBUG::2013-11-20 02:53:20,880::task::495::TaskManager.Task::(_debug) Task 8ba0637f-174b-4443-9078-48bced3a414f: moving from state init -> state preparing >Thread-15913072::DEBUG::2013-11-20 02:53:20,881::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913072::DEBUG::2013-11-20 02:53:20,881::task::495::TaskManager.Task::(_debug) Task 8ba0637f-174b-4443-9078-48bced3a414f: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913072::DEBUG::2013-11-20 02:53:20,882::task::495::TaskManager.Task::(_debug) Task 8ba0637f-174b-4443-9078-48bced3a414f: moving from state preparing -> state finished >Thread-15913072::DEBUG::2013-11-20 02:53:20,883::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913072::DEBUG::2013-11-20 02:53:20,883::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913072::DEBUG::2013-11-20 02:53:20,884::task::495::TaskManager.Task::(_debug) Task 8ba0637f-174b-4443-9078-48bced3a414f: ref 0 aborting False >Thread-15913072::INFO::2013-11-20 02:53:20,884::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913073::DEBUG::2013-11-20 02:53:20,888::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913073::INFO::2013-11-20 02:53:20,889::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913073::DEBUG::2013-11-20 02:53:20,890::task::495::TaskManager.Task::(_debug) Task 8c424a84-eff1-423e-bc2b-8b0071ae3171: moving from state init -> state preparing >Thread-15913073::DEBUG::2013-11-20 02:53:20,891::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9be51bfa-70dd-4b58-9490-021ae7d594c0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913073::DEBUG::2013-11-20 02:53:20,892::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913073::DEBUG::2013-11-20 02:53:20,893::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913073::DEBUG::2013-11-20 02:53:20,893::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9be51bfa-70dd-4b58-9490-021ae7d594c0`::Granted request >Thread-15913073::DEBUG::2013-11-20 02:53:20,895::task::495::TaskManager.Task::(_debug) Task 8c424a84-eff1-423e-bc2b-8b0071ae3171: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913073::DEBUG::2013-11-20 02:53:20,895::task::495::TaskManager.Task::(_debug) Task 8c424a84-eff1-423e-bc2b-8b0071ae3171: ref 1 aborting False >Thread-15913073::WARNING::2013-11-20 02:53:20,897::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913073::DEBUG::2013-11-20 02:53:20,898::task::495::TaskManager.Task::(_debug) Task 8c424a84-eff1-423e-bc2b-8b0071ae3171: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913073::DEBUG::2013-11-20 02:53:20,899::task::495::TaskManager.Task::(_debug) Task 8c424a84-eff1-423e-bc2b-8b0071ae3171: moving from state preparing -> state finished >Thread-15913073::DEBUG::2013-11-20 02:53:20,899::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913073::DEBUG::2013-11-20 02:53:20,900::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913073::DEBUG::2013-11-20 02:53:20,901::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913073::DEBUG::2013-11-20 02:53:20,901::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913073::DEBUG::2013-11-20 02:53:20,902::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913073::DEBUG::2013-11-20 02:53:20,902::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913073::DEBUG::2013-11-20 02:53:20,903::task::495::TaskManager.Task::(_debug) Task 8c424a84-eff1-423e-bc2b-8b0071ae3171: ref 0 aborting False >Thread-15913073::INFO::2013-11-20 02:53:20,904::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:53:21,836::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:22,368::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0733921 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:53:24,383::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:24,877::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0556808 s, 18.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:53:26,891::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:27,410::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0583996 s, 17.5 MB/s\n'; <rc> = 0 >Thread-15913076::INFO::2013-11-20 02:53:28,570::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913076::DEBUG::2013-11-20 02:53:28,570::task::495::TaskManager.Task::(_debug) Task a1eccaa2-394e-47dc-8657-495e892084ad: moving from state init -> state preparing >Thread-15913076::DEBUG::2013-11-20 02:53:28,571::task::495::TaskManager.Task::(_debug) Task a1eccaa2-394e-47dc-8657-495e892084ad: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000354051589966', 'lastCheck': 1384915998.969451, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00585985183716', 'lastCheck': 1384916007.624815, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00170588493347', 'lastCheck': 1384915998.427392, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000580072402954', 'lastCheck': 1384916006.6167281, 'valid': True, 'code': 0}} >Thread-15913076::DEBUG::2013-11-20 02:53:28,571::task::495::TaskManager.Task::(_debug) Task a1eccaa2-394e-47dc-8657-495e892084ad: moving from state preparing -> state finished >Thread-15913076::DEBUG::2013-11-20 02:53:28,572::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913076::DEBUG::2013-11-20 02:53:28,572::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913076::DEBUG::2013-11-20 02:53:28,572::task::495::TaskManager.Task::(_debug) Task a1eccaa2-394e-47dc-8657-495e892084ad: ref 0 aborting False >Thread-15913076::INFO::2013-11-20 02:53:28,573::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000354051589966', 'lastCheck': 1384915998.969451, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00585985183716', 'lastCheck': 1384916007.624815, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00170588493347', 'lastCheck': 1384915998.427392, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000580072402954', 'lastCheck': 1384916006.6167281, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:53:29,421::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:29,904::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0587088 s, 17.4 MB/s\n'; <rc> = 0 >Thread-15913078::DEBUG::2013-11-20 02:53:31,207::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913078::INFO::2013-11-20 02:53:31,209::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913078::DEBUG::2013-11-20 02:53:31,209::task::495::TaskManager.Task::(_debug) Task e2643eb5-deff-4d09-af53-560734ffe268: moving from state init -> state preparing >Thread-15913078::DEBUG::2013-11-20 02:53:31,210::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913078::DEBUG::2013-11-20 02:53:31,211::task::495::TaskManager.Task::(_debug) Task e2643eb5-deff-4d09-af53-560734ffe268: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913078::DEBUG::2013-11-20 02:53:31,211::task::495::TaskManager.Task::(_debug) Task e2643eb5-deff-4d09-af53-560734ffe268: moving from state preparing -> state finished >Thread-15913078::DEBUG::2013-11-20 02:53:31,212::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913078::DEBUG::2013-11-20 02:53:31,212::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913078::DEBUG::2013-11-20 02:53:31,213::task::495::TaskManager.Task::(_debug) Task e2643eb5-deff-4d09-af53-560734ffe268: ref 0 aborting False >Thread-15913078::INFO::2013-11-20 02:53:31,213::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913079::DEBUG::2013-11-20 02:53:31,218::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913079::INFO::2013-11-20 02:53:31,219::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913079::DEBUG::2013-11-20 02:53:31,219::task::495::TaskManager.Task::(_debug) Task c4a7bbbd-6203-4cb5-b44c-e34e3576809d: moving from state init -> state preparing >Thread-15913079::DEBUG::2013-11-20 02:53:31,221::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d69ddc3e-598b-491c-ad0e-8863d825b8e7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913079::DEBUG::2013-11-20 02:53:31,221::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913079::DEBUG::2013-11-20 02:53:31,222::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913079::DEBUG::2013-11-20 02:53:31,223::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d69ddc3e-598b-491c-ad0e-8863d825b8e7`::Granted request >Thread-15913079::DEBUG::2013-11-20 02:53:31,224::task::495::TaskManager.Task::(_debug) Task c4a7bbbd-6203-4cb5-b44c-e34e3576809d: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913079::DEBUG::2013-11-20 02:53:31,224::task::495::TaskManager.Task::(_debug) Task c4a7bbbd-6203-4cb5-b44c-e34e3576809d: ref 1 aborting False >Thread-15913079::WARNING::2013-11-20 02:53:31,226::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913079::DEBUG::2013-11-20 02:53:31,227::task::495::TaskManager.Task::(_debug) Task c4a7bbbd-6203-4cb5-b44c-e34e3576809d: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913079::DEBUG::2013-11-20 02:53:31,228::task::495::TaskManager.Task::(_debug) Task c4a7bbbd-6203-4cb5-b44c-e34e3576809d: moving from state preparing -> state finished >Thread-15913079::DEBUG::2013-11-20 02:53:31,229::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913079::DEBUG::2013-11-20 02:53:31,230::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913079::DEBUG::2013-11-20 02:53:31,231::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913079::DEBUG::2013-11-20 02:53:31,231::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913079::DEBUG::2013-11-20 02:53:31,232::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913079::DEBUG::2013-11-20 02:53:31,232::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913079::DEBUG::2013-11-20 02:53:31,233::task::495::TaskManager.Task::(_debug) Task c4a7bbbd-6203-4cb5-b44c-e34e3576809d: ref 0 aborting False >Thread-15913079::INFO::2013-11-20 02:53:31,234::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:53:31,919::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:32,471::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.071518 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:53:34,484::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:35,006::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0730648 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:53:37,021::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:37,554::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0729374 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:53:39,570::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:40,119::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0771091 s, 13.3 MB/s\n'; <rc> = 0 >Thread-15913084::DEBUG::2013-11-20 02:53:41,396::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913084::INFO::2013-11-20 02:53:41,397::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913084::DEBUG::2013-11-20 02:53:41,398::task::495::TaskManager.Task::(_debug) Task 4ca15dea-fea0-4517-9d78-c8c3588979e1: moving from state init -> state preparing >Thread-15913084::DEBUG::2013-11-20 02:53:41,399::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913084::DEBUG::2013-11-20 02:53:41,400::task::495::TaskManager.Task::(_debug) Task 4ca15dea-fea0-4517-9d78-c8c3588979e1: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913084::DEBUG::2013-11-20 02:53:41,400::task::495::TaskManager.Task::(_debug) Task 4ca15dea-fea0-4517-9d78-c8c3588979e1: moving from state preparing -> state finished >Thread-15913084::DEBUG::2013-11-20 02:53:41,401::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913084::DEBUG::2013-11-20 02:53:41,401::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913084::DEBUG::2013-11-20 02:53:41,402::task::495::TaskManager.Task::(_debug) Task 4ca15dea-fea0-4517-9d78-c8c3588979e1: ref 0 aborting False >Thread-15913084::INFO::2013-11-20 02:53:41,402::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913085::DEBUG::2013-11-20 02:53:41,406::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913085::INFO::2013-11-20 02:53:41,407::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913085::DEBUG::2013-11-20 02:53:41,407::task::495::TaskManager.Task::(_debug) Task b75eeb42-9075-403d-9f0b-823d03b74c89: moving from state init -> state preparing >Thread-15913085::DEBUG::2013-11-20 02:53:41,409::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`0697a4de-0790-42d1-a864-b4c673b37eb1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913085::DEBUG::2013-11-20 02:53:41,409::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913085::DEBUG::2013-11-20 02:53:41,410::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913085::DEBUG::2013-11-20 02:53:41,411::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`0697a4de-0790-42d1-a864-b4c673b37eb1`::Granted request >Thread-15913085::DEBUG::2013-11-20 02:53:41,412::task::495::TaskManager.Task::(_debug) Task b75eeb42-9075-403d-9f0b-823d03b74c89: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913085::DEBUG::2013-11-20 02:53:41,413::task::495::TaskManager.Task::(_debug) Task b75eeb42-9075-403d-9f0b-823d03b74c89: ref 1 aborting False >Thread-15913085::WARNING::2013-11-20 02:53:41,415::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913085::DEBUG::2013-11-20 02:53:41,416::task::495::TaskManager.Task::(_debug) Task b75eeb42-9075-403d-9f0b-823d03b74c89: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913085::DEBUG::2013-11-20 02:53:41,416::task::495::TaskManager.Task::(_debug) Task b75eeb42-9075-403d-9f0b-823d03b74c89: moving from state preparing -> state finished >Thread-15913085::DEBUG::2013-11-20 02:53:41,417::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913085::DEBUG::2013-11-20 02:53:41,418::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913085::DEBUG::2013-11-20 02:53:41,418::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913085::DEBUG::2013-11-20 02:53:41,419::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913085::DEBUG::2013-11-20 02:53:41,420::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913085::DEBUG::2013-11-20 02:53:41,421::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913085::DEBUG::2013-11-20 02:53:41,422::task::495::TaskManager.Task::(_debug) Task b75eeb42-9075-403d-9f0b-823d03b74c89: ref 0 aborting False >Thread-15913085::INFO::2013-11-20 02:53:41,422::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:53:42,135::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:42,658::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0556004 s, 18.4 MB/s\n'; <rc> = 0 >Thread-15913086::INFO::2013-11-20 02:53:44,205::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913086::DEBUG::2013-11-20 02:53:44,206::task::495::TaskManager.Task::(_debug) Task 4321a752-233a-41a9-88be-6f3da5790ec2: moving from state init -> state preparing >Thread-15913086::DEBUG::2013-11-20 02:53:44,207::task::495::TaskManager.Task::(_debug) Task 4321a752-233a-41a9-88be-6f3da5790ec2: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000328063964844', 'lastCheck': 1384916019.2441721, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00145816802979', 'lastCheck': 1384916017.716085, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00133109092712', 'lastCheck': 1384916018.7147679, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000559091567993', 'lastCheck': 1384916016.7025559, 'valid': True, 'code': 0}} >Thread-15913086::DEBUG::2013-11-20 02:53:44,207::task::495::TaskManager.Task::(_debug) Task 4321a752-233a-41a9-88be-6f3da5790ec2: moving from state preparing -> state finished >Thread-15913086::DEBUG::2013-11-20 02:53:44,208::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913086::DEBUG::2013-11-20 02:53:44,209::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913086::DEBUG::2013-11-20 02:53:44,209::task::495::TaskManager.Task::(_debug) Task 4321a752-233a-41a9-88be-6f3da5790ec2: ref 0 aborting False >Thread-15913086::INFO::2013-11-20 02:53:44,210::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000328063964844', 'lastCheck': 1384916019.2441721, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00145816802979', 'lastCheck': 1384916017.716085, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00133109092712', 'lastCheck': 1384916018.7147679, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000559091567993', 'lastCheck': 1384916016.7025559, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:53:44,668::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:45,138::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0703268 s, 14.6 MB/s\n'; <rc> = 0 >Thread-42840::DEBUG::2013-11-20 02:53:46,851::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:47,154::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-42840::DEBUG::2013-11-20 02:53:47,650::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:53:47,718::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0874692 s, 11.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:53:49,733::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:50,311::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0744929 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15913090::DEBUG::2013-11-20 02:53:51,587::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913090::INFO::2013-11-20 02:53:51,589::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913090::DEBUG::2013-11-20 02:53:51,589::task::495::TaskManager.Task::(_debug) Task da0f7c5f-1e7e-4c6e-a547-c34c13c7733e: moving from state init -> state preparing >Thread-15913090::DEBUG::2013-11-20 02:53:51,590::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913090::DEBUG::2013-11-20 02:53:51,590::task::495::TaskManager.Task::(_debug) Task da0f7c5f-1e7e-4c6e-a547-c34c13c7733e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913090::DEBUG::2013-11-20 02:53:51,591::task::495::TaskManager.Task::(_debug) Task da0f7c5f-1e7e-4c6e-a547-c34c13c7733e: moving from state preparing -> state finished >Thread-15913090::DEBUG::2013-11-20 02:53:51,591::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913090::DEBUG::2013-11-20 02:53:51,592::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913090::DEBUG::2013-11-20 02:53:51,592::task::495::TaskManager.Task::(_debug) Task da0f7c5f-1e7e-4c6e-a547-c34c13c7733e: ref 0 aborting False >Thread-15913090::INFO::2013-11-20 02:53:51,593::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913091::DEBUG::2013-11-20 02:53:51,597::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913091::INFO::2013-11-20 02:53:51,598::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913091::DEBUG::2013-11-20 02:53:51,598::task::495::TaskManager.Task::(_debug) Task d7f221e2-c6d6-4379-a447-acbf2eaf4184: moving from state init -> state preparing >Thread-15913091::DEBUG::2013-11-20 02:53:51,599::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`03e29636-013f-4af1-a52f-be320f089f46`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913091::DEBUG::2013-11-20 02:53:51,600::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913091::DEBUG::2013-11-20 02:53:51,601::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913091::DEBUG::2013-11-20 02:53:51,601::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`03e29636-013f-4af1-a52f-be320f089f46`::Granted request >Thread-15913091::DEBUG::2013-11-20 02:53:51,602::task::495::TaskManager.Task::(_debug) Task d7f221e2-c6d6-4379-a447-acbf2eaf4184: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913091::DEBUG::2013-11-20 02:53:51,603::task::495::TaskManager.Task::(_debug) Task d7f221e2-c6d6-4379-a447-acbf2eaf4184: ref 1 aborting False >Thread-15913091::WARNING::2013-11-20 02:53:51,604::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913091::DEBUG::2013-11-20 02:53:51,605::task::495::TaskManager.Task::(_debug) Task d7f221e2-c6d6-4379-a447-acbf2eaf4184: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913091::DEBUG::2013-11-20 02:53:51,606::task::495::TaskManager.Task::(_debug) Task d7f221e2-c6d6-4379-a447-acbf2eaf4184: moving from state preparing -> state finished >Thread-15913091::DEBUG::2013-11-20 02:53:51,607::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913091::DEBUG::2013-11-20 02:53:51,607::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913091::DEBUG::2013-11-20 02:53:51,608::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913091::DEBUG::2013-11-20 02:53:51,608::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913091::DEBUG::2013-11-20 02:53:51,609::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913091::DEBUG::2013-11-20 02:53:51,609::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913091::DEBUG::2013-11-20 02:53:51,610::task::495::TaskManager.Task::(_debug) Task d7f221e2-c6d6-4379-a447-acbf2eaf4184: ref 0 aborting False >Thread-15913091::INFO::2013-11-20 02:53:51,610::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:53:52,327::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:52,857::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0739461 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:53:54,872::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:55,396::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0737237 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:53:57,411::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:53:57,945::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0732893 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:53:59,959::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913094::INFO::2013-11-20 02:54:00,183::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913094::DEBUG::2013-11-20 02:54:00,185::task::495::TaskManager.Task::(_debug) Task 9341d916-0b3b-4bb3-8611-7e8062e93b2e: moving from state init -> state preparing >Thread-15913094::DEBUG::2013-11-20 02:54:00,185::task::495::TaskManager.Task::(_debug) Task 9341d916-0b3b-4bb3-8611-7e8062e93b2e: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000306844711304', 'lastCheck': 1384916039.6294391, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00153589248657', 'lastCheck': 1384916038.101275, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00163912773132', 'lastCheck': 1384916039.1042919, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000545978546143', 'lastCheck': 1384916037.6657751, 'valid': True, 'code': 0}} >Thread-15913094::DEBUG::2013-11-20 02:54:00,186::task::495::TaskManager.Task::(_debug) Task 9341d916-0b3b-4bb3-8611-7e8062e93b2e: moving from state preparing -> state finished >Thread-15913094::DEBUG::2013-11-20 02:54:00,187::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913094::DEBUG::2013-11-20 02:54:00,188::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913094::DEBUG::2013-11-20 02:54:00,188::task::495::TaskManager.Task::(_debug) Task 9341d916-0b3b-4bb3-8611-7e8062e93b2e: ref 0 aborting False >Thread-15913094::INFO::2013-11-20 02:54:00,189::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000306844711304', 'lastCheck': 1384916039.6294391, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00153589248657', 'lastCheck': 1384916038.101275, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00163912773132', 'lastCheck': 1384916039.1042919, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000545978546143', 'lastCheck': 1384916037.6657751, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:54:00,511::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0877834 s, 11.7 MB/s\n'; <rc> = 0 >Thread-15913096::DEBUG::2013-11-20 02:54:01,887::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913096::INFO::2013-11-20 02:54:01,888::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913096::DEBUG::2013-11-20 02:54:01,888::task::495::TaskManager.Task::(_debug) Task 27dc70e6-03bc-4388-90d3-1230ee4d15ca: moving from state init -> state preparing >Thread-15913096::DEBUG::2013-11-20 02:54:01,889::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913096::DEBUG::2013-11-20 02:54:01,890::task::495::TaskManager.Task::(_debug) Task 27dc70e6-03bc-4388-90d3-1230ee4d15ca: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913096::DEBUG::2013-11-20 02:54:01,890::task::495::TaskManager.Task::(_debug) Task 27dc70e6-03bc-4388-90d3-1230ee4d15ca: moving from state preparing -> state finished >Thread-15913096::DEBUG::2013-11-20 02:54:01,891::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913096::DEBUG::2013-11-20 02:54:01,892::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913096::DEBUG::2013-11-20 02:54:01,892::task::495::TaskManager.Task::(_debug) Task 27dc70e6-03bc-4388-90d3-1230ee4d15ca: ref 0 aborting False >Thread-15913096::INFO::2013-11-20 02:54:01,893::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913097::DEBUG::2013-11-20 02:54:01,897::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913097::INFO::2013-11-20 02:54:01,898::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913097::DEBUG::2013-11-20 02:54:01,899::task::495::TaskManager.Task::(_debug) Task d62de80e-50b3-4667-aaff-fd7bd143abfb: moving from state init -> state preparing >Thread-15913097::DEBUG::2013-11-20 02:54:01,900::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c511a962-5404-4701-81d5-fe8e42974e17`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913097::DEBUG::2013-11-20 02:54:01,900::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913097::DEBUG::2013-11-20 02:54:01,901::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913097::DEBUG::2013-11-20 02:54:01,902::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c511a962-5404-4701-81d5-fe8e42974e17`::Granted request >Thread-15913097::DEBUG::2013-11-20 02:54:01,903::task::495::TaskManager.Task::(_debug) Task d62de80e-50b3-4667-aaff-fd7bd143abfb: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913097::DEBUG::2013-11-20 02:54:01,903::task::495::TaskManager.Task::(_debug) Task d62de80e-50b3-4667-aaff-fd7bd143abfb: ref 1 aborting False >Thread-15913097::WARNING::2013-11-20 02:54:01,905::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913097::DEBUG::2013-11-20 02:54:01,906::task::495::TaskManager.Task::(_debug) Task d62de80e-50b3-4667-aaff-fd7bd143abfb: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913097::DEBUG::2013-11-20 02:54:01,907::task::495::TaskManager.Task::(_debug) Task d62de80e-50b3-4667-aaff-fd7bd143abfb: moving from state preparing -> state finished >Thread-15913097::DEBUG::2013-11-20 02:54:01,908::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913097::DEBUG::2013-11-20 02:54:01,908::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913097::DEBUG::2013-11-20 02:54:01,909::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913097::DEBUG::2013-11-20 02:54:01,909::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913097::DEBUG::2013-11-20 02:54:01,910::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913097::DEBUG::2013-11-20 02:54:01,910::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913097::DEBUG::2013-11-20 02:54:01,911::task::495::TaskManager.Task::(_debug) Task d62de80e-50b3-4667-aaff-fd7bd143abfb: ref 0 aborting False >Thread-15913097::INFO::2013-11-20 02:54:01,912::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:54:02,527::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:54:03,113::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0727365 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:54:05,131::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:54:05,696::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0705204 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:54:07,712::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:54:08,242::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0730522 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:54:10,257::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 02:54:10,487::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:54:10,488::task::495::TaskManager.Task::(_debug) Task 240553fa-4a4e-43ed-a12a-ab6b54aa91e3: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:54:10,491::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`08d1d030-304e-4fc3-8690-337e31be9708`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:54:10,492::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:54:10,493::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:54:10,493::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`08d1d030-304e-4fc3-8690-337e31be9708`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:54:10,494::task::495::TaskManager.Task::(_debug) Task 240553fa-4a4e-43ed-a12a-ab6b54aa91e3: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:54:10,495::task::495::TaskManager.Task::(_debug) Task 240553fa-4a4e-43ed-a12a-ab6b54aa91e3: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:54:10,496::task::495::TaskManager.Task::(_debug) Task 240553fa-4a4e-43ed-a12a-ab6b54aa91e3: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:54:10,497::task::495::TaskManager.Task::(_debug) Task 240553fa-4a4e-43ed-a12a-ab6b54aa91e3: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:54:10,497::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:54:10,498::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:54:10,498::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:54:10,499::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:54:10,499::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:54:10,500::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:54:10,501::task::495::TaskManager.Task::(_debug) Task 240553fa-4a4e-43ed-a12a-ab6b54aa91e3: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:54:10,501::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 02:54:10,820::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0834513 s, 12.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:54:12,835::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:54:13,373::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0794512 s, 12.9 MB/s\n'; <rc> = 0 >Thread-15913099::DEBUG::2013-11-20 02:54:13,784::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913099::INFO::2013-11-20 02:54:13,786::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913099::DEBUG::2013-11-20 02:54:13,786::task::495::TaskManager.Task::(_debug) Task 5d993587-2b58-435d-acfe-fa0d1207850e: moving from state init -> state preparing >Thread-15913099::DEBUG::2013-11-20 02:54:13,787::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913099::DEBUG::2013-11-20 02:54:13,788::task::495::TaskManager.Task::(_debug) Task 5d993587-2b58-435d-acfe-fa0d1207850e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913099::DEBUG::2013-11-20 02:54:13,788::task::495::TaskManager.Task::(_debug) Task 5d993587-2b58-435d-acfe-fa0d1207850e: moving from state preparing -> state finished >Thread-15913099::DEBUG::2013-11-20 02:54:13,789::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913099::DEBUG::2013-11-20 02:54:13,790::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913099::DEBUG::2013-11-20 02:54:13,790::task::495::TaskManager.Task::(_debug) Task 5d993587-2b58-435d-acfe-fa0d1207850e: ref 0 aborting False >Thread-15913099::INFO::2013-11-20 02:54:13,791::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913100::DEBUG::2013-11-20 02:54:13,796::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913100::INFO::2013-11-20 02:54:13,797::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913100::DEBUG::2013-11-20 02:54:13,798::task::495::TaskManager.Task::(_debug) Task 639ebe47-bbc9-4f8b-af0e-2cd4eef2a8f9: moving from state init -> state preparing >Thread-15913100::DEBUG::2013-11-20 02:54:13,799::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fbc926ac-64a7-4470-9745-358dd35875dc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913100::DEBUG::2013-11-20 02:54:13,799::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913100::DEBUG::2013-11-20 02:54:13,800::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913100::DEBUG::2013-11-20 02:54:13,801::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fbc926ac-64a7-4470-9745-358dd35875dc`::Granted request >Thread-15913100::DEBUG::2013-11-20 02:54:13,812::task::495::TaskManager.Task::(_debug) Task 639ebe47-bbc9-4f8b-af0e-2cd4eef2a8f9: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913100::DEBUG::2013-11-20 02:54:13,813::task::495::TaskManager.Task::(_debug) Task 639ebe47-bbc9-4f8b-af0e-2cd4eef2a8f9: ref 1 aborting False >Thread-15913100::WARNING::2013-11-20 02:54:13,815::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913100::DEBUG::2013-11-20 02:54:13,816::task::495::TaskManager.Task::(_debug) Task 639ebe47-bbc9-4f8b-af0e-2cd4eef2a8f9: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913100::DEBUG::2013-11-20 02:54:13,817::task::495::TaskManager.Task::(_debug) Task 639ebe47-bbc9-4f8b-af0e-2cd4eef2a8f9: moving from state preparing -> state finished >Thread-15913100::DEBUG::2013-11-20 02:54:13,817::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913100::DEBUG::2013-11-20 02:54:13,818::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913100::DEBUG::2013-11-20 02:54:13,819::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913100::DEBUG::2013-11-20 02:54:13,819::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913100::DEBUG::2013-11-20 02:54:13,820::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913100::DEBUG::2013-11-20 02:54:13,820::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913100::DEBUG::2013-11-20 02:54:13,821::task::495::TaskManager.Task::(_debug) Task 639ebe47-bbc9-4f8b-af0e-2cd4eef2a8f9: ref 0 aborting False >Thread-15913100::INFO::2013-11-20 02:54:13,822::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:54:15,390::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:54:15,943::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0823547 s, 12.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:54:17,959::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:54:18,492::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0782887 s, 13.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:54:20,507::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:54:21,051::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0711056 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:54:23,066::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:54:23,589::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0709498 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:54:25,604::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913104::INFO::2013-11-20 02:54:26,011::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913104::DEBUG::2013-11-20 02:54:26,012::task::495::TaskManager.Task::(_debug) Task 9e9f655b-6852-48f4-b721-ba01dfc48817: moving from state init -> state preparing >Thread-15913104::DEBUG::2013-11-20 02:54:26,013::task::495::TaskManager.Task::(_debug) Task 9e9f655b-6852-48f4-b721-ba01dfc48817: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00033712387085', 'lastCheck': 1384916060.1721649, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00171709060669', 'lastCheck': 1384916058.616981, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00169396400452', 'lastCheck': 1384916059.6241319, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00197196006775', 'lastCheck': 1384916058.170717, 'valid': True, 'code': 0}} >Thread-15913104::DEBUG::2013-11-20 02:54:26,014::task::495::TaskManager.Task::(_debug) Task 9e9f655b-6852-48f4-b721-ba01dfc48817: moving from state preparing -> state finished >Thread-15913104::DEBUG::2013-11-20 02:54:26,015::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913104::DEBUG::2013-11-20 02:54:26,016::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913104::DEBUG::2013-11-20 02:54:26,017::task::495::TaskManager.Task::(_debug) Task 9e9f655b-6852-48f4-b721-ba01dfc48817: ref 0 aborting False >Thread-15913104::INFO::2013-11-20 02:54:26,018::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00033712387085', 'lastCheck': 1384916060.1721649, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00171709060669', 'lastCheck': 1384916058.616981, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00169396400452', 'lastCheck': 1384916059.6241319, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00197196006775', 'lastCheck': 1384916058.170717, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:54:26,093::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0876849 s, 11.7 MB/s\n'; <rc> = 0 >Thread-15913106::DEBUG::2013-11-20 02:54:27,187::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913106::INFO::2013-11-20 02:54:27,188::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913106::DEBUG::2013-11-20 02:54:27,188::task::495::TaskManager.Task::(_debug) Task 5853a9fa-8b15-42c9-8fda-8586ccc4617f: moving from state init -> state preparing >Thread-15913106::DEBUG::2013-11-20 02:54:27,189::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913106::DEBUG::2013-11-20 02:54:27,190::task::495::TaskManager.Task::(_debug) Task 5853a9fa-8b15-42c9-8fda-8586ccc4617f: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913106::DEBUG::2013-11-20 02:54:27,190::task::495::TaskManager.Task::(_debug) Task 5853a9fa-8b15-42c9-8fda-8586ccc4617f: moving from state preparing -> state finished >Thread-15913106::DEBUG::2013-11-20 02:54:27,191::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913106::DEBUG::2013-11-20 02:54:27,191::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913106::DEBUG::2013-11-20 02:54:27,192::task::495::TaskManager.Task::(_debug) Task 5853a9fa-8b15-42c9-8fda-8586ccc4617f: ref 0 aborting False >Thread-15913106::INFO::2013-11-20 02:54:27,193::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913107::DEBUG::2013-11-20 02:54:27,197::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913107::INFO::2013-11-20 02:54:27,198::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913107::DEBUG::2013-11-20 02:54:27,198::task::495::TaskManager.Task::(_debug) Task acc88e56-fc7c-4e92-800b-f1f191f5bd32: moving from state init -> state preparing >Thread-15913107::DEBUG::2013-11-20 02:54:27,200::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f19c0447-4105-4a0d-9496-bef3c83e71a0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913107::DEBUG::2013-11-20 02:54:27,200::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913107::DEBUG::2013-11-20 02:54:27,201::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913107::DEBUG::2013-11-20 02:54:27,201::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f19c0447-4105-4a0d-9496-bef3c83e71a0`::Granted request >Thread-15913107::DEBUG::2013-11-20 02:54:27,202::task::495::TaskManager.Task::(_debug) Task acc88e56-fc7c-4e92-800b-f1f191f5bd32: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913107::DEBUG::2013-11-20 02:54:27,203::task::495::TaskManager.Task::(_debug) Task acc88e56-fc7c-4e92-800b-f1f191f5bd32: ref 1 aborting False >Thread-15913107::WARNING::2013-11-20 02:54:27,205::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913107::DEBUG::2013-11-20 02:54:27,206::task::495::TaskManager.Task::(_debug) Task acc88e56-fc7c-4e92-800b-f1f191f5bd32: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913107::DEBUG::2013-11-20 02:54:27,207::task::495::TaskManager.Task::(_debug) Task acc88e56-fc7c-4e92-800b-f1f191f5bd32: moving from state preparing -> state finished >Thread-15913107::DEBUG::2013-11-20 02:54:27,207::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913107::DEBUG::2013-11-20 02:54:27,208::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913107::DEBUG::2013-11-20 02:54:27,208::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913107::DEBUG::2013-11-20 02:54:27,209::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913107::DEBUG::2013-11-20 02:54:27,210::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913107::DEBUG::2013-11-20 02:54:27,210::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913107::DEBUG::2013-11-20 02:54:27,211::task::495::TaskManager.Task::(_debug) Task acc88e56-fc7c-4e92-800b-f1f191f5bd32: ref 0 aborting False >Thread-15913107::INFO::2013-11-20 02:54:27,211::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:54:28,104::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:54:28,635::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.077718 s, 13.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:54:30,651::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:54:31,206::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0763985 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:54:33,222::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:54:33,741::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.072845 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:54:35,752::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:54:36,252::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0761895 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:54:38,267::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:54:38,782::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0690676 s, 14.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:54:40,793::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:54:41,274::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0611572 s, 16.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:54:43,288::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:54:43,840::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0743571 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:54:45,855::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:54:46,289::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0459725 s, 22.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:54:48,299::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:54:48,838::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0733099 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:54:50,854::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:54:51,381::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0734805 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:54:53,396::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:54:53,948::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0732869 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:54:55,964::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:54:56,497::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0761405 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:54:58,512::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:54:59,096::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0799854 s, 12.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:55:01,110::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:55:01,666::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0737786 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15913111::DEBUG::2013-11-20 02:55:01,685::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913111::INFO::2013-11-20 02:55:01,687::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913111::DEBUG::2013-11-20 02:55:01,687::task::495::TaskManager.Task::(_debug) Task c9ae9fb4-fa25-4b16-97ab-f04d81f1b2b8: moving from state init -> state preparing >Thread-15913111::DEBUG::2013-11-20 02:55:01,689::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913111::DEBUG::2013-11-20 02:55:01,689::task::495::TaskManager.Task::(_debug) Task c9ae9fb4-fa25-4b16-97ab-f04d81f1b2b8: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913111::DEBUG::2013-11-20 02:55:01,690::task::495::TaskManager.Task::(_debug) Task c9ae9fb4-fa25-4b16-97ab-f04d81f1b2b8: moving from state preparing -> state finished >Thread-15913111::DEBUG::2013-11-20 02:55:01,691::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913111::DEBUG::2013-11-20 02:55:01,692::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913111::DEBUG::2013-11-20 02:55:01,693::task::495::TaskManager.Task::(_debug) Task c9ae9fb4-fa25-4b16-97ab-f04d81f1b2b8: ref 0 aborting False >Thread-15913111::INFO::2013-11-20 02:55:01,693::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913112::DEBUG::2013-11-20 02:55:01,699::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913112::INFO::2013-11-20 02:55:01,700::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913112::DEBUG::2013-11-20 02:55:01,700::task::495::TaskManager.Task::(_debug) Task a303fabe-cf25-49a2-9100-4da3942a7a5a: moving from state init -> state preparing >Thread-15913112::DEBUG::2013-11-20 02:55:01,702::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`731ae53a-c03e-4744-aae1-ffe21dac385d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913112::DEBUG::2013-11-20 02:55:01,702::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913112::DEBUG::2013-11-20 02:55:01,703::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913112::DEBUG::2013-11-20 02:55:01,704::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`731ae53a-c03e-4744-aae1-ffe21dac385d`::Granted request >Thread-15913112::DEBUG::2013-11-20 02:55:01,705::task::495::TaskManager.Task::(_debug) Task a303fabe-cf25-49a2-9100-4da3942a7a5a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913112::DEBUG::2013-11-20 02:55:01,705::task::495::TaskManager.Task::(_debug) Task a303fabe-cf25-49a2-9100-4da3942a7a5a: ref 1 aborting False >Thread-15913112::WARNING::2013-11-20 02:55:01,707::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913112::DEBUG::2013-11-20 02:55:01,708::task::495::TaskManager.Task::(_debug) Task a303fabe-cf25-49a2-9100-4da3942a7a5a: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913112::DEBUG::2013-11-20 02:55:01,709::task::495::TaskManager.Task::(_debug) Task a303fabe-cf25-49a2-9100-4da3942a7a5a: moving from state preparing -> state finished >Thread-15913112::DEBUG::2013-11-20 02:55:01,709::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913112::DEBUG::2013-11-20 02:55:01,710::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913112::DEBUG::2013-11-20 02:55:01,711::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913112::DEBUG::2013-11-20 02:55:01,711::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913112::DEBUG::2013-11-20 02:55:01,712::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913112::DEBUG::2013-11-20 02:55:01,712::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913112::DEBUG::2013-11-20 02:55:01,713::task::495::TaskManager.Task::(_debug) Task a303fabe-cf25-49a2-9100-4da3942a7a5a: ref 0 aborting False >Thread-15913112::INFO::2013-11-20 02:55:01,714::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:55:03,686::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:55:04,183::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0660613 s, 15.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:55:06,195::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:55:06,748::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0895006 s, 11.4 MB/s\n'; <rc> = 0 >Thread-15913114::INFO::2013-11-20 02:55:07,058::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913114::DEBUG::2013-11-20 02:55:07,059::task::495::TaskManager.Task::(_debug) Task da6c53fb-2c4a-48aa-b796-2f2964ac600b: moving from state init -> state preparing >Thread-15913114::DEBUG::2013-11-20 02:55:07,059::task::495::TaskManager.Task::(_debug) Task da6c53fb-2c4a-48aa-b796-2f2964ac600b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000434160232544', 'lastCheck': 1384916100.7624979, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00123691558838', 'lastCheck': 1384916099.1927941, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00143790245056', 'lastCheck': 1384916100.206526, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000623941421509', 'lastCheck': 1384916098.761503, 'valid': True, 'code': 0}} >Thread-15913114::DEBUG::2013-11-20 02:55:07,060::task::495::TaskManager.Task::(_debug) Task da6c53fb-2c4a-48aa-b796-2f2964ac600b: moving from state preparing -> state finished >Thread-15913114::DEBUG::2013-11-20 02:55:07,061::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913114::DEBUG::2013-11-20 02:55:07,061::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913114::DEBUG::2013-11-20 02:55:07,062::task::495::TaskManager.Task::(_debug) Task da6c53fb-2c4a-48aa-b796-2f2964ac600b: ref 0 aborting False >Thread-15913114::INFO::2013-11-20 02:55:07,063::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000434160232544', 'lastCheck': 1384916100.7624979, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00123691558838', 'lastCheck': 1384916099.1927941, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00143790245056', 'lastCheck': 1384916100.206526, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000623941421509', 'lastCheck': 1384916098.761503, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:55:08,764::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:55:09,329::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0829475 s, 12.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:55:11,344::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 02:55:11,560::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:55:11,562::task::495::TaskManager.Task::(_debug) Task 3c1bf5c1-3ca0-44ef-82a9-98fbf68eeca3: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:55:11,564::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b0e72355-a6fb-401c-9096-77703b38fc70`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:55:11,565::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:55:11,566::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:55:11,567::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b0e72355-a6fb-401c-9096-77703b38fc70`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:55:11,568::task::495::TaskManager.Task::(_debug) Task 3c1bf5c1-3ca0-44ef-82a9-98fbf68eeca3: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:55:11,568::task::495::TaskManager.Task::(_debug) Task 3c1bf5c1-3ca0-44ef-82a9-98fbf68eeca3: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:55:11,569::task::495::TaskManager.Task::(_debug) Task 3c1bf5c1-3ca0-44ef-82a9-98fbf68eeca3: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:55:11,570::task::495::TaskManager.Task::(_debug) Task 3c1bf5c1-3ca0-44ef-82a9-98fbf68eeca3: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:55:11,570::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:55:11,571::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:55:11,572::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:55:11,572::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:55:11,573::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:55:11,573::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:55:11,574::task::495::TaskManager.Task::(_debug) Task 3c1bf5c1-3ca0-44ef-82a9-98fbf68eeca3: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:55:11,574::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 02:55:11,887::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0732148 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15913116::DEBUG::2013-11-20 02:55:13,188::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913116::INFO::2013-11-20 02:55:13,189::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913116::DEBUG::2013-11-20 02:55:13,190::task::495::TaskManager.Task::(_debug) Task 1badf274-ae01-459c-b60c-820364e4197b: moving from state init -> state preparing >Thread-15913116::DEBUG::2013-11-20 02:55:13,190::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913116::DEBUG::2013-11-20 02:55:13,191::task::495::TaskManager.Task::(_debug) Task 1badf274-ae01-459c-b60c-820364e4197b: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913116::DEBUG::2013-11-20 02:55:13,192::task::495::TaskManager.Task::(_debug) Task 1badf274-ae01-459c-b60c-820364e4197b: moving from state preparing -> state finished >Thread-15913116::DEBUG::2013-11-20 02:55:13,192::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913116::DEBUG::2013-11-20 02:55:13,193::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913116::DEBUG::2013-11-20 02:55:13,193::task::495::TaskManager.Task::(_debug) Task 1badf274-ae01-459c-b60c-820364e4197b: ref 0 aborting False >Thread-15913116::INFO::2013-11-20 02:55:13,194::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913117::DEBUG::2013-11-20 02:55:13,198::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913117::INFO::2013-11-20 02:55:13,199::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913117::DEBUG::2013-11-20 02:55:13,200::task::495::TaskManager.Task::(_debug) Task 56bd4268-8bb0-400a-9b17-0d85c09689c0: moving from state init -> state preparing >Thread-15913117::DEBUG::2013-11-20 02:55:13,201::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`67ebe263-3220-4e46-b871-c0e91ab5ba42`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913117::DEBUG::2013-11-20 02:55:13,202::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913117::DEBUG::2013-11-20 02:55:13,202::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913117::DEBUG::2013-11-20 02:55:13,203::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`67ebe263-3220-4e46-b871-c0e91ab5ba42`::Granted request >Thread-15913117::DEBUG::2013-11-20 02:55:13,204::task::495::TaskManager.Task::(_debug) Task 56bd4268-8bb0-400a-9b17-0d85c09689c0: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913117::DEBUG::2013-11-20 02:55:13,205::task::495::TaskManager.Task::(_debug) Task 56bd4268-8bb0-400a-9b17-0d85c09689c0: ref 1 aborting False >Thread-15913117::WARNING::2013-11-20 02:55:13,206::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913117::DEBUG::2013-11-20 02:55:13,207::task::495::TaskManager.Task::(_debug) Task 56bd4268-8bb0-400a-9b17-0d85c09689c0: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913117::DEBUG::2013-11-20 02:55:13,208::task::495::TaskManager.Task::(_debug) Task 56bd4268-8bb0-400a-9b17-0d85c09689c0: moving from state preparing -> state finished >Thread-15913117::DEBUG::2013-11-20 02:55:13,209::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913117::DEBUG::2013-11-20 02:55:13,209::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913117::DEBUG::2013-11-20 02:55:13,210::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913117::DEBUG::2013-11-20 02:55:13,211::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913117::DEBUG::2013-11-20 02:55:13,211::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913117::DEBUG::2013-11-20 02:55:13,212::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913117::DEBUG::2013-11-20 02:55:13,212::task::495::TaskManager.Task::(_debug) Task 56bd4268-8bb0-400a-9b17-0d85c09689c0: ref 0 aborting False >Thread-15913117::INFO::2013-11-20 02:55:13,213::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:55:13,902::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:55:14,476::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0765125 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:55:16,488::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:55:17,053::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0739175 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:55:19,069::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:55:19,521::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0491067 s, 20.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:55:21,531::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:55:22,079::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0764697 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:55:24,094::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:55:24,635::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0784428 s, 13.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:55:26,650::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:55:27,212::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0706435 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15913122::DEBUG::2013-11-20 02:55:27,379::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913122::INFO::2013-11-20 02:55:27,381::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913122::DEBUG::2013-11-20 02:55:27,381::task::495::TaskManager.Task::(_debug) Task cc314da7-d604-4ab5-9036-2bf9ba719642: moving from state init -> state preparing >Thread-15913122::DEBUG::2013-11-20 02:55:27,382::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913122::DEBUG::2013-11-20 02:55:27,383::task::495::TaskManager.Task::(_debug) Task cc314da7-d604-4ab5-9036-2bf9ba719642: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913122::DEBUG::2013-11-20 02:55:27,383::task::495::TaskManager.Task::(_debug) Task cc314da7-d604-4ab5-9036-2bf9ba719642: moving from state preparing -> state finished >Thread-15913122::DEBUG::2013-11-20 02:55:27,384::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913122::DEBUG::2013-11-20 02:55:27,384::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913122::DEBUG::2013-11-20 02:55:27,385::task::495::TaskManager.Task::(_debug) Task cc314da7-d604-4ab5-9036-2bf9ba719642: ref 0 aborting False >Thread-15913122::INFO::2013-11-20 02:55:27,386::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913123::DEBUG::2013-11-20 02:55:27,390::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913123::INFO::2013-11-20 02:55:27,391::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913123::DEBUG::2013-11-20 02:55:27,392::task::495::TaskManager.Task::(_debug) Task 2439dd85-4e99-4dbe-a93e-58283bcd1ba6: moving from state init -> state preparing >Thread-15913123::DEBUG::2013-11-20 02:55:27,393::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`88b22d1b-08e3-47c6-b68c-79cd75895429`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913123::DEBUG::2013-11-20 02:55:27,394::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913123::DEBUG::2013-11-20 02:55:27,394::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913123::DEBUG::2013-11-20 02:55:27,395::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`88b22d1b-08e3-47c6-b68c-79cd75895429`::Granted request >Thread-15913123::DEBUG::2013-11-20 02:55:27,396::task::495::TaskManager.Task::(_debug) Task 2439dd85-4e99-4dbe-a93e-58283bcd1ba6: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913123::DEBUG::2013-11-20 02:55:27,397::task::495::TaskManager.Task::(_debug) Task 2439dd85-4e99-4dbe-a93e-58283bcd1ba6: ref 1 aborting False >Thread-15913123::WARNING::2013-11-20 02:55:27,399::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913123::DEBUG::2013-11-20 02:55:27,399::task::495::TaskManager.Task::(_debug) Task 2439dd85-4e99-4dbe-a93e-58283bcd1ba6: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913123::DEBUG::2013-11-20 02:55:27,400::task::495::TaskManager.Task::(_debug) Task 2439dd85-4e99-4dbe-a93e-58283bcd1ba6: moving from state preparing -> state finished >Thread-15913123::DEBUG::2013-11-20 02:55:27,401::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913123::DEBUG::2013-11-20 02:55:27,401::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913123::DEBUG::2013-11-20 02:55:27,402::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913123::DEBUG::2013-11-20 02:55:27,403::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913123::DEBUG::2013-11-20 02:55:27,403::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913123::DEBUG::2013-11-20 02:55:27,404::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913123::DEBUG::2013-11-20 02:55:27,404::task::495::TaskManager.Task::(_debug) Task 2439dd85-4e99-4dbe-a93e-58283bcd1ba6: ref 0 aborting False >Thread-15913123::INFO::2013-11-20 02:55:27,405::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913124::INFO::2013-11-20 02:55:28,927::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913124::DEBUG::2013-11-20 02:55:28,928::task::495::TaskManager.Task::(_debug) Task aeed87e5-de9c-4ae5-b5db-93636a525071: moving from state init -> state preparing >Thread-15913124::DEBUG::2013-11-20 02:55:28,929::task::495::TaskManager.Task::(_debug) Task aeed87e5-de9c-4ae5-b5db-93636a525071: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000278949737549', 'lastCheck': 1384916121.289798, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00231599807739', 'lastCheck': 1384916119.743093, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00173902511597', 'lastCheck': 1384916120.7429709, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00714993476868', 'lastCheck': 1384916119.289396, 'valid': True, 'code': 0}} >Thread-15913124::DEBUG::2013-11-20 02:55:28,930::task::495::TaskManager.Task::(_debug) Task aeed87e5-de9c-4ae5-b5db-93636a525071: moving from state preparing -> state finished >Thread-15913124::DEBUG::2013-11-20 02:55:28,930::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913124::DEBUG::2013-11-20 02:55:28,931::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913124::DEBUG::2013-11-20 02:55:28,931::task::495::TaskManager.Task::(_debug) Task aeed87e5-de9c-4ae5-b5db-93636a525071: ref 0 aborting False >Thread-15913124::INFO::2013-11-20 02:55:28,932::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000278949737549', 'lastCheck': 1384916121.289798, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00231599807739', 'lastCheck': 1384916119.743093, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00173902511597', 'lastCheck': 1384916120.7429709, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00714993476868', 'lastCheck': 1384916119.289396, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:55:29,227::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:55:29,774::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0950964 s, 10.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:55:31,788::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:55:32,340::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0893957 s, 11.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:55:34,359::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:55:34,880::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.074563 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:55:36,895::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:55:37,391::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0727639 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:55:39,405::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:55:39,994::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.075617 s, 13.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:55:42,008::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:55:42,566::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.072423 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:55:44,581::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:55:45,135::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0731716 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:55:47,150::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:55:47,659::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0592019 s, 17.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:55:49,668::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:55:50,224::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0751314 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:55:52,236::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:55:52,771::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0732972 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:55:54,786::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913129::DEBUG::2013-11-20 02:55:55,007::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913129::INFO::2013-11-20 02:55:55,010::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913129::DEBUG::2013-11-20 02:55:55,011::task::495::TaskManager.Task::(_debug) Task 4656ca2f-c939-43d4-b919-1f7453e869aa: moving from state init -> state preparing >Thread-15913129::DEBUG::2013-11-20 02:55:55,013::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913129::DEBUG::2013-11-20 02:55:55,014::task::495::TaskManager.Task::(_debug) Task 4656ca2f-c939-43d4-b919-1f7453e869aa: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913129::DEBUG::2013-11-20 02:55:55,015::task::495::TaskManager.Task::(_debug) Task 4656ca2f-c939-43d4-b919-1f7453e869aa: moving from state preparing -> state finished >Thread-15913129::DEBUG::2013-11-20 02:55:55,015::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913129::DEBUG::2013-11-20 02:55:55,016::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913129::DEBUG::2013-11-20 02:55:55,017::task::495::TaskManager.Task::(_debug) Task 4656ca2f-c939-43d4-b919-1f7453e869aa: ref 0 aborting False >Thread-15913129::INFO::2013-11-20 02:55:55,018::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913130::DEBUG::2013-11-20 02:55:55,024::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913130::INFO::2013-11-20 02:55:55,025::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913130::DEBUG::2013-11-20 02:55:55,026::task::495::TaskManager.Task::(_debug) Task 0e2b15bf-dcf1-40ca-9ea9-1e02515a2c8c: moving from state init -> state preparing >Thread-15913130::DEBUG::2013-11-20 02:55:55,027::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2200cb36-8adf-410b-8ed6-93e5d8dee1d2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913130::DEBUG::2013-11-20 02:55:55,028::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913130::DEBUG::2013-11-20 02:55:55,029::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913130::DEBUG::2013-11-20 02:55:55,029::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2200cb36-8adf-410b-8ed6-93e5d8dee1d2`::Granted request >Thread-15913130::DEBUG::2013-11-20 02:55:55,031::task::495::TaskManager.Task::(_debug) Task 0e2b15bf-dcf1-40ca-9ea9-1e02515a2c8c: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913130::DEBUG::2013-11-20 02:55:55,031::task::495::TaskManager.Task::(_debug) Task 0e2b15bf-dcf1-40ca-9ea9-1e02515a2c8c: ref 1 aborting False >Thread-15913130::WARNING::2013-11-20 02:55:55,034::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913130::DEBUG::2013-11-20 02:55:55,035::task::495::TaskManager.Task::(_debug) Task 0e2b15bf-dcf1-40ca-9ea9-1e02515a2c8c: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913130::DEBUG::2013-11-20 02:55:55,036::task::495::TaskManager.Task::(_debug) Task 0e2b15bf-dcf1-40ca-9ea9-1e02515a2c8c: moving from state preparing -> state finished >Thread-15913130::DEBUG::2013-11-20 02:55:55,036::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913130::DEBUG::2013-11-20 02:55:55,037::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913130::DEBUG::2013-11-20 02:55:55,037::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913130::DEBUG::2013-11-20 02:55:55,038::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913130::DEBUG::2013-11-20 02:55:55,039::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913130::DEBUG::2013-11-20 02:55:55,039::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913130::DEBUG::2013-11-20 02:55:55,040::task::495::TaskManager.Task::(_debug) Task 0e2b15bf-dcf1-40ca-9ea9-1e02515a2c8c: ref 0 aborting False >Thread-15913130::INFO::2013-11-20 02:55:55,041::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:55:55,318::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0736744 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:55:57,333::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:55:57,863::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0855231 s, 12.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:55:59,879::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913132::INFO::2013-11-20 02:56:00,120::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913132::DEBUG::2013-11-20 02:56:00,121::task::495::TaskManager.Task::(_debug) Task 45b581e5-c96a-43ff-80ba-d318bf37eba0: moving from state init -> state preparing >Thread-15913132::DEBUG::2013-11-20 02:56:00,122::task::495::TaskManager.Task::(_debug) Task 45b581e5-c96a-43ff-80ba-d318bf37eba0: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000478982925415', 'lastCheck': 1384916151.9082601, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00574803352356', 'lastCheck': 1384916150.4076769, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00153398513794', 'lastCheck': 1384916151.391309, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00165390968323', 'lastCheck': 1384916160.1167221, 'valid': True, 'code': 0}} >Thread-15913132::DEBUG::2013-11-20 02:56:00,123::task::495::TaskManager.Task::(_debug) Task 45b581e5-c96a-43ff-80ba-d318bf37eba0: moving from state preparing -> state finished >Thread-15913132::DEBUG::2013-11-20 02:56:00,123::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913132::DEBUG::2013-11-20 02:56:00,123::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913132::DEBUG::2013-11-20 02:56:00,124::task::495::TaskManager.Task::(_debug) Task 45b581e5-c96a-43ff-80ba-d318bf37eba0: ref 0 aborting False >Thread-15913132::INFO::2013-11-20 02:56:00,124::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000478982925415', 'lastCheck': 1384916151.9082601, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00574803352356', 'lastCheck': 1384916150.4076769, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00153398513794', 'lastCheck': 1384916151.391309, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00165390968323', 'lastCheck': 1384916160.1167221, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:56:00,442::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0856653 s, 12.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:56:02,457::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:03,008::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0871502 s, 11.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:56:05,024::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:05,564::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0737444 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:56:07,579::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:08,104::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0731949 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15913135::DEBUG::2013-11-20 02:56:08,762::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913135::INFO::2013-11-20 02:56:08,763::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913135::DEBUG::2013-11-20 02:56:08,764::task::495::TaskManager.Task::(_debug) Task 29b0cbb3-4e6b-427c-bd19-1575a289e77e: moving from state init -> state preparing >Thread-15913135::DEBUG::2013-11-20 02:56:08,765::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913135::DEBUG::2013-11-20 02:56:08,765::task::495::TaskManager.Task::(_debug) Task 29b0cbb3-4e6b-427c-bd19-1575a289e77e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913135::DEBUG::2013-11-20 02:56:08,766::task::495::TaskManager.Task::(_debug) Task 29b0cbb3-4e6b-427c-bd19-1575a289e77e: moving from state preparing -> state finished >Thread-15913135::DEBUG::2013-11-20 02:56:08,766::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913135::DEBUG::2013-11-20 02:56:08,767::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913135::DEBUG::2013-11-20 02:56:08,767::task::495::TaskManager.Task::(_debug) Task 29b0cbb3-4e6b-427c-bd19-1575a289e77e: ref 0 aborting False >Thread-15913135::INFO::2013-11-20 02:56:08,768::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913136::DEBUG::2013-11-20 02:56:08,772::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913136::INFO::2013-11-20 02:56:08,773::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913136::DEBUG::2013-11-20 02:56:08,774::task::495::TaskManager.Task::(_debug) Task 11b8bc7b-87dd-4dee-83e1-12bee367bccf: moving from state init -> state preparing >Thread-15913136::DEBUG::2013-11-20 02:56:08,775::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`affcb56f-f055-4099-8c9e-5071ffcb5a72`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913136::DEBUG::2013-11-20 02:56:08,776::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913136::DEBUG::2013-11-20 02:56:08,777::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913136::DEBUG::2013-11-20 02:56:08,777::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`affcb56f-f055-4099-8c9e-5071ffcb5a72`::Granted request >Thread-15913136::DEBUG::2013-11-20 02:56:08,778::task::495::TaskManager.Task::(_debug) Task 11b8bc7b-87dd-4dee-83e1-12bee367bccf: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913136::DEBUG::2013-11-20 02:56:08,779::task::495::TaskManager.Task::(_debug) Task 11b8bc7b-87dd-4dee-83e1-12bee367bccf: ref 1 aborting False >Thread-15913136::WARNING::2013-11-20 02:56:08,781::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913136::DEBUG::2013-11-20 02:56:08,782::task::495::TaskManager.Task::(_debug) Task 11b8bc7b-87dd-4dee-83e1-12bee367bccf: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913136::DEBUG::2013-11-20 02:56:08,782::task::495::TaskManager.Task::(_debug) Task 11b8bc7b-87dd-4dee-83e1-12bee367bccf: moving from state preparing -> state finished >Thread-15913136::DEBUG::2013-11-20 02:56:08,783::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913136::DEBUG::2013-11-20 02:56:08,783::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913136::DEBUG::2013-11-20 02:56:08,784::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913136::DEBUG::2013-11-20 02:56:08,785::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913136::DEBUG::2013-11-20 02:56:08,785::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913136::DEBUG::2013-11-20 02:56:08,786::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913136::DEBUG::2013-11-20 02:56:08,786::task::495::TaskManager.Task::(_debug) Task 11b8bc7b-87dd-4dee-83e1-12bee367bccf: ref 0 aborting False >Thread-15913136::INFO::2013-11-20 02:56:08,787::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:56:10,120::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:10,640::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0733682 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:56:12,655::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 02:56:12,883::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:56:12,894::task::495::TaskManager.Task::(_debug) Task 4c04dd5d-1a13-44d7-8e1f-26ddf73fab43: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:56:12,897::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`4b1e662b-bc9e-4349-b0b4-48742726bf11`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:56:12,897::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:56:12,898::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:56:12,899::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`4b1e662b-bc9e-4349-b0b4-48742726bf11`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:56:12,900::task::495::TaskManager.Task::(_debug) Task 4c04dd5d-1a13-44d7-8e1f-26ddf73fab43: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:56:12,900::task::495::TaskManager.Task::(_debug) Task 4c04dd5d-1a13-44d7-8e1f-26ddf73fab43: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:56:12,901::task::495::TaskManager.Task::(_debug) Task 4c04dd5d-1a13-44d7-8e1f-26ddf73fab43: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:56:12,902::task::495::TaskManager.Task::(_debug) Task 4c04dd5d-1a13-44d7-8e1f-26ddf73fab43: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:56:12,902::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:56:12,903::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:56:12,904::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:56:12,904::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:56:12,905::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:56:12,905::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:56:12,906::task::495::TaskManager.Task::(_debug) Task 4c04dd5d-1a13-44d7-8e1f-26ddf73fab43: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:56:12,906::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 02:56:13,188::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0735713 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:56:15,203::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:15,721::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0756686 s, 13.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:56:17,734::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:18,253::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0751019 s, 13.6 MB/s\n'; <rc> = 0 >Thread-15913140::INFO::2013-11-20 02:56:19,985::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913140::DEBUG::2013-11-20 02:56:19,986::task::495::TaskManager.Task::(_debug) Task d9a411e1-d5d4-40aa-b228-363688588418: moving from state init -> state preparing >Thread-15913140::DEBUG::2013-11-20 02:56:19,987::task::495::TaskManager.Task::(_debug) Task d9a411e1-d5d4-40aa-b228-363688588418: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.0002760887146', 'lastCheck': 1384916172.341984, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152707099915', 'lastCheck': 1384916170.7997651, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0015709400177', 'lastCheck': 1384916171.7980311, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000684976577759', 'lastCheck': 1384916170.3384769, 'valid': True, 'code': 0}} >Thread-15913140::DEBUG::2013-11-20 02:56:19,988::task::495::TaskManager.Task::(_debug) Task d9a411e1-d5d4-40aa-b228-363688588418: moving from state preparing -> state finished >Thread-15913140::DEBUG::2013-11-20 02:56:19,988::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913140::DEBUG::2013-11-20 02:56:19,989::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913140::DEBUG::2013-11-20 02:56:19,989::task::495::TaskManager.Task::(_debug) Task d9a411e1-d5d4-40aa-b228-363688588418: ref 0 aborting False >Thread-15913140::INFO::2013-11-20 02:56:19,990::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.0002760887146', 'lastCheck': 1384916172.341984, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152707099915', 'lastCheck': 1384916170.7997651, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0015709400177', 'lastCheck': 1384916171.7980311, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000684976577759', 'lastCheck': 1384916170.3384769, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:56:20,269::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:20,825::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.077143 s, 13.3 MB/s\n'; <rc> = 0 >Thread-42838::DEBUG::2013-11-20 02:56:22,514::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:22,841::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-42838::DEBUG::2013-11-20 02:56:23,269::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:56:23,304::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0458807 s, 22.3 MB/s\n'; <rc> = 0 >Thread-15913142::DEBUG::2013-11-20 02:56:23,356::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913142::INFO::2013-11-20 02:56:23,358::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913142::DEBUG::2013-11-20 02:56:23,358::task::495::TaskManager.Task::(_debug) Task 7a4e2153-29d1-4960-90c6-63aaafaeefc1: moving from state init -> state preparing >Thread-15913142::DEBUG::2013-11-20 02:56:23,359::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913142::DEBUG::2013-11-20 02:56:23,360::task::495::TaskManager.Task::(_debug) Task 7a4e2153-29d1-4960-90c6-63aaafaeefc1: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913142::DEBUG::2013-11-20 02:56:23,360::task::495::TaskManager.Task::(_debug) Task 7a4e2153-29d1-4960-90c6-63aaafaeefc1: moving from state preparing -> state finished >Thread-15913142::DEBUG::2013-11-20 02:56:23,361::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913142::DEBUG::2013-11-20 02:56:23,361::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913142::DEBUG::2013-11-20 02:56:23,362::task::495::TaskManager.Task::(_debug) Task 7a4e2153-29d1-4960-90c6-63aaafaeefc1: ref 0 aborting False >Thread-15913142::INFO::2013-11-20 02:56:23,362::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913143::DEBUG::2013-11-20 02:56:23,367::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913143::INFO::2013-11-20 02:56:23,368::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913143::DEBUG::2013-11-20 02:56:23,369::task::495::TaskManager.Task::(_debug) Task 1cacb3ed-56d3-4588-af5e-5884a07d4808: moving from state init -> state preparing >Thread-15913143::DEBUG::2013-11-20 02:56:23,370::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c934213b-b82d-425f-9417-17364385f610`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913143::DEBUG::2013-11-20 02:56:23,371::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913143::DEBUG::2013-11-20 02:56:23,371::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913143::DEBUG::2013-11-20 02:56:23,372::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c934213b-b82d-425f-9417-17364385f610`::Granted request >Thread-15913143::DEBUG::2013-11-20 02:56:23,373::task::495::TaskManager.Task::(_debug) Task 1cacb3ed-56d3-4588-af5e-5884a07d4808: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913143::DEBUG::2013-11-20 02:56:23,373::task::495::TaskManager.Task::(_debug) Task 1cacb3ed-56d3-4588-af5e-5884a07d4808: ref 1 aborting False >Thread-15913143::WARNING::2013-11-20 02:56:23,375::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913143::DEBUG::2013-11-20 02:56:23,376::task::495::TaskManager.Task::(_debug) Task 1cacb3ed-56d3-4588-af5e-5884a07d4808: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913143::DEBUG::2013-11-20 02:56:23,377::task::495::TaskManager.Task::(_debug) Task 1cacb3ed-56d3-4588-af5e-5884a07d4808: moving from state preparing -> state finished >Thread-15913143::DEBUG::2013-11-20 02:56:23,378::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913143::DEBUG::2013-11-20 02:56:23,378::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913143::DEBUG::2013-11-20 02:56:23,379::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913143::DEBUG::2013-11-20 02:56:23,380::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913143::DEBUG::2013-11-20 02:56:23,380::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913143::DEBUG::2013-11-20 02:56:23,380::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913143::DEBUG::2013-11-20 02:56:23,381::task::495::TaskManager.Task::(_debug) Task 1cacb3ed-56d3-4588-af5e-5884a07d4808: ref 0 aborting False >Thread-15913143::INFO::2013-11-20 02:56:23,382::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:56:25,314::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:25,842::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0731701 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:56:27,856::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:28,376::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0754717 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:56:30,389::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:30,901::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0752473 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:56:32,919::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:33,476::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0743747 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:56:35,490::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:36,048::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0768599 s, 13.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:56:38,063::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:38,498::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0468766 s, 21.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:56:40,508::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:41,085::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0737383 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:56:43,101::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:43,626::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0737032 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15913147::DEBUG::2013-11-20 02:56:44,618::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913147::INFO::2013-11-20 02:56:44,619::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913147::DEBUG::2013-11-20 02:56:44,620::task::495::TaskManager.Task::(_debug) Task 4ae5b15b-dfe6-4d5d-98ed-fd9f9d75b8a3: moving from state init -> state preparing >Thread-15913147::DEBUG::2013-11-20 02:56:44,621::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913147::DEBUG::2013-11-20 02:56:44,621::task::495::TaskManager.Task::(_debug) Task 4ae5b15b-dfe6-4d5d-98ed-fd9f9d75b8a3: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913147::DEBUG::2013-11-20 02:56:44,622::task::495::TaskManager.Task::(_debug) Task 4ae5b15b-dfe6-4d5d-98ed-fd9f9d75b8a3: moving from state preparing -> state finished >Thread-15913147::DEBUG::2013-11-20 02:56:44,622::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913147::DEBUG::2013-11-20 02:56:44,623::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913147::DEBUG::2013-11-20 02:56:44,624::task::495::TaskManager.Task::(_debug) Task 4ae5b15b-dfe6-4d5d-98ed-fd9f9d75b8a3: ref 0 aborting False >Thread-15913147::INFO::2013-11-20 02:56:44,624::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913148::DEBUG::2013-11-20 02:56:44,629::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913148::INFO::2013-11-20 02:56:44,630::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913148::DEBUG::2013-11-20 02:56:44,630::task::495::TaskManager.Task::(_debug) Task 2c1364b8-2949-49cc-b0ba-84c8b5e2099a: moving from state init -> state preparing >Thread-15913148::DEBUG::2013-11-20 02:56:44,631::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1bef695b-0bc1-4cbb-a361-98a67bf11028`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913148::DEBUG::2013-11-20 02:56:44,632::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913148::DEBUG::2013-11-20 02:56:44,633::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913148::DEBUG::2013-11-20 02:56:44,633::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1bef695b-0bc1-4cbb-a361-98a67bf11028`::Granted request >Thread-15913148::DEBUG::2013-11-20 02:56:44,634::task::495::TaskManager.Task::(_debug) Task 2c1364b8-2949-49cc-b0ba-84c8b5e2099a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913148::DEBUG::2013-11-20 02:56:44,635::task::495::TaskManager.Task::(_debug) Task 2c1364b8-2949-49cc-b0ba-84c8b5e2099a: ref 1 aborting False >Thread-15913148::WARNING::2013-11-20 02:56:44,637::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913148::DEBUG::2013-11-20 02:56:44,639::task::495::TaskManager.Task::(_debug) Task 2c1364b8-2949-49cc-b0ba-84c8b5e2099a: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913148::DEBUG::2013-11-20 02:56:44,640::task::495::TaskManager.Task::(_debug) Task 2c1364b8-2949-49cc-b0ba-84c8b5e2099a: moving from state preparing -> state finished >Thread-15913148::DEBUG::2013-11-20 02:56:44,641::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913148::DEBUG::2013-11-20 02:56:44,642::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913148::DEBUG::2013-11-20 02:56:44,643::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913148::DEBUG::2013-11-20 02:56:44,644::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913148::DEBUG::2013-11-20 02:56:44,644::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913148::DEBUG::2013-11-20 02:56:44,645::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913148::DEBUG::2013-11-20 02:56:44,646::task::495::TaskManager.Task::(_debug) Task 2c1364b8-2949-49cc-b0ba-84c8b5e2099a: ref 0 aborting False >Thread-15913148::INFO::2013-11-20 02:56:44,647::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:56:45,642::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:46,199::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0777266 s, 13.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:56:48,215::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:48,738::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0703409 s, 14.6 MB/s\n'; <rc> = 0 >Thread-15913150::INFO::2013-11-20 02:56:50,368::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913150::DEBUG::2013-11-20 02:56:50,369::task::495::TaskManager.Task::(_debug) Task 2b0b9c95-cb2b-425e-a74c-c5c84bac52bf: moving from state init -> state preparing >Thread-15913150::DEBUG::2013-11-20 02:56:50,370::task::495::TaskManager.Task::(_debug) Task 2b0b9c95-cb2b-425e-a74c-c5c84bac52bf: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00043511390686', 'lastCheck': 1384916203.3164229, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00145983695984', 'lastCheck': 1384916201.2962019, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00132203102112', 'lastCheck': 1384916202.2966571, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00254797935486', 'lastCheck': 1384916200.7797329, 'valid': True, 'code': 0}} >Thread-15913150::DEBUG::2013-11-20 02:56:50,371::task::495::TaskManager.Task::(_debug) Task 2b0b9c95-cb2b-425e-a74c-c5c84bac52bf: moving from state preparing -> state finished >Thread-15913150::DEBUG::2013-11-20 02:56:50,371::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913150::DEBUG::2013-11-20 02:56:50,372::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913150::DEBUG::2013-11-20 02:56:50,372::task::495::TaskManager.Task::(_debug) Task 2b0b9c95-cb2b-425e-a74c-c5c84bac52bf: ref 0 aborting False >Thread-15913150::INFO::2013-11-20 02:56:50,373::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00043511390686', 'lastCheck': 1384916203.3164229, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00145983695984', 'lastCheck': 1384916201.2962019, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00132203102112', 'lastCheck': 1384916202.2966571, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00254797935486', 'lastCheck': 1384916200.7797329, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:56:50,753::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:51,321::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0817211 s, 12.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:56:53,337::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:53,875::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0942589 s, 10.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:56:55,891::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:56,440::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0734101 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15913153::DEBUG::2013-11-20 02:56:58,318::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913153::INFO::2013-11-20 02:56:58,320::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913153::DEBUG::2013-11-20 02:56:58,321::task::495::TaskManager.Task::(_debug) Task 2c9b5420-09bc-46d5-b07c-d81237414ab3: moving from state init -> state preparing >Thread-15913153::DEBUG::2013-11-20 02:56:58,321::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913153::DEBUG::2013-11-20 02:56:58,322::task::495::TaskManager.Task::(_debug) Task 2c9b5420-09bc-46d5-b07c-d81237414ab3: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913153::DEBUG::2013-11-20 02:56:58,323::task::495::TaskManager.Task::(_debug) Task 2c9b5420-09bc-46d5-b07c-d81237414ab3: moving from state preparing -> state finished >Thread-15913153::DEBUG::2013-11-20 02:56:58,323::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913153::DEBUG::2013-11-20 02:56:58,324::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913153::DEBUG::2013-11-20 02:56:58,324::task::495::TaskManager.Task::(_debug) Task 2c9b5420-09bc-46d5-b07c-d81237414ab3: ref 0 aborting False >Thread-15913153::INFO::2013-11-20 02:56:58,325::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913154::DEBUG::2013-11-20 02:56:58,329::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913154::INFO::2013-11-20 02:56:58,330::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913154::DEBUG::2013-11-20 02:56:58,331::task::495::TaskManager.Task::(_debug) Task 9ae24c19-ebb9-407e-a0a5-d49835e436fb: moving from state init -> state preparing >Thread-15913154::DEBUG::2013-11-20 02:56:58,332::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f7739e9c-34ab-4a1b-9ceb-741422d3ac85`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913154::DEBUG::2013-11-20 02:56:58,333::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913154::DEBUG::2013-11-20 02:56:58,333::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913154::DEBUG::2013-11-20 02:56:58,334::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f7739e9c-34ab-4a1b-9ceb-741422d3ac85`::Granted request >Thread-15913154::DEBUG::2013-11-20 02:56:58,335::task::495::TaskManager.Task::(_debug) Task 9ae24c19-ebb9-407e-a0a5-d49835e436fb: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913154::DEBUG::2013-11-20 02:56:58,335::task::495::TaskManager.Task::(_debug) Task 9ae24c19-ebb9-407e-a0a5-d49835e436fb: ref 1 aborting False >Thread-15913154::WARNING::2013-11-20 02:56:58,337::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913154::DEBUG::2013-11-20 02:56:58,338::task::495::TaskManager.Task::(_debug) Task 9ae24c19-ebb9-407e-a0a5-d49835e436fb: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913154::DEBUG::2013-11-20 02:56:58,339::task::495::TaskManager.Task::(_debug) Task 9ae24c19-ebb9-407e-a0a5-d49835e436fb: moving from state preparing -> state finished >Thread-15913154::DEBUG::2013-11-20 02:56:58,340::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913154::DEBUG::2013-11-20 02:56:58,340::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913154::DEBUG::2013-11-20 02:56:58,341::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913154::DEBUG::2013-11-20 02:56:58,342::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913154::DEBUG::2013-11-20 02:56:58,342::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913154::DEBUG::2013-11-20 02:56:58,343::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913154::DEBUG::2013-11-20 02:56:58,343::task::495::TaskManager.Task::(_debug) Task 9ae24c19-ebb9-407e-a0a5-d49835e436fb: ref 0 aborting False >Thread-15913154::INFO::2013-11-20 02:56:58,344::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:56:58,455::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:56:59,000::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0737532 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:57:01,015::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:01,574::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0715127 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:57:03,590::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:04,118::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0726809 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:57:06,133::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:06,681::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0713049 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:57:08,696::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:09,229::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0729577 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15913158::INFO::2013-11-20 02:57:09,864::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913158::DEBUG::2013-11-20 02:57:09,865::task::495::TaskManager.Task::(_debug) Task 37b73f5b-a1b5-44c6-bf41-fcf2e1ffa561: moving from state init -> state preparing >Thread-15913158::DEBUG::2013-11-20 02:57:09,866::task::495::TaskManager.Task::(_debug) Task 37b73f5b-a1b5-44c6-bf41-fcf2e1ffa561: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000982999801636', 'lastCheck': 1384916223.8282421, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00153708457947', 'lastCheck': 1384916221.6845579, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00139093399048', 'lastCheck': 1384916222.685411, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00415992736816', 'lastCheck': 1384916221.2657909, 'valid': True, 'code': 0}} >Thread-15913158::DEBUG::2013-11-20 02:57:09,866::task::495::TaskManager.Task::(_debug) Task 37b73f5b-a1b5-44c6-bf41-fcf2e1ffa561: moving from state preparing -> state finished >Thread-15913158::DEBUG::2013-11-20 02:57:09,867::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913158::DEBUG::2013-11-20 02:57:09,868::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913158::DEBUG::2013-11-20 02:57:09,868::task::495::TaskManager.Task::(_debug) Task 37b73f5b-a1b5-44c6-bf41-fcf2e1ffa561: ref 0 aborting False >Thread-15913158::INFO::2013-11-20 02:57:09,869::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000982999801636', 'lastCheck': 1384916223.8282421, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00153708457947', 'lastCheck': 1384916221.6845579, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00139093399048', 'lastCheck': 1384916222.685411, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00415992736816', 'lastCheck': 1384916221.2657909, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:57:11,243::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:11,806::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0805771 s, 12.7 MB/s\n'; <rc> = 0 >Thread-15913160::DEBUG::2013-11-20 02:57:11,863::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913160::INFO::2013-11-20 02:57:11,864::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913160::DEBUG::2013-11-20 02:57:11,865::task::495::TaskManager.Task::(_debug) Task 86129046-31dd-4dc3-9040-17105fc5a09c: moving from state init -> state preparing >Thread-15913160::DEBUG::2013-11-20 02:57:11,866::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913160::DEBUG::2013-11-20 02:57:11,866::task::495::TaskManager.Task::(_debug) Task 86129046-31dd-4dc3-9040-17105fc5a09c: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913160::DEBUG::2013-11-20 02:57:11,867::task::495::TaskManager.Task::(_debug) Task 86129046-31dd-4dc3-9040-17105fc5a09c: moving from state preparing -> state finished >Thread-15913160::DEBUG::2013-11-20 02:57:11,867::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913160::DEBUG::2013-11-20 02:57:11,868::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913160::DEBUG::2013-11-20 02:57:11,868::task::495::TaskManager.Task::(_debug) Task 86129046-31dd-4dc3-9040-17105fc5a09c: ref 0 aborting False >Thread-15913160::INFO::2013-11-20 02:57:11,869::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913161::DEBUG::2013-11-20 02:57:11,874::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913161::INFO::2013-11-20 02:57:11,875::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913161::DEBUG::2013-11-20 02:57:11,875::task::495::TaskManager.Task::(_debug) Task 2676d28b-33ce-41ab-bf42-50f1c9ed7549: moving from state init -> state preparing >Thread-15913161::DEBUG::2013-11-20 02:57:11,877::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ce68fa29-a188-4175-9344-be9f50a692f4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913161::DEBUG::2013-11-20 02:57:11,877::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913161::DEBUG::2013-11-20 02:57:11,878::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913161::DEBUG::2013-11-20 02:57:11,879::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ce68fa29-a188-4175-9344-be9f50a692f4`::Granted request >Thread-15913161::DEBUG::2013-11-20 02:57:11,880::task::495::TaskManager.Task::(_debug) Task 2676d28b-33ce-41ab-bf42-50f1c9ed7549: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913161::DEBUG::2013-11-20 02:57:11,880::task::495::TaskManager.Task::(_debug) Task 2676d28b-33ce-41ab-bf42-50f1c9ed7549: ref 1 aborting False >Thread-15913161::WARNING::2013-11-20 02:57:11,882::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913161::DEBUG::2013-11-20 02:57:11,883::task::495::TaskManager.Task::(_debug) Task 2676d28b-33ce-41ab-bf42-50f1c9ed7549: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913161::DEBUG::2013-11-20 02:57:11,884::task::495::TaskManager.Task::(_debug) Task 2676d28b-33ce-41ab-bf42-50f1c9ed7549: moving from state preparing -> state finished >Thread-15913161::DEBUG::2013-11-20 02:57:11,884::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913161::DEBUG::2013-11-20 02:57:11,885::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913161::DEBUG::2013-11-20 02:57:11,886::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913161::DEBUG::2013-11-20 02:57:11,886::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913161::DEBUG::2013-11-20 02:57:11,887::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913161::DEBUG::2013-11-20 02:57:11,887::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913161::DEBUG::2013-11-20 02:57:11,888::task::495::TaskManager.Task::(_debug) Task 2676d28b-33ce-41ab-bf42-50f1c9ed7549: ref 0 aborting False >Thread-15913161::INFO::2013-11-20 02:57:11,889::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:57:13,822::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 02:57:13,998::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:57:14,001::task::495::TaskManager.Task::(_debug) Task 72d7bdb5-d00b-42fa-9cb8-56470da65d41: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:57:14,006::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`088dc98c-5fea-4593-80c8-0df5e8c52fff`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:57:14,007::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:57:14,009::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:57:14,010::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`088dc98c-5fea-4593-80c8-0df5e8c52fff`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:57:14,011::task::495::TaskManager.Task::(_debug) Task 72d7bdb5-d00b-42fa-9cb8-56470da65d41: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:57:14,011::task::495::TaskManager.Task::(_debug) Task 72d7bdb5-d00b-42fa-9cb8-56470da65d41: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:57:14,012::task::495::TaskManager.Task::(_debug) Task 72d7bdb5-d00b-42fa-9cb8-56470da65d41: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:57:14,013::task::495::TaskManager.Task::(_debug) Task 72d7bdb5-d00b-42fa-9cb8-56470da65d41: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:57:14,013::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:57:14,014::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:57:14,015::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:57:14,015::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:57:14,016::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:57:14,016::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:57:14,017::task::495::TaskManager.Task::(_debug) Task 72d7bdb5-d00b-42fa-9cb8-56470da65d41: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:57:14,017::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 02:57:14,323::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0732776 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:57:16,338::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:16,895::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0868101 s, 11.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:57:18,911::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:19,463::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0740697 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:57:21,478::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:22,007::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0732648 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15913166::DEBUG::2013-11-20 02:57:23,755::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913166::INFO::2013-11-20 02:57:23,756::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913166::DEBUG::2013-11-20 02:57:23,757::task::495::TaskManager.Task::(_debug) Task 2c63fa7e-28d4-40f7-93fe-7fec7e832cf3: moving from state init -> state preparing >Thread-15913166::DEBUG::2013-11-20 02:57:23,757::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913166::DEBUG::2013-11-20 02:57:23,758::task::495::TaskManager.Task::(_debug) Task 2c63fa7e-28d4-40f7-93fe-7fec7e832cf3: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913166::DEBUG::2013-11-20 02:57:23,759::task::495::TaskManager.Task::(_debug) Task 2c63fa7e-28d4-40f7-93fe-7fec7e832cf3: moving from state preparing -> state finished >Thread-15913166::DEBUG::2013-11-20 02:57:23,759::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913166::DEBUG::2013-11-20 02:57:23,760::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913166::DEBUG::2013-11-20 02:57:23,760::task::495::TaskManager.Task::(_debug) Task 2c63fa7e-28d4-40f7-93fe-7fec7e832cf3: ref 0 aborting False >Thread-15913166::INFO::2013-11-20 02:57:23,761::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913167::DEBUG::2013-11-20 02:57:23,766::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913167::INFO::2013-11-20 02:57:23,767::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913167::DEBUG::2013-11-20 02:57:23,767::task::495::TaskManager.Task::(_debug) Task 851da2e2-9180-4328-929b-f28236923ac3: moving from state init -> state preparing >Thread-15913167::DEBUG::2013-11-20 02:57:23,768::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`7c36a5e3-8ba5-4b49-9147-4689b5b7cd36`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913167::DEBUG::2013-11-20 02:57:23,769::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913167::DEBUG::2013-11-20 02:57:23,770::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913167::DEBUG::2013-11-20 02:57:23,770::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`7c36a5e3-8ba5-4b49-9147-4689b5b7cd36`::Granted request >Thread-15913167::DEBUG::2013-11-20 02:57:23,771::task::495::TaskManager.Task::(_debug) Task 851da2e2-9180-4328-929b-f28236923ac3: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913167::DEBUG::2013-11-20 02:57:23,772::task::495::TaskManager.Task::(_debug) Task 851da2e2-9180-4328-929b-f28236923ac3: ref 1 aborting False >Thread-15913167::WARNING::2013-11-20 02:57:23,774::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913167::DEBUG::2013-11-20 02:57:23,775::task::495::TaskManager.Task::(_debug) Task 851da2e2-9180-4328-929b-f28236923ac3: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913167::DEBUG::2013-11-20 02:57:23,776::task::495::TaskManager.Task::(_debug) Task 851da2e2-9180-4328-929b-f28236923ac3: moving from state preparing -> state finished >Thread-15913167::DEBUG::2013-11-20 02:57:23,776::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913167::DEBUG::2013-11-20 02:57:23,777::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913167::DEBUG::2013-11-20 02:57:23,777::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913167::DEBUG::2013-11-20 02:57:23,778::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913167::DEBUG::2013-11-20 02:57:23,778::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913167::DEBUG::2013-11-20 02:57:23,779::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913167::DEBUG::2013-11-20 02:57:23,780::task::495::TaskManager.Task::(_debug) Task 851da2e2-9180-4328-929b-f28236923ac3: ref 0 aborting False >Thread-15913167::INFO::2013-11-20 02:57:23,780::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:57:24,023::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:24,564::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0736123 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15913168::INFO::2013-11-20 02:57:25,616::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913168::DEBUG::2013-11-20 02:57:25,616::task::495::TaskManager.Task::(_debug) Task eb117fc1-7161-486b-9ee1-de7b7f522018: moving from state init -> state preparing >Thread-15913168::DEBUG::2013-11-20 02:57:25,617::task::495::TaskManager.Task::(_debug) Task eb117fc1-7161-486b-9ee1-de7b7f522018: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00039005279541', 'lastCheck': 1384916244.250705, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00199794769287', 'lastCheck': 1384916242.147239, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0015971660614', 'lastCheck': 1384916243.1467209, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000602960586548', 'lastCheck': 1384916241.6837521, 'valid': True, 'code': 0}} >Thread-15913168::DEBUG::2013-11-20 02:57:25,618::task::495::TaskManager.Task::(_debug) Task eb117fc1-7161-486b-9ee1-de7b7f522018: moving from state preparing -> state finished >Thread-15913168::DEBUG::2013-11-20 02:57:25,619::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913168::DEBUG::2013-11-20 02:57:25,619::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913168::DEBUG::2013-11-20 02:57:25,620::task::495::TaskManager.Task::(_debug) Task eb117fc1-7161-486b-9ee1-de7b7f522018: ref 0 aborting False >Thread-15913168::INFO::2013-11-20 02:57:25,620::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00039005279541', 'lastCheck': 1384916244.250705, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00199794769287', 'lastCheck': 1384916242.147239, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0015971660614', 'lastCheck': 1384916243.1467209, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000602960586548', 'lastCheck': 1384916241.6837521, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:57:26,579::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:27,151::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.102043 s, 10.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:57:29,167::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:29,694::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0734433 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:57:31,710::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:32,165::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0551042 s, 18.6 MB/s\n'; <rc> = 0 >Thread-15913172::DEBUG::2013-11-20 02:57:33,976::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913172::INFO::2013-11-20 02:57:33,978::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913172::DEBUG::2013-11-20 02:57:33,978::task::495::TaskManager.Task::(_debug) Task a3e496d9-d61f-4c65-bedf-8374edb5eae6: moving from state init -> state preparing >Thread-15913172::DEBUG::2013-11-20 02:57:33,979::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913172::DEBUG::2013-11-20 02:57:33,979::task::495::TaskManager.Task::(_debug) Task a3e496d9-d61f-4c65-bedf-8374edb5eae6: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913172::DEBUG::2013-11-20 02:57:33,979::task::495::TaskManager.Task::(_debug) Task a3e496d9-d61f-4c65-bedf-8374edb5eae6: moving from state preparing -> state finished >Thread-15913172::DEBUG::2013-11-20 02:57:33,980::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913172::DEBUG::2013-11-20 02:57:33,980::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913172::DEBUG::2013-11-20 02:57:33,980::task::495::TaskManager.Task::(_debug) Task a3e496d9-d61f-4c65-bedf-8374edb5eae6: ref 0 aborting False >Thread-15913172::INFO::2013-11-20 02:57:33,981::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913173::DEBUG::2013-11-20 02:57:33,984::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913173::INFO::2013-11-20 02:57:33,985::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913173::DEBUG::2013-11-20 02:57:33,985::task::495::TaskManager.Task::(_debug) Task 4e49640b-9dc4-4809-ac2c-f24decd5b1f3: moving from state init -> state preparing >Thread-15913173::DEBUG::2013-11-20 02:57:33,986::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`44c3e1d8-da1f-4ded-997b-00f49bdabd25`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913173::DEBUG::2013-11-20 02:57:33,986::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913173::DEBUG::2013-11-20 02:57:33,987::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913173::DEBUG::2013-11-20 02:57:33,987::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`44c3e1d8-da1f-4ded-997b-00f49bdabd25`::Granted request >Thread-15913173::DEBUG::2013-11-20 02:57:33,988::task::495::TaskManager.Task::(_debug) Task 4e49640b-9dc4-4809-ac2c-f24decd5b1f3: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913173::DEBUG::2013-11-20 02:57:33,988::task::495::TaskManager.Task::(_debug) Task 4e49640b-9dc4-4809-ac2c-f24decd5b1f3: ref 1 aborting False >Thread-15913173::WARNING::2013-11-20 02:57:33,989::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913173::DEBUG::2013-11-20 02:57:33,990::task::495::TaskManager.Task::(_debug) Task 4e49640b-9dc4-4809-ac2c-f24decd5b1f3: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913173::DEBUG::2013-11-20 02:57:33,990::task::495::TaskManager.Task::(_debug) Task 4e49640b-9dc4-4809-ac2c-f24decd5b1f3: moving from state preparing -> state finished >Thread-15913173::DEBUG::2013-11-20 02:57:33,991::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913173::DEBUG::2013-11-20 02:57:33,991::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913173::DEBUG::2013-11-20 02:57:33,992::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913173::DEBUG::2013-11-20 02:57:33,992::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913173::DEBUG::2013-11-20 02:57:33,992::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913173::DEBUG::2013-11-20 02:57:33,992::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913173::DEBUG::2013-11-20 02:57:33,993::task::495::TaskManager.Task::(_debug) Task 4e49640b-9dc4-4809-ac2c-f24decd5b1f3: ref 0 aborting False >Thread-15913173::INFO::2013-11-20 02:57:33,993::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:57:34,177::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:34,710::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0747212 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:57:36,727::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:37,256::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0734809 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:57:39,271::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:39,826::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0706066 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15913176::INFO::2013-11-20 02:57:41,103::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913176::DEBUG::2013-11-20 02:57:41,104::task::495::TaskManager.Task::(_debug) Task 008ff3e4-d7f5-4b3a-a079-45169d22ca7d: moving from state init -> state preparing >Thread-15913176::DEBUG::2013-11-20 02:57:41,105::task::495::TaskManager.Task::(_debug) Task 008ff3e4-d7f5-4b3a-a079-45169d22ca7d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000314950942993', 'lastCheck': 1384916254.409488, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00159692764282', 'lastCheck': 1384916252.3902881, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00330018997192', 'lastCheck': 1384916253.393724, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00049901008606', 'lastCheck': 1384916251.9233911, 'valid': True, 'code': 0}} >Thread-15913176::DEBUG::2013-11-20 02:57:41,106::task::495::TaskManager.Task::(_debug) Task 008ff3e4-d7f5-4b3a-a079-45169d22ca7d: moving from state preparing -> state finished >Thread-15913176::DEBUG::2013-11-20 02:57:41,106::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913176::DEBUG::2013-11-20 02:57:41,107::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913176::DEBUG::2013-11-20 02:57:41,107::task::495::TaskManager.Task::(_debug) Task 008ff3e4-d7f5-4b3a-a079-45169d22ca7d: ref 0 aborting False >Thread-15913176::INFO::2013-11-20 02:57:41,108::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000314950942993', 'lastCheck': 1384916254.409488, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00159692764282', 'lastCheck': 1384916252.3902881, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00330018997192', 'lastCheck': 1384916253.393724, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00049901008606', 'lastCheck': 1384916251.9233911, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:57:41,842::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:42,333::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0707183 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15913178::DEBUG::2013-11-20 02:57:44,109::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913178::INFO::2013-11-20 02:57:44,110::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913178::DEBUG::2013-11-20 02:57:44,111::task::495::TaskManager.Task::(_debug) Task a433fefe-c014-432f-ac2b-516c20498255: moving from state init -> state preparing >Thread-15913178::DEBUG::2013-11-20 02:57:44,112::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913178::DEBUG::2013-11-20 02:57:44,112::task::495::TaskManager.Task::(_debug) Task a433fefe-c014-432f-ac2b-516c20498255: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913178::DEBUG::2013-11-20 02:57:44,113::task::495::TaskManager.Task::(_debug) Task a433fefe-c014-432f-ac2b-516c20498255: moving from state preparing -> state finished >Thread-15913178::DEBUG::2013-11-20 02:57:44,113::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913178::DEBUG::2013-11-20 02:57:44,114::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913178::DEBUG::2013-11-20 02:57:44,115::task::495::TaskManager.Task::(_debug) Task a433fefe-c014-432f-ac2b-516c20498255: ref 0 aborting False >Thread-15913178::INFO::2013-11-20 02:57:44,115::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913179::DEBUG::2013-11-20 02:57:44,120::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913179::INFO::2013-11-20 02:57:44,121::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913179::DEBUG::2013-11-20 02:57:44,121::task::495::TaskManager.Task::(_debug) Task 6b4fd3a4-925b-4c3c-ae64-5becdaf04aa0: moving from state init -> state preparing >Thread-15913179::DEBUG::2013-11-20 02:57:44,123::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`7c986911-dad0-4529-b9e8-d544afd74732`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913179::DEBUG::2013-11-20 02:57:44,123::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913179::DEBUG::2013-11-20 02:57:44,124::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913179::DEBUG::2013-11-20 02:57:44,125::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`7c986911-dad0-4529-b9e8-d544afd74732`::Granted request >Thread-15913179::DEBUG::2013-11-20 02:57:44,126::task::495::TaskManager.Task::(_debug) Task 6b4fd3a4-925b-4c3c-ae64-5becdaf04aa0: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913179::DEBUG::2013-11-20 02:57:44,126::task::495::TaskManager.Task::(_debug) Task 6b4fd3a4-925b-4c3c-ae64-5becdaf04aa0: ref 1 aborting False >Thread-15913179::WARNING::2013-11-20 02:57:44,128::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913179::DEBUG::2013-11-20 02:57:44,129::task::495::TaskManager.Task::(_debug) Task 6b4fd3a4-925b-4c3c-ae64-5becdaf04aa0: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913179::DEBUG::2013-11-20 02:57:44,130::task::495::TaskManager.Task::(_debug) Task 6b4fd3a4-925b-4c3c-ae64-5becdaf04aa0: moving from state preparing -> state finished >Thread-15913179::DEBUG::2013-11-20 02:57:44,130::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913179::DEBUG::2013-11-20 02:57:44,131::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913179::DEBUG::2013-11-20 02:57:44,132::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913179::DEBUG::2013-11-20 02:57:44,132::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913179::DEBUG::2013-11-20 02:57:44,133::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913179::DEBUG::2013-11-20 02:57:44,133::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913179::DEBUG::2013-11-20 02:57:44,134::task::495::TaskManager.Task::(_debug) Task 6b4fd3a4-925b-4c3c-ae64-5becdaf04aa0: ref 0 aborting False >Thread-15913179::INFO::2013-11-20 02:57:44,135::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:57:44,348::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:44,882::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0726808 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:57:46,898::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:47,441::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0737351 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:57:49,457::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:49,978::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0724681 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:57:51,993::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:52,531::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0815752 s, 12.6 MB/s\n'; <rc> = 0 >Thread-15913184::DEBUG::2013-11-20 02:57:54,282::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913184::INFO::2013-11-20 02:57:54,284::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913184::DEBUG::2013-11-20 02:57:54,284::task::495::TaskManager.Task::(_debug) Task e7d6a26d-9b41-4178-86f0-edaca0c77189: moving from state init -> state preparing >Thread-15913184::DEBUG::2013-11-20 02:57:54,285::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913184::DEBUG::2013-11-20 02:57:54,286::task::495::TaskManager.Task::(_debug) Task e7d6a26d-9b41-4178-86f0-edaca0c77189: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913184::DEBUG::2013-11-20 02:57:54,286::task::495::TaskManager.Task::(_debug) Task e7d6a26d-9b41-4178-86f0-edaca0c77189: moving from state preparing -> state finished >Thread-15913184::DEBUG::2013-11-20 02:57:54,286::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913184::DEBUG::2013-11-20 02:57:54,287::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913184::DEBUG::2013-11-20 02:57:54,287::task::495::TaskManager.Task::(_debug) Task e7d6a26d-9b41-4178-86f0-edaca0c77189: ref 0 aborting False >Thread-15913184::INFO::2013-11-20 02:57:54,287::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913185::DEBUG::2013-11-20 02:57:54,291::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913185::INFO::2013-11-20 02:57:54,292::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913185::DEBUG::2013-11-20 02:57:54,292::task::495::TaskManager.Task::(_debug) Task 74852903-51bc-462c-b7cb-e5caf4eaeb5c: moving from state init -> state preparing >Thread-15913185::DEBUG::2013-11-20 02:57:54,293::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`66f6711e-9011-4ff0-b344-6edafa1f5b6a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913185::DEBUG::2013-11-20 02:57:54,293::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913185::DEBUG::2013-11-20 02:57:54,293::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913185::DEBUG::2013-11-20 02:57:54,294::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`66f6711e-9011-4ff0-b344-6edafa1f5b6a`::Granted request >Thread-15913185::DEBUG::2013-11-20 02:57:54,299::task::495::TaskManager.Task::(_debug) Task 74852903-51bc-462c-b7cb-e5caf4eaeb5c: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913185::DEBUG::2013-11-20 02:57:54,299::task::495::TaskManager.Task::(_debug) Task 74852903-51bc-462c-b7cb-e5caf4eaeb5c: ref 1 aborting False >Thread-15913185::WARNING::2013-11-20 02:57:54,300::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913185::DEBUG::2013-11-20 02:57:54,301::task::495::TaskManager.Task::(_debug) Task 74852903-51bc-462c-b7cb-e5caf4eaeb5c: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913185::DEBUG::2013-11-20 02:57:54,301::task::495::TaskManager.Task::(_debug) Task 74852903-51bc-462c-b7cb-e5caf4eaeb5c: moving from state preparing -> state finished >Thread-15913185::DEBUG::2013-11-20 02:57:54,301::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913185::DEBUG::2013-11-20 02:57:54,302::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913185::DEBUG::2013-11-20 02:57:54,302::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913185::DEBUG::2013-11-20 02:57:54,303::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913185::DEBUG::2013-11-20 02:57:54,303::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913185::DEBUG::2013-11-20 02:57:54,303::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913185::DEBUG::2013-11-20 02:57:54,304::task::495::TaskManager.Task::(_debug) Task 74852903-51bc-462c-b7cb-e5caf4eaeb5c: ref 0 aborting False >Thread-15913185::INFO::2013-11-20 02:57:54,304::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:57:54,547::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:55,086::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0751298 s, 13.6 MB/s\n'; <rc> = 0 >Thread-15913186::INFO::2013-11-20 02:57:57,007::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913186::DEBUG::2013-11-20 02:57:57,008::task::495::TaskManager.Task::(_debug) Task c51d0568-1dfd-4762-84f0-2243b4084675: moving from state init -> state preparing >Thread-15913186::DEBUG::2013-11-20 02:57:57,009::task::495::TaskManager.Task::(_debug) Task c51d0568-1dfd-4762-84f0-2243b4084675: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000292778015137', 'lastCheck': 1384916274.7786219, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0014009475708', 'lastCheck': 1384916272.670645, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00188088417053', 'lastCheck': 1384916273.672101, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000557899475098', 'lastCheck': 1384916272.216763, 'valid': True, 'code': 0}} >Thread-15913186::DEBUG::2013-11-20 02:57:57,010::task::495::TaskManager.Task::(_debug) Task c51d0568-1dfd-4762-84f0-2243b4084675: moving from state preparing -> state finished >Thread-15913186::DEBUG::2013-11-20 02:57:57,010::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913186::DEBUG::2013-11-20 02:57:57,011::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913186::DEBUG::2013-11-20 02:57:57,011::task::495::TaskManager.Task::(_debug) Task c51d0568-1dfd-4762-84f0-2243b4084675: ref 0 aborting False >Thread-15913186::INFO::2013-11-20 02:57:57,012::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000292778015137', 'lastCheck': 1384916274.7786219, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0014009475708', 'lastCheck': 1384916272.670645, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00188088417053', 'lastCheck': 1384916273.672101, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000557899475098', 'lastCheck': 1384916272.216763, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:57:57,102::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:57:57,638::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0972072 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:57:59,654::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:00,216::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0887499 s, 11.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:58:02,231::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:02,730::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0545971 s, 18.8 MB/s\n'; <rc> = 0 >Thread-15913190::DEBUG::2013-11-20 02:58:04,448::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913190::INFO::2013-11-20 02:58:04,450::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913190::DEBUG::2013-11-20 02:58:04,450::task::495::TaskManager.Task::(_debug) Task 4237b9fa-025d-430d-8149-018b3b39f555: moving from state init -> state preparing >Thread-15913190::DEBUG::2013-11-20 02:58:04,451::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913190::DEBUG::2013-11-20 02:58:04,452::task::495::TaskManager.Task::(_debug) Task 4237b9fa-025d-430d-8149-018b3b39f555: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913190::DEBUG::2013-11-20 02:58:04,452::task::495::TaskManager.Task::(_debug) Task 4237b9fa-025d-430d-8149-018b3b39f555: moving from state preparing -> state finished >Thread-15913190::DEBUG::2013-11-20 02:58:04,453::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913190::DEBUG::2013-11-20 02:58:04,453::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913190::DEBUG::2013-11-20 02:58:04,454::task::495::TaskManager.Task::(_debug) Task 4237b9fa-025d-430d-8149-018b3b39f555: ref 0 aborting False >Thread-15913190::INFO::2013-11-20 02:58:04,455::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913191::DEBUG::2013-11-20 02:58:04,459::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913191::INFO::2013-11-20 02:58:04,460::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913191::DEBUG::2013-11-20 02:58:04,461::task::495::TaskManager.Task::(_debug) Task 1ae799d6-976f-4dbf-bb21-9e26799c843c: moving from state init -> state preparing >Thread-15913191::DEBUG::2013-11-20 02:58:04,462::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`774237ee-fe85-4db5-bf90-f938e0a87f67`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913191::DEBUG::2013-11-20 02:58:04,463::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913191::DEBUG::2013-11-20 02:58:04,464::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913191::DEBUG::2013-11-20 02:58:04,465::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`774237ee-fe85-4db5-bf90-f938e0a87f67`::Granted request >Thread-15913191::DEBUG::2013-11-20 02:58:04,466::task::495::TaskManager.Task::(_debug) Task 1ae799d6-976f-4dbf-bb21-9e26799c843c: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913191::DEBUG::2013-11-20 02:58:04,466::task::495::TaskManager.Task::(_debug) Task 1ae799d6-976f-4dbf-bb21-9e26799c843c: ref 1 aborting False >Thread-15913191::WARNING::2013-11-20 02:58:04,468::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913191::DEBUG::2013-11-20 02:58:04,469::task::495::TaskManager.Task::(_debug) Task 1ae799d6-976f-4dbf-bb21-9e26799c843c: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913191::DEBUG::2013-11-20 02:58:04,470::task::495::TaskManager.Task::(_debug) Task 1ae799d6-976f-4dbf-bb21-9e26799c843c: moving from state preparing -> state finished >Thread-15913191::DEBUG::2013-11-20 02:58:04,471::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913191::DEBUG::2013-11-20 02:58:04,471::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913191::DEBUG::2013-11-20 02:58:04,472::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913191::DEBUG::2013-11-20 02:58:04,473::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913191::DEBUG::2013-11-20 02:58:04,473::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913191::DEBUG::2013-11-20 02:58:04,474::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913191::DEBUG::2013-11-20 02:58:04,475::task::495::TaskManager.Task::(_debug) Task 1ae799d6-976f-4dbf-bb21-9e26799c843c: ref 0 aborting False >Thread-15913191::INFO::2013-11-20 02:58:04,475::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:58:04,745::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:05,235::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0580124 s, 17.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:58:07,245::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:07,767::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0729963 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:58:09,783::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:10,356::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0767875 s, 13.3 MB/s\n'; <rc> = 0 >Thread-15913194::INFO::2013-11-20 02:58:12,260::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913194::DEBUG::2013-11-20 02:58:12,261::task::495::TaskManager.Task::(_debug) Task dce65abc-491d-439e-b924-c9e108f17823: moving from state init -> state preparing >Thread-15913194::DEBUG::2013-11-20 02:58:12,262::task::495::TaskManager.Task::(_debug) Task dce65abc-491d-439e-b924-c9e108f17823: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00039005279541', 'lastCheck': 1384916284.990808, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00161695480347', 'lastCheck': 1384916282.8854771, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00132703781128', 'lastCheck': 1384916283.8859019, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000602960586548', 'lastCheck': 1384916282.4275939, 'valid': True, 'code': 0}} >Thread-15913194::DEBUG::2013-11-20 02:58:12,263::task::495::TaskManager.Task::(_debug) Task dce65abc-491d-439e-b924-c9e108f17823: moving from state preparing -> state finished >Thread-15913194::DEBUG::2013-11-20 02:58:12,264::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913194::DEBUG::2013-11-20 02:58:12,264::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913194::DEBUG::2013-11-20 02:58:12,265::task::495::TaskManager.Task::(_debug) Task dce65abc-491d-439e-b924-c9e108f17823: ref 0 aborting False >Thread-15913194::INFO::2013-11-20 02:58:12,265::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00039005279541', 'lastCheck': 1384916284.990808, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00161695480347', 'lastCheck': 1384916282.8854771, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00132703781128', 'lastCheck': 1384916283.8859019, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000602960586548', 'lastCheck': 1384916282.4275939, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:58:12,370::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:12,987::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.116145 s, 8.8 MB/s\n'; <rc> = 0 >Thread-15913196::DEBUG::2013-11-20 02:58:14,606::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913196::INFO::2013-11-20 02:58:14,608::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913196::DEBUG::2013-11-20 02:58:14,608::task::495::TaskManager.Task::(_debug) Task ba113087-968a-499c-9c23-4396b804c164: moving from state init -> state preparing >Thread-15913196::DEBUG::2013-11-20 02:58:14,609::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913196::DEBUG::2013-11-20 02:58:14,610::task::495::TaskManager.Task::(_debug) Task ba113087-968a-499c-9c23-4396b804c164: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913196::DEBUG::2013-11-20 02:58:14,610::task::495::TaskManager.Task::(_debug) Task ba113087-968a-499c-9c23-4396b804c164: moving from state preparing -> state finished >Thread-15913196::DEBUG::2013-11-20 02:58:14,611::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913196::DEBUG::2013-11-20 02:58:14,611::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913196::DEBUG::2013-11-20 02:58:14,612::task::495::TaskManager.Task::(_debug) Task ba113087-968a-499c-9c23-4396b804c164: ref 0 aborting False >Thread-15913196::INFO::2013-11-20 02:58:14,612::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913197::DEBUG::2013-11-20 02:58:14,617::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913197::INFO::2013-11-20 02:58:14,618::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913197::DEBUG::2013-11-20 02:58:14,618::task::495::TaskManager.Task::(_debug) Task 6a46442a-0c39-45b9-bfdc-a3fcf105dac8: moving from state init -> state preparing >Thread-15913197::DEBUG::2013-11-20 02:58:14,619::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2e6f4010-d964-4e45-adf2-e5e4abc0d89e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913197::DEBUG::2013-11-20 02:58:14,620::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913197::DEBUG::2013-11-20 02:58:14,621::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913197::DEBUG::2013-11-20 02:58:14,621::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2e6f4010-d964-4e45-adf2-e5e4abc0d89e`::Granted request >Thread-15913197::DEBUG::2013-11-20 02:58:14,622::task::495::TaskManager.Task::(_debug) Task 6a46442a-0c39-45b9-bfdc-a3fcf105dac8: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913197::DEBUG::2013-11-20 02:58:14,623::task::495::TaskManager.Task::(_debug) Task 6a46442a-0c39-45b9-bfdc-a3fcf105dac8: ref 1 aborting False >Thread-15913197::WARNING::2013-11-20 02:58:14,625::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913197::DEBUG::2013-11-20 02:58:14,626::task::495::TaskManager.Task::(_debug) Task 6a46442a-0c39-45b9-bfdc-a3fcf105dac8: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913197::DEBUG::2013-11-20 02:58:14,626::task::495::TaskManager.Task::(_debug) Task 6a46442a-0c39-45b9-bfdc-a3fcf105dac8: moving from state preparing -> state finished >Thread-15913197::DEBUG::2013-11-20 02:58:14,627::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913197::DEBUG::2013-11-20 02:58:14,628::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913197::DEBUG::2013-11-20 02:58:14,628::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913197::DEBUG::2013-11-20 02:58:14,629::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913197::DEBUG::2013-11-20 02:58:14,629::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913197::DEBUG::2013-11-20 02:58:14,630::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913197::DEBUG::2013-11-20 02:58:14,630::task::495::TaskManager.Task::(_debug) Task 6a46442a-0c39-45b9-bfdc-a3fcf105dac8: ref 0 aborting False >Thread-15913197::INFO::2013-11-20 02:58:14,631::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:58:14,997::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 02:58:15,265::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:58:15,271::task::495::TaskManager.Task::(_debug) Task 1af01982-80b7-4513-8655-7f049a6c2e38: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:58:15,273::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`4714b54d-b82e-4c01-9fa7-a61773f90d54`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:58:15,274::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:58:15,274::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:58:15,275::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`4714b54d-b82e-4c01-9fa7-a61773f90d54`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:58:15,275::task::495::TaskManager.Task::(_debug) Task 1af01982-80b7-4513-8655-7f049a6c2e38: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:58:15,276::task::495::TaskManager.Task::(_debug) Task 1af01982-80b7-4513-8655-7f049a6c2e38: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:58:15,276::task::495::TaskManager.Task::(_debug) Task 1af01982-80b7-4513-8655-7f049a6c2e38: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:58:15,277::task::495::TaskManager.Task::(_debug) Task 1af01982-80b7-4513-8655-7f049a6c2e38: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:58:15,277::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:58:15,277::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:58:15,278::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:58:15,278::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:58:15,278::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:58:15,279::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:58:15,279::task::495::TaskManager.Task::(_debug) Task 1af01982-80b7-4513-8655-7f049a6c2e38: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:58:15,279::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 02:58:15,510::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0761971 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:58:17,534::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:17,997::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0455901 s, 22.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:58:20,008::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:20,546::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0759617 s, 13.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:58:22,563::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:23,091::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0766346 s, 13.4 MB/s\n'; <rc> = 0 >Thread-15913202::DEBUG::2013-11-20 02:58:24,763::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913202::INFO::2013-11-20 02:58:24,764::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913202::DEBUG::2013-11-20 02:58:24,764::task::495::TaskManager.Task::(_debug) Task c46e6aa9-10f1-4e76-881b-7629934eb270: moving from state init -> state preparing >Thread-15913202::DEBUG::2013-11-20 02:58:24,765::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913202::DEBUG::2013-11-20 02:58:24,766::task::495::TaskManager.Task::(_debug) Task c46e6aa9-10f1-4e76-881b-7629934eb270: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913202::DEBUG::2013-11-20 02:58:24,766::task::495::TaskManager.Task::(_debug) Task c46e6aa9-10f1-4e76-881b-7629934eb270: moving from state preparing -> state finished >Thread-15913202::DEBUG::2013-11-20 02:58:24,767::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913202::DEBUG::2013-11-20 02:58:24,768::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913202::DEBUG::2013-11-20 02:58:24,768::task::495::TaskManager.Task::(_debug) Task c46e6aa9-10f1-4e76-881b-7629934eb270: ref 0 aborting False >Thread-15913202::INFO::2013-11-20 02:58:24,769::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913203::DEBUG::2013-11-20 02:58:24,773::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913203::INFO::2013-11-20 02:58:24,774::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913203::DEBUG::2013-11-20 02:58:24,775::task::495::TaskManager.Task::(_debug) Task f9693cf3-570a-4511-9fef-aa4708fc3fd0: moving from state init -> state preparing >Thread-15913203::DEBUG::2013-11-20 02:58:24,776::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2092780a-5b76-4e85-9146-e12df08eaa35`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913203::DEBUG::2013-11-20 02:58:24,777::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913203::DEBUG::2013-11-20 02:58:24,778::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913203::DEBUG::2013-11-20 02:58:24,778::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2092780a-5b76-4e85-9146-e12df08eaa35`::Granted request >Thread-15913203::DEBUG::2013-11-20 02:58:24,779::task::495::TaskManager.Task::(_debug) Task f9693cf3-570a-4511-9fef-aa4708fc3fd0: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913203::DEBUG::2013-11-20 02:58:24,780::task::495::TaskManager.Task::(_debug) Task f9693cf3-570a-4511-9fef-aa4708fc3fd0: ref 1 aborting False >Thread-15913203::WARNING::2013-11-20 02:58:24,782::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913203::DEBUG::2013-11-20 02:58:24,783::task::495::TaskManager.Task::(_debug) Task f9693cf3-570a-4511-9fef-aa4708fc3fd0: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913203::DEBUG::2013-11-20 02:58:24,784::task::495::TaskManager.Task::(_debug) Task f9693cf3-570a-4511-9fef-aa4708fc3fd0: moving from state preparing -> state finished >Thread-15913203::DEBUG::2013-11-20 02:58:24,784::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913203::DEBUG::2013-11-20 02:58:24,785::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913203::DEBUG::2013-11-20 02:58:24,786::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913203::DEBUG::2013-11-20 02:58:24,786::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913203::DEBUG::2013-11-20 02:58:24,787::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913203::DEBUG::2013-11-20 02:58:24,787::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913203::DEBUG::2013-11-20 02:58:24,788::task::495::TaskManager.Task::(_debug) Task f9693cf3-570a-4511-9fef-aa4708fc3fd0: ref 0 aborting False >Thread-15913203::INFO::2013-11-20 02:58:24,789::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:58:25,106::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:25,693::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.103655 s, 9.9 MB/s\n'; <rc> = 0 >Thread-15913204::INFO::2013-11-20 02:58:27,513::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913204::DEBUG::2013-11-20 02:58:27,514::task::495::TaskManager.Task::(_debug) Task f18d68fa-7604-4e00-90d7-1dba6f6c07e1: moving from state init -> state preparing >Thread-15913204::DEBUG::2013-11-20 02:58:27,515::task::495::TaskManager.Task::(_debug) Task f18d68fa-7604-4e00-90d7-1dba6f6c07e1: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000434875488281', 'lastCheck': 1384916305.3080609, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00228691101074', 'lastCheck': 1384916303.2722261, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00173807144165', 'lastCheck': 1384916304.2608531, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000536918640137', 'lastCheck': 1384916302.7748311, 'valid': True, 'code': 0}} >Thread-15913204::DEBUG::2013-11-20 02:58:27,515::task::495::TaskManager.Task::(_debug) Task f18d68fa-7604-4e00-90d7-1dba6f6c07e1: moving from state preparing -> state finished >Thread-15913204::DEBUG::2013-11-20 02:58:27,516::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913204::DEBUG::2013-11-20 02:58:27,517::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913204::DEBUG::2013-11-20 02:58:27,517::task::495::TaskManager.Task::(_debug) Task f18d68fa-7604-4e00-90d7-1dba6f6c07e1: ref 0 aborting False >Thread-15913204::INFO::2013-11-20 02:58:27,518::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000434875488281', 'lastCheck': 1384916305.3080609, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00228691101074', 'lastCheck': 1384916303.2722261, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00173807144165', 'lastCheck': 1384916304.2608531, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000536918640137', 'lastCheck': 1384916302.7748311, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:58:27,717::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:28,257::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0744243 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:58:30,271::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:30,817::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0731876 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:58:32,832::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:33,296::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0523005 s, 19.6 MB/s\n'; <rc> = 0 >Thread-15913208::DEBUG::2013-11-20 02:58:34,894::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913208::INFO::2013-11-20 02:58:34,895::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913208::DEBUG::2013-11-20 02:58:34,896::task::495::TaskManager.Task::(_debug) Task ea7be426-66b0-4a6c-b838-9d0d971db179: moving from state init -> state preparing >Thread-15913208::DEBUG::2013-11-20 02:58:34,897::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913208::DEBUG::2013-11-20 02:58:34,897::task::495::TaskManager.Task::(_debug) Task ea7be426-66b0-4a6c-b838-9d0d971db179: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913208::DEBUG::2013-11-20 02:58:34,898::task::495::TaskManager.Task::(_debug) Task ea7be426-66b0-4a6c-b838-9d0d971db179: moving from state preparing -> state finished >Thread-15913208::DEBUG::2013-11-20 02:58:34,899::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913208::DEBUG::2013-11-20 02:58:34,899::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913208::DEBUG::2013-11-20 02:58:34,900::task::495::TaskManager.Task::(_debug) Task ea7be426-66b0-4a6c-b838-9d0d971db179: ref 0 aborting False >Thread-15913208::INFO::2013-11-20 02:58:34,900::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913209::DEBUG::2013-11-20 02:58:34,905::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913209::INFO::2013-11-20 02:58:34,906::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913209::DEBUG::2013-11-20 02:58:34,906::task::495::TaskManager.Task::(_debug) Task 64b64f5d-848f-41a4-9a35-4b592df5a7a7: moving from state init -> state preparing >Thread-15913209::DEBUG::2013-11-20 02:58:34,907::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`390c4272-abfc-420e-9727-2e52a27256cb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913209::DEBUG::2013-11-20 02:58:34,908::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913209::DEBUG::2013-11-20 02:58:34,909::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913209::DEBUG::2013-11-20 02:58:34,909::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`390c4272-abfc-420e-9727-2e52a27256cb`::Granted request >Thread-15913209::DEBUG::2013-11-20 02:58:34,910::task::495::TaskManager.Task::(_debug) Task 64b64f5d-848f-41a4-9a35-4b592df5a7a7: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913209::DEBUG::2013-11-20 02:58:34,911::task::495::TaskManager.Task::(_debug) Task 64b64f5d-848f-41a4-9a35-4b592df5a7a7: ref 1 aborting False >Thread-15913209::WARNING::2013-11-20 02:58:34,913::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913209::DEBUG::2013-11-20 02:58:34,914::task::495::TaskManager.Task::(_debug) Task 64b64f5d-848f-41a4-9a35-4b592df5a7a7: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913209::DEBUG::2013-11-20 02:58:34,914::task::495::TaskManager.Task::(_debug) Task 64b64f5d-848f-41a4-9a35-4b592df5a7a7: moving from state preparing -> state finished >Thread-15913209::DEBUG::2013-11-20 02:58:34,915::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913209::DEBUG::2013-11-20 02:58:34,916::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913209::DEBUG::2013-11-20 02:58:34,916::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913209::DEBUG::2013-11-20 02:58:34,917::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913209::DEBUG::2013-11-20 02:58:34,917::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913209::DEBUG::2013-11-20 02:58:34,918::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913209::DEBUG::2013-11-20 02:58:34,918::task::495::TaskManager.Task::(_debug) Task 64b64f5d-848f-41a4-9a35-4b592df5a7a7: ref 0 aborting False >Thread-15913209::INFO::2013-11-20 02:58:34,919::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:58:35,307::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:35,830::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0776648 s, 13.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:58:37,845::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:38,407::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0733467 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:58:40,423::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:40,960::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0751801 s, 13.6 MB/s\n'; <rc> = 0 >Thread-15913212::INFO::2013-11-20 02:58:42,825::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913212::DEBUG::2013-11-20 02:58:42,826::task::495::TaskManager.Task::(_debug) Task 8580fde6-849b-48ef-a005-2c5ff1db4215: moving from state init -> state preparing >Thread-15913212::DEBUG::2013-11-20 02:58:42,826::task::495::TaskManager.Task::(_debug) Task 8580fde6-849b-48ef-a005-2c5ff1db4215: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000668048858643', 'lastCheck': 1384916315.518204, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0011100769043', 'lastCheck': 1384916313.507431, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00123286247253', 'lastCheck': 1384916314.5126541, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000712871551514', 'lastCheck': 1384916313.0202761, 'valid': True, 'code': 0}} >Thread-15913212::DEBUG::2013-11-20 02:58:42,827::task::495::TaskManager.Task::(_debug) Task 8580fde6-849b-48ef-a005-2c5ff1db4215: moving from state preparing -> state finished >Thread-15913212::DEBUG::2013-11-20 02:58:42,828::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913212::DEBUG::2013-11-20 02:58:42,828::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913212::DEBUG::2013-11-20 02:58:42,829::task::495::TaskManager.Task::(_debug) Task 8580fde6-849b-48ef-a005-2c5ff1db4215: ref 0 aborting False >Thread-15913212::INFO::2013-11-20 02:58:42,830::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000668048858643', 'lastCheck': 1384916315.518204, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0011100769043', 'lastCheck': 1384916313.507431, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00123286247253', 'lastCheck': 1384916314.5126541, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000712871551514', 'lastCheck': 1384916313.0202761, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:58:42,975::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:43,491::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0835166 s, 12.3 MB/s\n'; <rc> = 0 >Thread-15913214::DEBUG::2013-11-20 02:58:45,132::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913214::INFO::2013-11-20 02:58:45,133::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913214::DEBUG::2013-11-20 02:58:45,134::task::495::TaskManager.Task::(_debug) Task 2485399d-7f95-41fd-9a44-710261c5a268: moving from state init -> state preparing >Thread-15913214::DEBUG::2013-11-20 02:58:45,135::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913214::DEBUG::2013-11-20 02:58:45,135::task::495::TaskManager.Task::(_debug) Task 2485399d-7f95-41fd-9a44-710261c5a268: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913214::DEBUG::2013-11-20 02:58:45,136::task::495::TaskManager.Task::(_debug) Task 2485399d-7f95-41fd-9a44-710261c5a268: moving from state preparing -> state finished >Thread-15913214::DEBUG::2013-11-20 02:58:45,136::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913214::DEBUG::2013-11-20 02:58:45,137::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913214::DEBUG::2013-11-20 02:58:45,137::task::495::TaskManager.Task::(_debug) Task 2485399d-7f95-41fd-9a44-710261c5a268: ref 0 aborting False >Thread-15913214::INFO::2013-11-20 02:58:45,138::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913215::DEBUG::2013-11-20 02:58:45,279::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913215::INFO::2013-11-20 02:58:45,280::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913215::DEBUG::2013-11-20 02:58:45,281::task::495::TaskManager.Task::(_debug) Task 145bd003-6dfe-4f88-a861-01a959815248: moving from state init -> state preparing >Thread-15913215::DEBUG::2013-11-20 02:58:45,282::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`32a00f17-bf7d-4167-9f10-45d8b1fbc689`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913215::DEBUG::2013-11-20 02:58:45,283::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913215::DEBUG::2013-11-20 02:58:45,284::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913215::DEBUG::2013-11-20 02:58:45,284::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`32a00f17-bf7d-4167-9f10-45d8b1fbc689`::Granted request >Thread-15913215::DEBUG::2013-11-20 02:58:45,285::task::495::TaskManager.Task::(_debug) Task 145bd003-6dfe-4f88-a861-01a959815248: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913215::DEBUG::2013-11-20 02:58:45,286::task::495::TaskManager.Task::(_debug) Task 145bd003-6dfe-4f88-a861-01a959815248: ref 1 aborting False >Thread-15913215::WARNING::2013-11-20 02:58:45,287::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913215::DEBUG::2013-11-20 02:58:45,288::task::495::TaskManager.Task::(_debug) Task 145bd003-6dfe-4f88-a861-01a959815248: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913215::DEBUG::2013-11-20 02:58:45,289::task::495::TaskManager.Task::(_debug) Task 145bd003-6dfe-4f88-a861-01a959815248: moving from state preparing -> state finished >Thread-15913215::DEBUG::2013-11-20 02:58:45,290::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913215::DEBUG::2013-11-20 02:58:45,290::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913215::DEBUG::2013-11-20 02:58:45,291::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913215::DEBUG::2013-11-20 02:58:45,291::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913215::DEBUG::2013-11-20 02:58:45,292::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913215::DEBUG::2013-11-20 02:58:45,292::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913215::DEBUG::2013-11-20 02:58:45,293::task::495::TaskManager.Task::(_debug) Task 145bd003-6dfe-4f88-a861-01a959815248: ref 0 aborting False >Thread-15913215::INFO::2013-11-20 02:58:45,294::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:58:45,506::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:46,072::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.073462 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:58:48,087::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:48,627::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0742011 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:58:50,642::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:51,198::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0749834 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:58:53,213::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-42840::DEBUG::2013-11-20 02:58:53,426::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:53,861::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.127751 s, 8.0 MB/s\n'; <rc> = 0 >Thread-42840::DEBUG::2013-11-20 02:58:54,205::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913220::DEBUG::2013-11-20 02:58:55,473::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913220::INFO::2013-11-20 02:58:55,475::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913220::DEBUG::2013-11-20 02:58:55,475::task::495::TaskManager.Task::(_debug) Task 6943cb9d-0d9a-4fd8-a2ed-c0b99cc2a526: moving from state init -> state preparing >Thread-15913220::DEBUG::2013-11-20 02:58:55,476::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913220::DEBUG::2013-11-20 02:58:55,477::task::495::TaskManager.Task::(_debug) Task 6943cb9d-0d9a-4fd8-a2ed-c0b99cc2a526: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913220::DEBUG::2013-11-20 02:58:55,477::task::495::TaskManager.Task::(_debug) Task 6943cb9d-0d9a-4fd8-a2ed-c0b99cc2a526: moving from state preparing -> state finished >Thread-15913220::DEBUG::2013-11-20 02:58:55,477::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913220::DEBUG::2013-11-20 02:58:55,478::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913220::DEBUG::2013-11-20 02:58:55,478::task::495::TaskManager.Task::(_debug) Task 6943cb9d-0d9a-4fd8-a2ed-c0b99cc2a526: ref 0 aborting False >Thread-15913220::INFO::2013-11-20 02:58:55,479::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913221::DEBUG::2013-11-20 02:58:55,482::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913221::INFO::2013-11-20 02:58:55,482::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913221::DEBUG::2013-11-20 02:58:55,483::task::495::TaskManager.Task::(_debug) Task 3e58ceff-a3be-4827-ae6d-8a8b880a84cc: moving from state init -> state preparing >Thread-15913221::DEBUG::2013-11-20 02:58:55,483::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c559c498-407a-44d1-af56-9e7492f6df99`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913221::DEBUG::2013-11-20 02:58:55,484::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913221::DEBUG::2013-11-20 02:58:55,484::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913221::DEBUG::2013-11-20 02:58:55,485::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c559c498-407a-44d1-af56-9e7492f6df99`::Granted request >Thread-15913221::DEBUG::2013-11-20 02:58:55,486::task::495::TaskManager.Task::(_debug) Task 3e58ceff-a3be-4827-ae6d-8a8b880a84cc: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913221::DEBUG::2013-11-20 02:58:55,487::task::495::TaskManager.Task::(_debug) Task 3e58ceff-a3be-4827-ae6d-8a8b880a84cc: ref 1 aborting False >Thread-15913221::WARNING::2013-11-20 02:58:55,488::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913221::DEBUG::2013-11-20 02:58:55,489::task::495::TaskManager.Task::(_debug) Task 3e58ceff-a3be-4827-ae6d-8a8b880a84cc: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913221::DEBUG::2013-11-20 02:58:55,490::task::495::TaskManager.Task::(_debug) Task 3e58ceff-a3be-4827-ae6d-8a8b880a84cc: moving from state preparing -> state finished >Thread-15913221::DEBUG::2013-11-20 02:58:55,491::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913221::DEBUG::2013-11-20 02:58:55,491::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913221::DEBUG::2013-11-20 02:58:55,492::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913221::DEBUG::2013-11-20 02:58:55,493::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913221::DEBUG::2013-11-20 02:58:55,493::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913221::DEBUG::2013-11-20 02:58:55,493::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913221::DEBUG::2013-11-20 02:58:55,494::task::495::TaskManager.Task::(_debug) Task 3e58ceff-a3be-4827-ae6d-8a8b880a84cc: ref 0 aborting False >Thread-15913221::INFO::2013-11-20 02:58:55,495::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:58:55,872::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:56,407::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0793054 s, 12.9 MB/s\n'; <rc> = 0 >Thread-15913222::INFO::2013-11-20 02:58:58,312::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913222::DEBUG::2013-11-20 02:58:58,312::task::495::TaskManager.Task::(_debug) Task bc4d6559-b598-4bc2-80ca-fabd7254a2a4: moving from state init -> state preparing >Thread-15913222::DEBUG::2013-11-20 02:58:58,313::task::495::TaskManager.Task::(_debug) Task bc4d6559-b598-4bc2-80ca-fabd7254a2a4: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000589847564697', 'lastCheck': 1384916336.1002679, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150799751282', 'lastCheck': 1384916333.889853, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00127482414246', 'lastCheck': 1384916334.8867331, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000324010848999', 'lastCheck': 1384916334.2075441, 'valid': True, 'code': 0}} >Thread-15913222::DEBUG::2013-11-20 02:58:58,314::task::495::TaskManager.Task::(_debug) Task bc4d6559-b598-4bc2-80ca-fabd7254a2a4: moving from state preparing -> state finished >Thread-15913222::DEBUG::2013-11-20 02:58:58,315::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913222::DEBUG::2013-11-20 02:58:58,315::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913222::DEBUG::2013-11-20 02:58:58,316::task::495::TaskManager.Task::(_debug) Task bc4d6559-b598-4bc2-80ca-fabd7254a2a4: ref 0 aborting False >Thread-15913222::INFO::2013-11-20 02:58:58,316::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000589847564697', 'lastCheck': 1384916336.1002679, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150799751282', 'lastCheck': 1384916333.889853, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00127482414246', 'lastCheck': 1384916334.8867331, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000324010848999', 'lastCheck': 1384916334.2075441, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:58:58,422::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:58:59,012::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0878927 s, 11.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:59:01,027::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:01,614::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0737087 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:59:03,629::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:04,181::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0736066 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15913226::DEBUG::2013-11-20 02:59:05,634::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913226::INFO::2013-11-20 02:59:05,635::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913226::DEBUG::2013-11-20 02:59:05,636::task::495::TaskManager.Task::(_debug) Task 891d3a21-0836-4c93-883c-ec690ae3d460: moving from state init -> state preparing >Thread-15913226::DEBUG::2013-11-20 02:59:05,637::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913226::DEBUG::2013-11-20 02:59:05,637::task::495::TaskManager.Task::(_debug) Task 891d3a21-0836-4c93-883c-ec690ae3d460: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913226::DEBUG::2013-11-20 02:59:05,638::task::495::TaskManager.Task::(_debug) Task 891d3a21-0836-4c93-883c-ec690ae3d460: moving from state preparing -> state finished >Thread-15913226::DEBUG::2013-11-20 02:59:05,638::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913226::DEBUG::2013-11-20 02:59:05,639::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913226::DEBUG::2013-11-20 02:59:05,639::task::495::TaskManager.Task::(_debug) Task 891d3a21-0836-4c93-883c-ec690ae3d460: ref 0 aborting False >Thread-15913226::INFO::2013-11-20 02:59:05,640::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913227::DEBUG::2013-11-20 02:59:05,644::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913227::INFO::2013-11-20 02:59:05,645::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913227::DEBUG::2013-11-20 02:59:05,646::task::495::TaskManager.Task::(_debug) Task 7d4f6125-c199-4646-adc5-8909ddfec23d: moving from state init -> state preparing >Thread-15913227::DEBUG::2013-11-20 02:59:05,647::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`10ec247b-253c-44c9-accd-56083dcc9072`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913227::DEBUG::2013-11-20 02:59:05,648::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913227::DEBUG::2013-11-20 02:59:05,649::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913227::DEBUG::2013-11-20 02:59:05,649::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`10ec247b-253c-44c9-accd-56083dcc9072`::Granted request >Thread-15913227::DEBUG::2013-11-20 02:59:05,650::task::495::TaskManager.Task::(_debug) Task 7d4f6125-c199-4646-adc5-8909ddfec23d: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913227::DEBUG::2013-11-20 02:59:05,651::task::495::TaskManager.Task::(_debug) Task 7d4f6125-c199-4646-adc5-8909ddfec23d: ref 1 aborting False >Thread-15913227::WARNING::2013-11-20 02:59:05,653::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913227::DEBUG::2013-11-20 02:59:05,653::task::495::TaskManager.Task::(_debug) Task 7d4f6125-c199-4646-adc5-8909ddfec23d: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913227::DEBUG::2013-11-20 02:59:05,654::task::495::TaskManager.Task::(_debug) Task 7d4f6125-c199-4646-adc5-8909ddfec23d: moving from state preparing -> state finished >Thread-15913227::DEBUG::2013-11-20 02:59:05,655::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913227::DEBUG::2013-11-20 02:59:05,655::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913227::DEBUG::2013-11-20 02:59:05,656::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913227::DEBUG::2013-11-20 02:59:05,657::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913227::DEBUG::2013-11-20 02:59:05,657::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913227::DEBUG::2013-11-20 02:59:05,658::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913227::DEBUG::2013-11-20 02:59:05,658::task::495::TaskManager.Task::(_debug) Task 7d4f6125-c199-4646-adc5-8909ddfec23d: ref 0 aborting False >Thread-15913227::INFO::2013-11-20 02:59:05,659::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:59:06,196::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:06,734::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.110716 s, 9.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:59:08,749::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:09,196::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0443501 s, 23.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:59:11,211::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:11,735::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0729556 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15913230::INFO::2013-11-20 02:59:13,654::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913230::DEBUG::2013-11-20 02:59:13,655::task::495::TaskManager.Task::(_debug) Task bce888a1-9c6a-484b-a727-f90b80d42be9: moving from state init -> state preparing >Thread-15913230::DEBUG::2013-11-20 02:59:13,656::task::495::TaskManager.Task::(_debug) Task bce888a1-9c6a-484b-a727-f90b80d42be9: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000385046005249', 'lastCheck': 1384916346.4443099, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00134205818176', 'lastCheck': 1384916344.3229661, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00131607055664', 'lastCheck': 1384916345.296495, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00235104560852', 'lastCheck': 1384916344.314662, 'valid': True, 'code': 0}} >Thread-15913230::DEBUG::2013-11-20 02:59:13,657::task::495::TaskManager.Task::(_debug) Task bce888a1-9c6a-484b-a727-f90b80d42be9: moving from state preparing -> state finished >Thread-15913230::DEBUG::2013-11-20 02:59:13,657::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913230::DEBUG::2013-11-20 02:59:13,658::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913230::DEBUG::2013-11-20 02:59:13,659::task::495::TaskManager.Task::(_debug) Task bce888a1-9c6a-484b-a727-f90b80d42be9: ref 0 aborting False >Thread-15913230::INFO::2013-11-20 02:59:13,661::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000385046005249', 'lastCheck': 1384916346.4443099, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00134205818176', 'lastCheck': 1384916344.3229661, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00131607055664', 'lastCheck': 1384916345.296495, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00235104560852', 'lastCheck': 1384916344.314662, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:59:13,750::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:14,287::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0735717 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15913232::DEBUG::2013-11-20 02:59:15,764::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913232::INFO::2013-11-20 02:59:15,766::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913232::DEBUG::2013-11-20 02:59:15,766::task::495::TaskManager.Task::(_debug) Task 5df1fd71-96fb-44b4-a171-5057a550f6a4: moving from state init -> state preparing >Thread-15913232::DEBUG::2013-11-20 02:59:15,767::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913232::DEBUG::2013-11-20 02:59:15,768::task::495::TaskManager.Task::(_debug) Task 5df1fd71-96fb-44b4-a171-5057a550f6a4: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913232::DEBUG::2013-11-20 02:59:15,768::task::495::TaskManager.Task::(_debug) Task 5df1fd71-96fb-44b4-a171-5057a550f6a4: moving from state preparing -> state finished >Thread-15913232::DEBUG::2013-11-20 02:59:15,769::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913232::DEBUG::2013-11-20 02:59:15,769::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913232::DEBUG::2013-11-20 02:59:15,770::task::495::TaskManager.Task::(_debug) Task 5df1fd71-96fb-44b4-a171-5057a550f6a4: ref 0 aborting False >Thread-15913232::INFO::2013-11-20 02:59:15,771::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913233::DEBUG::2013-11-20 02:59:15,775::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913233::INFO::2013-11-20 02:59:15,776::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913233::DEBUG::2013-11-20 02:59:15,777::task::495::TaskManager.Task::(_debug) Task b669f193-017a-403e-bea5-6c32bd107ab1: moving from state init -> state preparing >Thread-15913233::DEBUG::2013-11-20 02:59:15,778::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`691efe9c-713f-49c8-8ff3-0ed75491b8e5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913233::DEBUG::2013-11-20 02:59:15,779::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913233::DEBUG::2013-11-20 02:59:15,780::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913233::DEBUG::2013-11-20 02:59:15,780::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`691efe9c-713f-49c8-8ff3-0ed75491b8e5`::Granted request >Thread-15913233::DEBUG::2013-11-20 02:59:15,781::task::495::TaskManager.Task::(_debug) Task b669f193-017a-403e-bea5-6c32bd107ab1: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913233::DEBUG::2013-11-20 02:59:15,782::task::495::TaskManager.Task::(_debug) Task b669f193-017a-403e-bea5-6c32bd107ab1: ref 1 aborting False >Thread-15913233::WARNING::2013-11-20 02:59:15,784::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913233::DEBUG::2013-11-20 02:59:15,785::task::495::TaskManager.Task::(_debug) Task b669f193-017a-403e-bea5-6c32bd107ab1: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913233::DEBUG::2013-11-20 02:59:15,786::task::495::TaskManager.Task::(_debug) Task b669f193-017a-403e-bea5-6c32bd107ab1: moving from state preparing -> state finished >Thread-15913233::DEBUG::2013-11-20 02:59:15,787::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913233::DEBUG::2013-11-20 02:59:15,787::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913233::DEBUG::2013-11-20 02:59:15,788::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913233::DEBUG::2013-11-20 02:59:15,789::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913233::DEBUG::2013-11-20 02:59:15,789::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913233::DEBUG::2013-11-20 02:59:15,790::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913233::DEBUG::2013-11-20 02:59:15,791::task::495::TaskManager.Task::(_debug) Task b669f193-017a-403e-bea5-6c32bd107ab1: ref 0 aborting False >Thread-15913233::INFO::2013-11-20 02:59:15,791::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:59:16,302::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 02:59:16,557::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 02:59:16,559::task::495::TaskManager.Task::(_debug) Task f2ec60f1-f520-47c5-98c2-3522d756cd88: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 02:59:16,562::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`6d1b3bf5-2d10-4042-bcfb-ac815ae8b21a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 02:59:16,562::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 02:59:16,563::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 02:59:16,564::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`6d1b3bf5-2d10-4042-bcfb-ac815ae8b21a`::Granted request >Thread-15912392::DEBUG::2013-11-20 02:59:16,565::task::495::TaskManager.Task::(_debug) Task f2ec60f1-f520-47c5-98c2-3522d756cd88: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 02:59:16,566::task::495::TaskManager.Task::(_debug) Task f2ec60f1-f520-47c5-98c2-3522d756cd88: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 02:59:16,566::task::495::TaskManager.Task::(_debug) Task f2ec60f1-f520-47c5-98c2-3522d756cd88: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 02:59:16,567::task::495::TaskManager.Task::(_debug) Task f2ec60f1-f520-47c5-98c2-3522d756cd88: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 02:59:16,567::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 02:59:16,568::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 02:59:16,569::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 02:59:16,569::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 02:59:16,570::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 02:59:16,570::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 02:59:16,571::task::495::TaskManager.Task::(_debug) Task f2ec60f1-f520-47c5-98c2-3522d756cd88: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 02:59:16,571::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 02:59:16,867::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0738402 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:59:18,881::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:19,428::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0815262 s, 12.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:59:21,443::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:21,976::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0821618 s, 12.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:59:23,991::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:24,538::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0969849 s, 10.6 MB/s\n'; <rc> = 0 >Thread-15913238::DEBUG::2013-11-20 02:59:25,971::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913238::INFO::2013-11-20 02:59:25,973::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913238::DEBUG::2013-11-20 02:59:25,973::task::495::TaskManager.Task::(_debug) Task fb7bdce7-45f9-4b30-a473-a5fdfa8bbc6e: moving from state init -> state preparing >Thread-15913238::DEBUG::2013-11-20 02:59:25,974::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913238::DEBUG::2013-11-20 02:59:25,975::task::495::TaskManager.Task::(_debug) Task fb7bdce7-45f9-4b30-a473-a5fdfa8bbc6e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913238::DEBUG::2013-11-20 02:59:25,975::task::495::TaskManager.Task::(_debug) Task fb7bdce7-45f9-4b30-a473-a5fdfa8bbc6e: moving from state preparing -> state finished >Thread-15913238::DEBUG::2013-11-20 02:59:25,976::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913238::DEBUG::2013-11-20 02:59:25,977::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913238::DEBUG::2013-11-20 02:59:25,977::task::495::TaskManager.Task::(_debug) Task fb7bdce7-45f9-4b30-a473-a5fdfa8bbc6e: ref 0 aborting False >Thread-15913238::INFO::2013-11-20 02:59:25,978::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913239::DEBUG::2013-11-20 02:59:25,983::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913239::INFO::2013-11-20 02:59:25,984::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913239::DEBUG::2013-11-20 02:59:25,984::task::495::TaskManager.Task::(_debug) Task bec4b0e8-f4a1-4e2d-9827-ea1c8239eb13: moving from state init -> state preparing >Thread-15913239::DEBUG::2013-11-20 02:59:25,986::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`259379b0-fcbd-4b39-a60f-0f4fdfa39414`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913239::DEBUG::2013-11-20 02:59:25,986::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913239::DEBUG::2013-11-20 02:59:25,987::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913239::DEBUG::2013-11-20 02:59:25,988::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`259379b0-fcbd-4b39-a60f-0f4fdfa39414`::Granted request >Thread-15913239::DEBUG::2013-11-20 02:59:25,989::task::495::TaskManager.Task::(_debug) Task bec4b0e8-f4a1-4e2d-9827-ea1c8239eb13: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913239::DEBUG::2013-11-20 02:59:25,989::task::495::TaskManager.Task::(_debug) Task bec4b0e8-f4a1-4e2d-9827-ea1c8239eb13: ref 1 aborting False >Thread-15913239::WARNING::2013-11-20 02:59:25,991::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913239::DEBUG::2013-11-20 02:59:25,992::task::495::TaskManager.Task::(_debug) Task bec4b0e8-f4a1-4e2d-9827-ea1c8239eb13: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913239::DEBUG::2013-11-20 02:59:25,993::task::495::TaskManager.Task::(_debug) Task bec4b0e8-f4a1-4e2d-9827-ea1c8239eb13: moving from state preparing -> state finished >Thread-15913239::DEBUG::2013-11-20 02:59:25,993::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913239::DEBUG::2013-11-20 02:59:25,994::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913239::DEBUG::2013-11-20 02:59:25,995::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913239::DEBUG::2013-11-20 02:59:25,995::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913239::DEBUG::2013-11-20 02:59:25,996::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913239::DEBUG::2013-11-20 02:59:25,996::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913239::DEBUG::2013-11-20 02:59:25,997::task::495::TaskManager.Task::(_debug) Task bec4b0e8-f4a1-4e2d-9827-ea1c8239eb13: ref 0 aborting False >Thread-15913239::INFO::2013-11-20 02:59:25,998::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:59:26,553::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:27,103::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0706231 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15913240::INFO::2013-11-20 02:59:28,899::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913240::DEBUG::2013-11-20 02:59:28,900::task::495::TaskManager.Task::(_debug) Task 20906ae3-e1cb-4569-b25b-7fd5432feb7b: moving from state init -> state preparing >Thread-15913240::DEBUG::2013-11-20 02:59:28,901::task::495::TaskManager.Task::(_debug) Task 20906ae3-e1cb-4569-b25b-7fd5432feb7b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000354051589966', 'lastCheck': 1384916366.8071809, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00211191177368', 'lastCheck': 1384916364.6658731, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00181293487549', 'lastCheck': 1384916365.6644361, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.002366065979', 'lastCheck': 1384916364.6600189, 'valid': True, 'code': 0}} >Thread-15913240::DEBUG::2013-11-20 02:59:28,901::task::495::TaskManager.Task::(_debug) Task 20906ae3-e1cb-4569-b25b-7fd5432feb7b: moving from state preparing -> state finished >Thread-15913240::DEBUG::2013-11-20 02:59:28,902::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913240::DEBUG::2013-11-20 02:59:28,903::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913240::DEBUG::2013-11-20 02:59:28,903::task::495::TaskManager.Task::(_debug) Task 20906ae3-e1cb-4569-b25b-7fd5432feb7b: ref 0 aborting False >Thread-15913240::INFO::2013-11-20 02:59:28,904::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000354051589966', 'lastCheck': 1384916366.8071809, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00211191177368', 'lastCheck': 1384916364.6658731, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00181293487549', 'lastCheck': 1384916365.6644361, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.002366065979', 'lastCheck': 1384916364.6600189, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:59:29,116::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:29,583::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0482029 s, 21.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:59:31,598::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:32,075::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0459884 s, 22.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:59:34,084::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:34,639::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0848739 s, 12.1 MB/s\n'; <rc> = 0 >Thread-15913244::DEBUG::2013-11-20 02:59:36,179::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913244::INFO::2013-11-20 02:59:36,180::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913244::DEBUG::2013-11-20 02:59:36,181::task::495::TaskManager.Task::(_debug) Task 2061d586-df8c-45bb-8652-766491829589: moving from state init -> state preparing >Thread-15913244::DEBUG::2013-11-20 02:59:36,182::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913244::DEBUG::2013-11-20 02:59:36,182::task::495::TaskManager.Task::(_debug) Task 2061d586-df8c-45bb-8652-766491829589: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913244::DEBUG::2013-11-20 02:59:36,183::task::495::TaskManager.Task::(_debug) Task 2061d586-df8c-45bb-8652-766491829589: moving from state preparing -> state finished >Thread-15913244::DEBUG::2013-11-20 02:59:36,183::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913244::DEBUG::2013-11-20 02:59:36,184::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913244::DEBUG::2013-11-20 02:59:36,185::task::495::TaskManager.Task::(_debug) Task 2061d586-df8c-45bb-8652-766491829589: ref 0 aborting False >Thread-15913244::INFO::2013-11-20 02:59:36,185::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913245::DEBUG::2013-11-20 02:59:36,190::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913245::INFO::2013-11-20 02:59:36,191::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913245::DEBUG::2013-11-20 02:59:36,191::task::495::TaskManager.Task::(_debug) Task 8bf4bcb9-a8b1-42ae-8f53-071e31c6fdd1: moving from state init -> state preparing >Thread-15913245::DEBUG::2013-11-20 02:59:36,192::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`017a95bc-dffe-40f8-9329-fd38296e7850`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913245::DEBUG::2013-11-20 02:59:36,193::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913245::DEBUG::2013-11-20 02:59:36,194::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913245::DEBUG::2013-11-20 02:59:36,194::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`017a95bc-dffe-40f8-9329-fd38296e7850`::Granted request >Thread-15913245::DEBUG::2013-11-20 02:59:36,195::task::495::TaskManager.Task::(_debug) Task 8bf4bcb9-a8b1-42ae-8f53-071e31c6fdd1: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913245::DEBUG::2013-11-20 02:59:36,196::task::495::TaskManager.Task::(_debug) Task 8bf4bcb9-a8b1-42ae-8f53-071e31c6fdd1: ref 1 aborting False >Thread-15913245::WARNING::2013-11-20 02:59:36,198::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913245::DEBUG::2013-11-20 02:59:36,199::task::495::TaskManager.Task::(_debug) Task 8bf4bcb9-a8b1-42ae-8f53-071e31c6fdd1: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913245::DEBUG::2013-11-20 02:59:36,200::task::495::TaskManager.Task::(_debug) Task 8bf4bcb9-a8b1-42ae-8f53-071e31c6fdd1: moving from state preparing -> state finished >Thread-15913245::DEBUG::2013-11-20 02:59:36,200::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913245::DEBUG::2013-11-20 02:59:36,201::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913245::DEBUG::2013-11-20 02:59:36,201::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913245::DEBUG::2013-11-20 02:59:36,202::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913245::DEBUG::2013-11-20 02:59:36,202::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913245::DEBUG::2013-11-20 02:59:36,203::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913245::DEBUG::2013-11-20 02:59:36,204::task::495::TaskManager.Task::(_debug) Task 8bf4bcb9-a8b1-42ae-8f53-071e31c6fdd1: ref 0 aborting False >Thread-15913245::INFO::2013-11-20 02:59:36,204::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:59:36,654::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:37,210::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0751123 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:59:39,225::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:39,763::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0741205 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:59:41,778::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:42,328::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0780058 s, 13.1 MB/s\n'; <rc> = 0 >Thread-15913248::INFO::2013-11-20 02:59:44,185::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913248::DEBUG::2013-11-20 02:59:44,186::task::495::TaskManager.Task::(_debug) Task 63999f2f-70c3-476a-bf26-609273dd3e8f: moving from state init -> state preparing >Thread-15913248::DEBUG::2013-11-20 02:59:44,187::task::495::TaskManager.Task::(_debug) Task 63999f2f-70c3-476a-bf26-609273dd3e8f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000416994094849', 'lastCheck': 1384916376.906837, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00488805770874', 'lastCheck': 1384916374.8139031, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0013689994812', 'lastCheck': 1384916375.8095939, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00142002105713', 'lastCheck': 1384916374.8098741, 'valid': True, 'code': 0}} >Thread-15913248::DEBUG::2013-11-20 02:59:44,188::task::495::TaskManager.Task::(_debug) Task 63999f2f-70c3-476a-bf26-609273dd3e8f: moving from state preparing -> state finished >Thread-15913248::DEBUG::2013-11-20 02:59:44,189::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913248::DEBUG::2013-11-20 02:59:44,189::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913248::DEBUG::2013-11-20 02:59:44,190::task::495::TaskManager.Task::(_debug) Task 63999f2f-70c3-476a-bf26-609273dd3e8f: ref 0 aborting False >Thread-15913248::INFO::2013-11-20 02:59:44,190::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000416994094849', 'lastCheck': 1384916376.906837, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00488805770874', 'lastCheck': 1384916374.8139031, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0013689994812', 'lastCheck': 1384916375.8095939, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00142002105713', 'lastCheck': 1384916374.8098741, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:59:44,344::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:44,874::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0731782 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15913250::DEBUG::2013-11-20 02:59:46,393::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913250::INFO::2013-11-20 02:59:46,395::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913250::DEBUG::2013-11-20 02:59:46,395::task::495::TaskManager.Task::(_debug) Task fbb40641-8783-4aef-8f40-01d20149916d: moving from state init -> state preparing >Thread-15913250::DEBUG::2013-11-20 02:59:46,396::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913250::DEBUG::2013-11-20 02:59:46,397::task::495::TaskManager.Task::(_debug) Task fbb40641-8783-4aef-8f40-01d20149916d: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913250::DEBUG::2013-11-20 02:59:46,397::task::495::TaskManager.Task::(_debug) Task fbb40641-8783-4aef-8f40-01d20149916d: moving from state preparing -> state finished >Thread-15913250::DEBUG::2013-11-20 02:59:46,398::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913250::DEBUG::2013-11-20 02:59:46,399::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913250::DEBUG::2013-11-20 02:59:46,399::task::495::TaskManager.Task::(_debug) Task fbb40641-8783-4aef-8f40-01d20149916d: ref 0 aborting False >Thread-15913250::INFO::2013-11-20 02:59:46,400::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913251::DEBUG::2013-11-20 02:59:46,404::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913251::INFO::2013-11-20 02:59:46,405::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913251::DEBUG::2013-11-20 02:59:46,406::task::495::TaskManager.Task::(_debug) Task 1694a27a-b0df-430c-b512-261898851425: moving from state init -> state preparing >Thread-15913251::DEBUG::2013-11-20 02:59:46,407::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bdfda5a9-6f59-4b96-aa18-e0a415e8174d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913251::DEBUG::2013-11-20 02:59:46,408::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913251::DEBUG::2013-11-20 02:59:46,409::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913251::DEBUG::2013-11-20 02:59:46,409::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bdfda5a9-6f59-4b96-aa18-e0a415e8174d`::Granted request >Thread-15913251::DEBUG::2013-11-20 02:59:46,410::task::495::TaskManager.Task::(_debug) Task 1694a27a-b0df-430c-b512-261898851425: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913251::DEBUG::2013-11-20 02:59:46,411::task::495::TaskManager.Task::(_debug) Task 1694a27a-b0df-430c-b512-261898851425: ref 1 aborting False >Thread-15913251::WARNING::2013-11-20 02:59:46,413::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913251::DEBUG::2013-11-20 02:59:46,414::task::495::TaskManager.Task::(_debug) Task 1694a27a-b0df-430c-b512-261898851425: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913251::DEBUG::2013-11-20 02:59:46,415::task::495::TaskManager.Task::(_debug) Task 1694a27a-b0df-430c-b512-261898851425: moving from state preparing -> state finished >Thread-15913251::DEBUG::2013-11-20 02:59:46,415::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913251::DEBUG::2013-11-20 02:59:46,416::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913251::DEBUG::2013-11-20 02:59:46,416::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913251::DEBUG::2013-11-20 02:59:46,417::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913251::DEBUG::2013-11-20 02:59:46,418::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913251::DEBUG::2013-11-20 02:59:46,418::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913251::DEBUG::2013-11-20 02:59:46,419::task::495::TaskManager.Task::(_debug) Task 1694a27a-b0df-430c-b512-261898851425: ref 0 aborting False >Thread-15913251::INFO::2013-11-20 02:59:46,419::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:59:46,889::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:47,434::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0741585 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:59:49,449::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:49,943::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0827072 s, 12.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:59:51,959::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:52,492::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0762666 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 02:59:54,508::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:55,023::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0787463 s, 13.0 MB/s\n'; <rc> = 0 >Thread-15913256::DEBUG::2013-11-20 02:59:56,667::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913256::INFO::2013-11-20 02:59:56,668::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913256::DEBUG::2013-11-20 02:59:56,669::task::495::TaskManager.Task::(_debug) Task 6da7c30e-bf0a-490b-8e5d-cecd4aedef25: moving from state init -> state preparing >Thread-15913256::DEBUG::2013-11-20 02:59:56,670::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913256::DEBUG::2013-11-20 02:59:56,670::task::495::TaskManager.Task::(_debug) Task 6da7c30e-bf0a-490b-8e5d-cecd4aedef25: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913256::DEBUG::2013-11-20 02:59:56,671::task::495::TaskManager.Task::(_debug) Task 6da7c30e-bf0a-490b-8e5d-cecd4aedef25: moving from state preparing -> state finished >Thread-15913256::DEBUG::2013-11-20 02:59:56,671::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913256::DEBUG::2013-11-20 02:59:56,672::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913256::DEBUG::2013-11-20 02:59:56,673::task::495::TaskManager.Task::(_debug) Task 6da7c30e-bf0a-490b-8e5d-cecd4aedef25: ref 0 aborting False >Thread-15913256::INFO::2013-11-20 02:59:56,673::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913257::DEBUG::2013-11-20 02:59:56,678::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913257::INFO::2013-11-20 02:59:56,679::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913257::DEBUG::2013-11-20 02:59:56,680::task::495::TaskManager.Task::(_debug) Task 3b700c0e-8bf6-4179-9a0c-e1ae656a8036: moving from state init -> state preparing >Thread-15913257::DEBUG::2013-11-20 02:59:56,681::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d8a0cf47-d54c-44d6-b1ad-255125c00ecb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913257::DEBUG::2013-11-20 02:59:56,682::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913257::DEBUG::2013-11-20 02:59:56,682::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913257::DEBUG::2013-11-20 02:59:56,683::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d8a0cf47-d54c-44d6-b1ad-255125c00ecb`::Granted request >Thread-15913257::DEBUG::2013-11-20 02:59:56,684::task::495::TaskManager.Task::(_debug) Task 3b700c0e-8bf6-4179-9a0c-e1ae656a8036: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913257::DEBUG::2013-11-20 02:59:56,685::task::495::TaskManager.Task::(_debug) Task 3b700c0e-8bf6-4179-9a0c-e1ae656a8036: ref 1 aborting False >Thread-15913257::WARNING::2013-11-20 02:59:56,686::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913257::DEBUG::2013-11-20 02:59:56,687::task::495::TaskManager.Task::(_debug) Task 3b700c0e-8bf6-4179-9a0c-e1ae656a8036: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913257::DEBUG::2013-11-20 02:59:56,688::task::495::TaskManager.Task::(_debug) Task 3b700c0e-8bf6-4179-9a0c-e1ae656a8036: moving from state preparing -> state finished >Thread-15913257::DEBUG::2013-11-20 02:59:56,689::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913257::DEBUG::2013-11-20 02:59:56,689::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913257::DEBUG::2013-11-20 02:59:56,690::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913257::DEBUG::2013-11-20 02:59:56,691::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913257::DEBUG::2013-11-20 02:59:56,691::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913257::DEBUG::2013-11-20 02:59:56,692::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913257::DEBUG::2013-11-20 02:59:56,692::task::495::TaskManager.Task::(_debug) Task 3b700c0e-8bf6-4179-9a0c-e1ae656a8036: ref 0 aborting False >Thread-15913257::INFO::2013-11-20 02:59:56,693::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 02:59:57,037::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 02:59:57,595::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0873732 s, 11.7 MB/s\n'; <rc> = 0 >Thread-15913258::INFO::2013-11-20 02:59:59,471::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913258::DEBUG::2013-11-20 02:59:59,472::task::495::TaskManager.Task::(_debug) Task c92ce316-2954-4c00-86af-64eacdcd0137: moving from state init -> state preparing >Thread-15913258::DEBUG::2013-11-20 02:59:59,473::task::495::TaskManager.Task::(_debug) Task c92ce316-2954-4c00-86af-64eacdcd0137: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000452995300293', 'lastCheck': 1384916397.2781799, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148510932922', 'lastCheck': 1384916395.1808209, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00183415412903', 'lastCheck': 1384916396.1822181, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000585079193115', 'lastCheck': 1384916395.1753011, 'valid': True, 'code': 0}} >Thread-15913258::DEBUG::2013-11-20 02:59:59,474::task::495::TaskManager.Task::(_debug) Task c92ce316-2954-4c00-86af-64eacdcd0137: moving from state preparing -> state finished >Thread-15913258::DEBUG::2013-11-20 02:59:59,474::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913258::DEBUG::2013-11-20 02:59:59,475::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913258::DEBUG::2013-11-20 02:59:59,475::task::495::TaskManager.Task::(_debug) Task c92ce316-2954-4c00-86af-64eacdcd0137: ref 0 aborting False >Thread-15913258::INFO::2013-11-20 02:59:59,476::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000452995300293', 'lastCheck': 1384916397.2781799, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148510932922', 'lastCheck': 1384916395.1808209, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00183415412903', 'lastCheck': 1384916396.1822181, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000585079193115', 'lastCheck': 1384916395.1753011, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 02:59:59,608::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:00,050::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0480607 s, 21.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:00:02,060::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:02,614::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0845828 s, 12.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:00:04,629::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:05,191::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0739048 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15913262::DEBUG::2013-11-20 03:00:06,841::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913262::INFO::2013-11-20 03:00:06,843::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913262::DEBUG::2013-11-20 03:00:06,860::task::495::TaskManager.Task::(_debug) Task 6336c8bd-b413-45b0-84d1-dc6ae961e67d: moving from state init -> state preparing >Thread-15913262::DEBUG::2013-11-20 03:00:06,861::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913262::DEBUG::2013-11-20 03:00:06,861::task::495::TaskManager.Task::(_debug) Task 6336c8bd-b413-45b0-84d1-dc6ae961e67d: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913262::DEBUG::2013-11-20 03:00:06,862::task::495::TaskManager.Task::(_debug) Task 6336c8bd-b413-45b0-84d1-dc6ae961e67d: moving from state preparing -> state finished >Thread-15913262::DEBUG::2013-11-20 03:00:06,862::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913262::DEBUG::2013-11-20 03:00:06,863::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913262::DEBUG::2013-11-20 03:00:06,863::task::495::TaskManager.Task::(_debug) Task 6336c8bd-b413-45b0-84d1-dc6ae961e67d: ref 0 aborting False >Thread-15913262::INFO::2013-11-20 03:00:06,864::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913263::DEBUG::2013-11-20 03:00:06,869::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913263::INFO::2013-11-20 03:00:06,870::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913263::DEBUG::2013-11-20 03:00:06,871::task::495::TaskManager.Task::(_debug) Task cc1fc145-b192-43d5-b569-eeac350662b4: moving from state init -> state preparing >Thread-15913263::DEBUG::2013-11-20 03:00:06,872::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`68379c8a-cb7a-4799-8986-bd1ffae56290`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913263::DEBUG::2013-11-20 03:00:06,873::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913263::DEBUG::2013-11-20 03:00:06,874::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913263::DEBUG::2013-11-20 03:00:06,874::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`68379c8a-cb7a-4799-8986-bd1ffae56290`::Granted request >Thread-15913263::DEBUG::2013-11-20 03:00:06,875::task::495::TaskManager.Task::(_debug) Task cc1fc145-b192-43d5-b569-eeac350662b4: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913263::DEBUG::2013-11-20 03:00:06,876::task::495::TaskManager.Task::(_debug) Task cc1fc145-b192-43d5-b569-eeac350662b4: ref 1 aborting False >Thread-15913263::WARNING::2013-11-20 03:00:06,880::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913263::DEBUG::2013-11-20 03:00:06,881::task::495::TaskManager.Task::(_debug) Task cc1fc145-b192-43d5-b569-eeac350662b4: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913263::DEBUG::2013-11-20 03:00:06,883::task::495::TaskManager.Task::(_debug) Task cc1fc145-b192-43d5-b569-eeac350662b4: moving from state preparing -> state finished >Thread-15913263::DEBUG::2013-11-20 03:00:06,884::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913263::DEBUG::2013-11-20 03:00:06,885::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913263::DEBUG::2013-11-20 03:00:06,887::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913263::DEBUG::2013-11-20 03:00:06,888::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913263::DEBUG::2013-11-20 03:00:06,889::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913263::DEBUG::2013-11-20 03:00:06,890::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913263::DEBUG::2013-11-20 03:00:06,892::task::495::TaskManager.Task::(_debug) Task cc1fc145-b192-43d5-b569-eeac350662b4: ref 0 aborting False >Thread-15913263::INFO::2013-11-20 03:00:06,893::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:00:07,207::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:07,727::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.069524 s, 14.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:00:09,740::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:10,263::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0768087 s, 13.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:00:12,279::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:12,815::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0739975 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15913266::INFO::2013-11-20 03:00:14,735::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913266::DEBUG::2013-11-20 03:00:14,736::task::495::TaskManager.Task::(_debug) Task 7cd4672f-2b05-45d2-9662-af26fb635778: moving from state init -> state preparing >Thread-15913266::DEBUG::2013-11-20 03:00:14,736::task::495::TaskManager.Task::(_debug) Task 7cd4672f-2b05-45d2-9662-af26fb635778: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000590085983276', 'lastCheck': 1384916407.4461031, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0014660358429', 'lastCheck': 1384916405.283742, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00134992599487', 'lastCheck': 1384916406.28438, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000519037246704', 'lastCheck': 1384916405.2775559, 'valid': True, 'code': 0}} >Thread-15913266::DEBUG::2013-11-20 03:00:14,737::task::495::TaskManager.Task::(_debug) Task 7cd4672f-2b05-45d2-9662-af26fb635778: moving from state preparing -> state finished >Thread-15913266::DEBUG::2013-11-20 03:00:14,737::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913266::DEBUG::2013-11-20 03:00:14,738::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913266::DEBUG::2013-11-20 03:00:14,738::task::495::TaskManager.Task::(_debug) Task 7cd4672f-2b05-45d2-9662-af26fb635778: ref 0 aborting False >Thread-15913266::INFO::2013-11-20 03:00:14,739::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000590085983276', 'lastCheck': 1384916407.4461031, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0014660358429', 'lastCheck': 1384916405.283742, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00134992599487', 'lastCheck': 1384916406.28438, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000519037246704', 'lastCheck': 1384916405.2775559, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:00:14,831::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:15,363::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0719022 s, 14.2 MB/s\n'; <rc> = 0 >Thread-15913268::DEBUG::2013-11-20 03:00:17,034::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913268::INFO::2013-11-20 03:00:17,035::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913268::DEBUG::2013-11-20 03:00:17,035::task::495::TaskManager.Task::(_debug) Task 72d38d60-f6d4-426b-98af-d6b84800fa60: moving from state init -> state preparing >Thread-15913268::DEBUG::2013-11-20 03:00:17,036::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913268::DEBUG::2013-11-20 03:00:17,036::task::495::TaskManager.Task::(_debug) Task 72d38d60-f6d4-426b-98af-d6b84800fa60: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913268::DEBUG::2013-11-20 03:00:17,036::task::495::TaskManager.Task::(_debug) Task 72d38d60-f6d4-426b-98af-d6b84800fa60: moving from state preparing -> state finished >Thread-15913268::DEBUG::2013-11-20 03:00:17,037::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913268::DEBUG::2013-11-20 03:00:17,037::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913268::DEBUG::2013-11-20 03:00:17,037::task::495::TaskManager.Task::(_debug) Task 72d38d60-f6d4-426b-98af-d6b84800fa60: ref 0 aborting False >Thread-15913268::INFO::2013-11-20 03:00:17,038::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913269::DEBUG::2013-11-20 03:00:17,041::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913269::INFO::2013-11-20 03:00:17,042::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913269::DEBUG::2013-11-20 03:00:17,042::task::495::TaskManager.Task::(_debug) Task 46df6ce6-7bf1-4179-b952-bd1b8c9b5b1e: moving from state init -> state preparing >Thread-15913269::DEBUG::2013-11-20 03:00:17,043::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`27a9d8e4-0fc4-4b41-bdf6-56d82375fbf9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913269::DEBUG::2013-11-20 03:00:17,043::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913269::DEBUG::2013-11-20 03:00:17,044::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913269::DEBUG::2013-11-20 03:00:17,044::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`27a9d8e4-0fc4-4b41-bdf6-56d82375fbf9`::Granted request >Thread-15913269::DEBUG::2013-11-20 03:00:17,045::task::495::TaskManager.Task::(_debug) Task 46df6ce6-7bf1-4179-b952-bd1b8c9b5b1e: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913269::DEBUG::2013-11-20 03:00:17,045::task::495::TaskManager.Task::(_debug) Task 46df6ce6-7bf1-4179-b952-bd1b8c9b5b1e: ref 1 aborting False >Thread-15913269::WARNING::2013-11-20 03:00:17,046::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913269::DEBUG::2013-11-20 03:00:17,047::task::495::TaskManager.Task::(_debug) Task 46df6ce6-7bf1-4179-b952-bd1b8c9b5b1e: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913269::DEBUG::2013-11-20 03:00:17,047::task::495::TaskManager.Task::(_debug) Task 46df6ce6-7bf1-4179-b952-bd1b8c9b5b1e: moving from state preparing -> state finished >Thread-15913269::DEBUG::2013-11-20 03:00:17,048::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913269::DEBUG::2013-11-20 03:00:17,048::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913269::DEBUG::2013-11-20 03:00:17,048::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913269::DEBUG::2013-11-20 03:00:17,049::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913269::DEBUG::2013-11-20 03:00:17,049::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913269::DEBUG::2013-11-20 03:00:17,049::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913269::DEBUG::2013-11-20 03:00:17,050::task::495::TaskManager.Task::(_debug) Task 46df6ce6-7bf1-4179-b952-bd1b8c9b5b1e: ref 0 aborting False >Thread-15913269::INFO::2013-11-20 03:00:17,050::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:00:17,378::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 03:00:17,619::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:00:17,637::task::495::TaskManager.Task::(_debug) Task 9fd40913-556a-40f6-85ff-eae4d164c491: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:00:17,640::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d8ba3c6b-8050-40b4-864d-6997fd53a8be`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:00:17,640::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:00:17,641::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:00:17,642::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d8ba3c6b-8050-40b4-864d-6997fd53a8be`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:00:17,643::task::495::TaskManager.Task::(_debug) Task 9fd40913-556a-40f6-85ff-eae4d164c491: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:00:17,644::task::495::TaskManager.Task::(_debug) Task 9fd40913-556a-40f6-85ff-eae4d164c491: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:00:17,644::task::495::TaskManager.Task::(_debug) Task 9fd40913-556a-40f6-85ff-eae4d164c491: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:00:17,645::task::495::TaskManager.Task::(_debug) Task 9fd40913-556a-40f6-85ff-eae4d164c491: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:00:17,645::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:00:17,646::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:00:17,647::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:00:17,647::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:00:17,648::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:00:17,648::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:00:17,649::task::495::TaskManager.Task::(_debug) Task 9fd40913-556a-40f6-85ff-eae4d164c491: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:00:17,649::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:00:17,958::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0757912 s, 13.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:00:19,974::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:20,513::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0767598 s, 13.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:00:22,529::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:23,044::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0717671 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:00:25,058::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:25,593::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0477387 s, 21.5 MB/s\n'; <rc> = 0 >Thread-15913274::DEBUG::2013-11-20 03:00:27,243::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913274::INFO::2013-11-20 03:00:27,244::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913274::DEBUG::2013-11-20 03:00:27,245::task::495::TaskManager.Task::(_debug) Task 866bf968-95bf-468f-8c2f-8661faec8306: moving from state init -> state preparing >Thread-15913274::DEBUG::2013-11-20 03:00:27,246::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913274::DEBUG::2013-11-20 03:00:27,246::task::495::TaskManager.Task::(_debug) Task 866bf968-95bf-468f-8c2f-8661faec8306: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913274::DEBUG::2013-11-20 03:00:27,247::task::495::TaskManager.Task::(_debug) Task 866bf968-95bf-468f-8c2f-8661faec8306: moving from state preparing -> state finished >Thread-15913274::DEBUG::2013-11-20 03:00:27,248::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913274::DEBUG::2013-11-20 03:00:27,248::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913274::DEBUG::2013-11-20 03:00:27,249::task::495::TaskManager.Task::(_debug) Task 866bf968-95bf-468f-8c2f-8661faec8306: ref 0 aborting False >Thread-15913274::INFO::2013-11-20 03:00:27,249::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913275::DEBUG::2013-11-20 03:00:27,254::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913275::INFO::2013-11-20 03:00:27,255::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913275::DEBUG::2013-11-20 03:00:27,255::task::495::TaskManager.Task::(_debug) Task 87125b1b-4468-4e16-8283-8eb40014c195: moving from state init -> state preparing >Thread-15913275::DEBUG::2013-11-20 03:00:27,263::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`4c94058b-0d6f-4120-8118-2c20463d790e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913275::DEBUG::2013-11-20 03:00:27,263::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913275::DEBUG::2013-11-20 03:00:27,264::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913275::DEBUG::2013-11-20 03:00:27,265::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`4c94058b-0d6f-4120-8118-2c20463d790e`::Granted request >Thread-15913275::DEBUG::2013-11-20 03:00:27,266::task::495::TaskManager.Task::(_debug) Task 87125b1b-4468-4e16-8283-8eb40014c195: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913275::DEBUG::2013-11-20 03:00:27,266::task::495::TaskManager.Task::(_debug) Task 87125b1b-4468-4e16-8283-8eb40014c195: ref 1 aborting False >Thread-15913275::WARNING::2013-11-20 03:00:27,268::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913275::DEBUG::2013-11-20 03:00:27,269::task::495::TaskManager.Task::(_debug) Task 87125b1b-4468-4e16-8283-8eb40014c195: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913275::DEBUG::2013-11-20 03:00:27,270::task::495::TaskManager.Task::(_debug) Task 87125b1b-4468-4e16-8283-8eb40014c195: moving from state preparing -> state finished >Thread-15913275::DEBUG::2013-11-20 03:00:27,270::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913275::DEBUG::2013-11-20 03:00:27,271::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913275::DEBUG::2013-11-20 03:00:27,272::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913275::DEBUG::2013-11-20 03:00:27,272::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913275::DEBUG::2013-11-20 03:00:27,273::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913275::DEBUG::2013-11-20 03:00:27,273::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913275::DEBUG::2013-11-20 03:00:27,274::task::495::TaskManager.Task::(_debug) Task 87125b1b-4468-4e16-8283-8eb40014c195: ref 0 aborting False >Thread-15913275::INFO::2013-11-20 03:00:27,275::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:00:27,613::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:28,146::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0749663 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15913276::INFO::2013-11-20 03:00:30,019::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913276::DEBUG::2013-11-20 03:00:30,020::task::495::TaskManager.Task::(_debug) Task c4e94885-fb2b-4482-8922-4e676d7ed6e0: moving from state init -> state preparing >Thread-15913276::DEBUG::2013-11-20 03:00:30,021::task::495::TaskManager.Task::(_debug) Task c4e94885-fb2b-4482-8922-4e676d7ed6e0: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000576972961426', 'lastCheck': 1384916427.8101799, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0118510723114', 'lastCheck': 1384916425.7849591, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0216488838196', 'lastCheck': 1384916426.796051, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000890970230103', 'lastCheck': 1384916425.7697649, 'valid': True, 'code': 0}} >Thread-15913276::DEBUG::2013-11-20 03:00:30,022::task::495::TaskManager.Task::(_debug) Task c4e94885-fb2b-4482-8922-4e676d7ed6e0: moving from state preparing -> state finished >Thread-15913276::DEBUG::2013-11-20 03:00:30,022::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913276::DEBUG::2013-11-20 03:00:30,023::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913276::DEBUG::2013-11-20 03:00:30,023::task::495::TaskManager.Task::(_debug) Task c4e94885-fb2b-4482-8922-4e676d7ed6e0: ref 0 aborting False >Thread-15913276::INFO::2013-11-20 03:00:30,024::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000576972961426', 'lastCheck': 1384916427.8101799, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0118510723114', 'lastCheck': 1384916425.7849591, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0216488838196', 'lastCheck': 1384916426.796051, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000890970230103', 'lastCheck': 1384916425.7697649, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:00:30,161::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:30,637::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0797865 s, 12.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:00:32,651::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:33,198::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0713212 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:00:35,214::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:35,738::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0820803 s, 12.5 MB/s\n'; <rc> = 0 >Thread-15913280::DEBUG::2013-11-20 03:00:37,443::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913280::INFO::2013-11-20 03:00:37,447::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913280::DEBUG::2013-11-20 03:00:37,450::task::495::TaskManager.Task::(_debug) Task 63a06972-1d59-4224-ad14-ddb04d68165b: moving from state init -> state preparing >Thread-15913280::DEBUG::2013-11-20 03:00:37,453::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913280::DEBUG::2013-11-20 03:00:37,455::task::495::TaskManager.Task::(_debug) Task 63a06972-1d59-4224-ad14-ddb04d68165b: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913280::DEBUG::2013-11-20 03:00:37,457::task::495::TaskManager.Task::(_debug) Task 63a06972-1d59-4224-ad14-ddb04d68165b: moving from state preparing -> state finished >Thread-15913280::DEBUG::2013-11-20 03:00:37,458::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913280::DEBUG::2013-11-20 03:00:37,459::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913280::DEBUG::2013-11-20 03:00:37,459::task::495::TaskManager.Task::(_debug) Task 63a06972-1d59-4224-ad14-ddb04d68165b: ref 0 aborting False >Thread-15913280::INFO::2013-11-20 03:00:37,460::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913281::DEBUG::2013-11-20 03:00:37,464::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913281::INFO::2013-11-20 03:00:37,465::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913281::DEBUG::2013-11-20 03:00:37,466::task::495::TaskManager.Task::(_debug) Task 4b6eb0e7-ad5f-41ea-af15-d2bc8e7a528d: moving from state init -> state preparing >Thread-15913281::DEBUG::2013-11-20 03:00:37,467::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`471f2c6c-6da5-4239-b68c-14ec07173be8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913281::DEBUG::2013-11-20 03:00:37,467::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913281::DEBUG::2013-11-20 03:00:37,468::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913281::DEBUG::2013-11-20 03:00:37,469::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`471f2c6c-6da5-4239-b68c-14ec07173be8`::Granted request >Thread-15913281::DEBUG::2013-11-20 03:00:37,470::task::495::TaskManager.Task::(_debug) Task 4b6eb0e7-ad5f-41ea-af15-d2bc8e7a528d: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913281::DEBUG::2013-11-20 03:00:37,470::task::495::TaskManager.Task::(_debug) Task 4b6eb0e7-ad5f-41ea-af15-d2bc8e7a528d: ref 1 aborting False >Thread-15913281::WARNING::2013-11-20 03:00:37,472::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913281::DEBUG::2013-11-20 03:00:37,473::task::495::TaskManager.Task::(_debug) Task 4b6eb0e7-ad5f-41ea-af15-d2bc8e7a528d: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913281::DEBUG::2013-11-20 03:00:37,474::task::495::TaskManager.Task::(_debug) Task 4b6eb0e7-ad5f-41ea-af15-d2bc8e7a528d: moving from state preparing -> state finished >Thread-15913281::DEBUG::2013-11-20 03:00:37,474::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913281::DEBUG::2013-11-20 03:00:37,475::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913281::DEBUG::2013-11-20 03:00:37,476::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913281::DEBUG::2013-11-20 03:00:37,477::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913281::DEBUG::2013-11-20 03:00:37,477::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913281::DEBUG::2013-11-20 03:00:37,478::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913281::DEBUG::2013-11-20 03:00:37,478::task::495::TaskManager.Task::(_debug) Task 4b6eb0e7-ad5f-41ea-af15-d2bc8e7a528d: ref 0 aborting False >Thread-15913281::INFO::2013-11-20 03:00:37,479::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:00:37,753::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:38,306::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0791031 s, 12.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:00:40,322::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:40,847::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0761559 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:00:42,859::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:43,412::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0745475 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15913284::INFO::2013-11-20 03:00:45,377::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913284::DEBUG::2013-11-20 03:00:45,378::task::495::TaskManager.Task::(_debug) Task 3798064d-2353-4346-a5ce-cb8bf3dcaed3: moving from state init -> state preparing >Thread-15913284::DEBUG::2013-11-20 03:00:45,387::task::495::TaskManager.Task::(_debug) Task 3798064d-2353-4346-a5ce-cb8bf3dcaed3: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000326156616211', 'lastCheck': 1384916438.023623, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154089927673', 'lastCheck': 1384916435.8853381, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00140810012817', 'lastCheck': 1384916436.884948, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0024631023407', 'lastCheck': 1384916435.882122, 'valid': True, 'code': 0}} >Thread-15913284::DEBUG::2013-11-20 03:00:45,388::task::495::TaskManager.Task::(_debug) Task 3798064d-2353-4346-a5ce-cb8bf3dcaed3: moving from state preparing -> state finished >Thread-15913284::DEBUG::2013-11-20 03:00:45,389::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913284::DEBUG::2013-11-20 03:00:45,389::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913284::DEBUG::2013-11-20 03:00:45,390::task::495::TaskManager.Task::(_debug) Task 3798064d-2353-4346-a5ce-cb8bf3dcaed3: ref 0 aborting False >Thread-15913284::INFO::2013-11-20 03:00:45,391::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000326156616211', 'lastCheck': 1384916438.023623, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154089927673', 'lastCheck': 1384916435.8853381, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00140810012817', 'lastCheck': 1384916436.884948, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0024631023407', 'lastCheck': 1384916435.882122, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:00:45,426::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:45,954::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0706525 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15913286::DEBUG::2013-11-20 03:00:47,641::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913286::INFO::2013-11-20 03:00:47,643::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913286::DEBUG::2013-11-20 03:00:47,643::task::495::TaskManager.Task::(_debug) Task d0b336bd-0cf5-40d6-8ab3-90d557580abf: moving from state init -> state preparing >Thread-15913286::DEBUG::2013-11-20 03:00:47,644::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913286::DEBUG::2013-11-20 03:00:47,645::task::495::TaskManager.Task::(_debug) Task d0b336bd-0cf5-40d6-8ab3-90d557580abf: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913286::DEBUG::2013-11-20 03:00:47,645::task::495::TaskManager.Task::(_debug) Task d0b336bd-0cf5-40d6-8ab3-90d557580abf: moving from state preparing -> state finished >Thread-15913286::DEBUG::2013-11-20 03:00:47,646::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913286::DEBUG::2013-11-20 03:00:47,646::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913286::DEBUG::2013-11-20 03:00:47,647::task::495::TaskManager.Task::(_debug) Task d0b336bd-0cf5-40d6-8ab3-90d557580abf: ref 0 aborting False >Thread-15913286::INFO::2013-11-20 03:00:47,648::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913287::DEBUG::2013-11-20 03:00:47,652::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913287::INFO::2013-11-20 03:00:47,653::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913287::DEBUG::2013-11-20 03:00:47,654::task::495::TaskManager.Task::(_debug) Task 548df7e3-16b7-4cfe-bafa-8fc9deb662df: moving from state init -> state preparing >Thread-15913287::DEBUG::2013-11-20 03:00:47,655::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1ee2889c-f199-4c82-9fed-111da68a3cd4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913287::DEBUG::2013-11-20 03:00:47,655::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913287::DEBUG::2013-11-20 03:00:47,656::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913287::DEBUG::2013-11-20 03:00:47,657::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1ee2889c-f199-4c82-9fed-111da68a3cd4`::Granted request >Thread-15913287::DEBUG::2013-11-20 03:00:47,658::task::495::TaskManager.Task::(_debug) Task 548df7e3-16b7-4cfe-bafa-8fc9deb662df: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913287::DEBUG::2013-11-20 03:00:47,659::task::495::TaskManager.Task::(_debug) Task 548df7e3-16b7-4cfe-bafa-8fc9deb662df: ref 1 aborting False >Thread-15913287::WARNING::2013-11-20 03:00:47,661::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913287::DEBUG::2013-11-20 03:00:47,662::task::495::TaskManager.Task::(_debug) Task 548df7e3-16b7-4cfe-bafa-8fc9deb662df: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913287::DEBUG::2013-11-20 03:00:47,662::task::495::TaskManager.Task::(_debug) Task 548df7e3-16b7-4cfe-bafa-8fc9deb662df: moving from state preparing -> state finished >Thread-15913287::DEBUG::2013-11-20 03:00:47,663::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913287::DEBUG::2013-11-20 03:00:47,664::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913287::DEBUG::2013-11-20 03:00:47,665::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913287::DEBUG::2013-11-20 03:00:47,666::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913287::DEBUG::2013-11-20 03:00:47,667::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913287::DEBUG::2013-11-20 03:00:47,668::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913287::DEBUG::2013-11-20 03:00:47,669::task::495::TaskManager.Task::(_debug) Task 548df7e3-16b7-4cfe-bafa-8fc9deb662df: ref 0 aborting False >Thread-15913287::INFO::2013-11-20 03:00:47,670::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:00:47,969::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:48,479::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0723366 s, 14.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:00:50,495::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:51,003::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.065653 s, 15.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:00:53,019::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:53,571::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0736796 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:00:55,586::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:56,116::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0867857 s, 11.8 MB/s\n'; <rc> = 0 >Thread-15913291::DEBUG::2013-11-20 03:00:57,834::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913291::INFO::2013-11-20 03:00:57,836::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913291::DEBUG::2013-11-20 03:00:57,837::task::495::TaskManager.Task::(_debug) Task a0a510ba-2eca-4dee-a5a1-3fac9938c37f: moving from state init -> state preparing >Thread-15913291::DEBUG::2013-11-20 03:00:57,837::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913291::DEBUG::2013-11-20 03:00:57,838::task::495::TaskManager.Task::(_debug) Task a0a510ba-2eca-4dee-a5a1-3fac9938c37f: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913291::DEBUG::2013-11-20 03:00:57,839::task::495::TaskManager.Task::(_debug) Task a0a510ba-2eca-4dee-a5a1-3fac9938c37f: moving from state preparing -> state finished >Thread-15913291::DEBUG::2013-11-20 03:00:57,839::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913291::DEBUG::2013-11-20 03:00:57,840::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913291::DEBUG::2013-11-20 03:00:57,840::task::495::TaskManager.Task::(_debug) Task a0a510ba-2eca-4dee-a5a1-3fac9938c37f: ref 0 aborting False >Thread-15913291::INFO::2013-11-20 03:00:57,841::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913292::DEBUG::2013-11-20 03:00:57,846::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913292::INFO::2013-11-20 03:00:57,847::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913292::DEBUG::2013-11-20 03:00:57,848::task::495::TaskManager.Task::(_debug) Task 62974978-27bc-4f63-a96e-e2f93759c206: moving from state init -> state preparing >Thread-15913292::DEBUG::2013-11-20 03:00:57,850::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`41a2d9db-8cd1-4c80-bb10-9fcdaa4897d4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913292::DEBUG::2013-11-20 03:00:57,851::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913292::DEBUG::2013-11-20 03:00:57,852::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913292::DEBUG::2013-11-20 03:00:57,852::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`41a2d9db-8cd1-4c80-bb10-9fcdaa4897d4`::Granted request >Thread-15913292::DEBUG::2013-11-20 03:00:57,853::task::495::TaskManager.Task::(_debug) Task 62974978-27bc-4f63-a96e-e2f93759c206: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913292::DEBUG::2013-11-20 03:00:57,854::task::495::TaskManager.Task::(_debug) Task 62974978-27bc-4f63-a96e-e2f93759c206: ref 1 aborting False >Thread-15913292::WARNING::2013-11-20 03:00:57,856::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913292::DEBUG::2013-11-20 03:00:57,857::task::495::TaskManager.Task::(_debug) Task 62974978-27bc-4f63-a96e-e2f93759c206: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913292::DEBUG::2013-11-20 03:00:57,857::task::495::TaskManager.Task::(_debug) Task 62974978-27bc-4f63-a96e-e2f93759c206: moving from state preparing -> state finished >Thread-15913292::DEBUG::2013-11-20 03:00:57,858::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913292::DEBUG::2013-11-20 03:00:57,859::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913292::DEBUG::2013-11-20 03:00:57,859::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913292::DEBUG::2013-11-20 03:00:57,859::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913292::DEBUG::2013-11-20 03:00:57,860::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913292::DEBUG::2013-11-20 03:00:57,860::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913292::DEBUG::2013-11-20 03:00:57,860::task::495::TaskManager.Task::(_debug) Task 62974978-27bc-4f63-a96e-e2f93759c206: ref 0 aborting False >Thread-15913292::INFO::2013-11-20 03:00:57,861::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:00:58,131::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:00:58,682::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0808616 s, 12.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:01:00,697::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913294::INFO::2013-11-20 03:01:00,916::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913294::DEBUG::2013-11-20 03:01:00,917::task::495::TaskManager.Task::(_debug) Task f3580c5c-90ec-418e-9456-c735ef8b9de8: moving from state init -> state preparing >Thread-15913294::DEBUG::2013-11-20 03:01:00,918::task::495::TaskManager.Task::(_debug) Task f3580c5c-90ec-418e-9456-c735ef8b9de8: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00044584274292', 'lastCheck': 1384916458.365896, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00327491760254', 'lastCheck': 1384916456.270303, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00177979469299', 'lastCheck': 1384916457.269845, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00108098983765', 'lastCheck': 1384916456.268971, 'valid': True, 'code': 0}} >Thread-15913294::DEBUG::2013-11-20 03:01:00,919::task::495::TaskManager.Task::(_debug) Task f3580c5c-90ec-418e-9456-c735ef8b9de8: moving from state preparing -> state finished >Thread-15913294::DEBUG::2013-11-20 03:01:00,919::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913294::DEBUG::2013-11-20 03:01:00,920::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913294::DEBUG::2013-11-20 03:01:00,921::task::495::TaskManager.Task::(_debug) Task f3580c5c-90ec-418e-9456-c735ef8b9de8: ref 0 aborting False >Thread-15913294::INFO::2013-11-20 03:01:00,921::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00044584274292', 'lastCheck': 1384916458.365896, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00327491760254', 'lastCheck': 1384916456.270303, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00177979469299', 'lastCheck': 1384916457.269845, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00108098983765', 'lastCheck': 1384916456.268971, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:01:01,230::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0858177 s, 11.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:01:03,244::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:01:03,773::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.090755 s, 11.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:01:05,788::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:01:06,334::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0742484 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15913298::DEBUG::2013-11-20 03:01:08,037::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913298::INFO::2013-11-20 03:01:08,039::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913298::DEBUG::2013-11-20 03:01:08,039::task::495::TaskManager.Task::(_debug) Task 51575fa4-9d42-4d4c-bbfb-0bc96c9dd66f: moving from state init -> state preparing >Thread-15913298::DEBUG::2013-11-20 03:01:08,040::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913298::DEBUG::2013-11-20 03:01:08,041::task::495::TaskManager.Task::(_debug) Task 51575fa4-9d42-4d4c-bbfb-0bc96c9dd66f: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913298::DEBUG::2013-11-20 03:01:08,041::task::495::TaskManager.Task::(_debug) Task 51575fa4-9d42-4d4c-bbfb-0bc96c9dd66f: moving from state preparing -> state finished >Thread-15913298::DEBUG::2013-11-20 03:01:08,042::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913298::DEBUG::2013-11-20 03:01:08,042::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913298::DEBUG::2013-11-20 03:01:08,043::task::495::TaskManager.Task::(_debug) Task 51575fa4-9d42-4d4c-bbfb-0bc96c9dd66f: ref 0 aborting False >Thread-15913298::INFO::2013-11-20 03:01:08,044::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913299::DEBUG::2013-11-20 03:01:08,048::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913299::INFO::2013-11-20 03:01:08,049::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913299::DEBUG::2013-11-20 03:01:08,050::task::495::TaskManager.Task::(_debug) Task c1f61cd8-70e1-4401-b993-a2e90624ff9d: moving from state init -> state preparing >Thread-15913299::DEBUG::2013-11-20 03:01:08,051::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`034a566a-76da-41e4-8028-cbe2474db773`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913299::DEBUG::2013-11-20 03:01:08,051::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913299::DEBUG::2013-11-20 03:01:08,052::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913299::DEBUG::2013-11-20 03:01:08,053::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`034a566a-76da-41e4-8028-cbe2474db773`::Granted request >Thread-15913299::DEBUG::2013-11-20 03:01:08,054::task::495::TaskManager.Task::(_debug) Task c1f61cd8-70e1-4401-b993-a2e90624ff9d: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913299::DEBUG::2013-11-20 03:01:08,054::task::495::TaskManager.Task::(_debug) Task c1f61cd8-70e1-4401-b993-a2e90624ff9d: ref 1 aborting False >Thread-15913299::WARNING::2013-11-20 03:01:08,056::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913299::DEBUG::2013-11-20 03:01:08,057::task::495::TaskManager.Task::(_debug) Task c1f61cd8-70e1-4401-b993-a2e90624ff9d: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913299::DEBUG::2013-11-20 03:01:08,058::task::495::TaskManager.Task::(_debug) Task c1f61cd8-70e1-4401-b993-a2e90624ff9d: moving from state preparing -> state finished >Thread-15913299::DEBUG::2013-11-20 03:01:08,059::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913299::DEBUG::2013-11-20 03:01:08,059::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913299::DEBUG::2013-11-20 03:01:08,060::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913299::DEBUG::2013-11-20 03:01:08,060::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913299::DEBUG::2013-11-20 03:01:08,061::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913299::DEBUG::2013-11-20 03:01:08,061::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913299::DEBUG::2013-11-20 03:01:08,062::task::495::TaskManager.Task::(_debug) Task c1f61cd8-70e1-4401-b993-a2e90624ff9d: ref 0 aborting False >Thread-15913299::INFO::2013-11-20 03:01:08,063::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:01:08,349::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:01:08,859::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.07359 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:01:10,874::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:01:11,422::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0741545 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:01:13,437::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:01:13,963::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0743286 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:01:15,978::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913302::INFO::2013-11-20 03:01:16,233::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913302::DEBUG::2013-11-20 03:01:16,234::task::495::TaskManager.Task::(_debug) Task 34b05ea9-08cc-429b-ba7d-012603a7435a: moving from state init -> state preparing >Thread-15913302::DEBUG::2013-11-20 03:01:16,235::task::495::TaskManager.Task::(_debug) Task 34b05ea9-08cc-429b-ba7d-012603a7435a: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000546932220459', 'lastCheck': 1384916468.555757, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142288208008', 'lastCheck': 1384916466.468246, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0014660358429', 'lastCheck': 1384916467.468725, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000311136245728', 'lastCheck': 1384916466.463021, 'valid': True, 'code': 0}} >Thread-15913302::DEBUG::2013-11-20 03:01:16,236::task::495::TaskManager.Task::(_debug) Task 34b05ea9-08cc-429b-ba7d-012603a7435a: moving from state preparing -> state finished >Thread-15913302::DEBUG::2013-11-20 03:01:16,236::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913302::DEBUG::2013-11-20 03:01:16,236::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913302::DEBUG::2013-11-20 03:01:16,237::task::495::TaskManager.Task::(_debug) Task 34b05ea9-08cc-429b-ba7d-012603a7435a: ref 0 aborting False >Thread-15913302::INFO::2013-11-20 03:01:16,237::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000546932220459', 'lastCheck': 1384916468.555757, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142288208008', 'lastCheck': 1384916466.468246, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0014660358429', 'lastCheck': 1384916467.468725, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000311136245728', 'lastCheck': 1384916466.463021, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:01:16,477::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0724543 s, 14.1 MB/s\n'; <rc> = 0 >Thread-15913304::DEBUG::2013-11-20 03:01:18,239::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913304::INFO::2013-11-20 03:01:18,240::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913304::DEBUG::2013-11-20 03:01:18,240::task::495::TaskManager.Task::(_debug) Task f20796bc-530e-4795-ad85-fe5bccee819c: moving from state init -> state preparing >Thread-15913304::DEBUG::2013-11-20 03:01:18,241::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913304::DEBUG::2013-11-20 03:01:18,242::task::495::TaskManager.Task::(_debug) Task f20796bc-530e-4795-ad85-fe5bccee819c: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913304::DEBUG::2013-11-20 03:01:18,242::task::495::TaskManager.Task::(_debug) Task f20796bc-530e-4795-ad85-fe5bccee819c: moving from state preparing -> state finished >Thread-15913304::DEBUG::2013-11-20 03:01:18,243::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913304::DEBUG::2013-11-20 03:01:18,243::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913304::DEBUG::2013-11-20 03:01:18,244::task::495::TaskManager.Task::(_debug) Task f20796bc-530e-4795-ad85-fe5bccee819c: ref 0 aborting False >Thread-15913304::INFO::2013-11-20 03:01:18,245::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913305::DEBUG::2013-11-20 03:01:18,249::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913305::INFO::2013-11-20 03:01:18,250::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913305::DEBUG::2013-11-20 03:01:18,250::task::495::TaskManager.Task::(_debug) Task 86d62929-c9ce-4472-9e77-ad90ab530cb9: moving from state init -> state preparing >Thread-15913305::DEBUG::2013-11-20 03:01:18,251::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`203cb164-b15e-4a42-8720-a47729b46785`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913305::DEBUG::2013-11-20 03:01:18,252::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913305::DEBUG::2013-11-20 03:01:18,253::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913305::DEBUG::2013-11-20 03:01:18,253::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`203cb164-b15e-4a42-8720-a47729b46785`::Granted request >Thread-15913305::DEBUG::2013-11-20 03:01:18,254::task::495::TaskManager.Task::(_debug) Task 86d62929-c9ce-4472-9e77-ad90ab530cb9: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913305::DEBUG::2013-11-20 03:01:18,255::task::495::TaskManager.Task::(_debug) Task 86d62929-c9ce-4472-9e77-ad90ab530cb9: ref 1 aborting False >Thread-15913305::WARNING::2013-11-20 03:01:18,257::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913305::DEBUG::2013-11-20 03:01:18,258::task::495::TaskManager.Task::(_debug) Task 86d62929-c9ce-4472-9e77-ad90ab530cb9: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913305::DEBUG::2013-11-20 03:01:18,259::task::495::TaskManager.Task::(_debug) Task 86d62929-c9ce-4472-9e77-ad90ab530cb9: moving from state preparing -> state finished >Thread-15913305::DEBUG::2013-11-20 03:01:18,259::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913305::DEBUG::2013-11-20 03:01:18,260::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913305::DEBUG::2013-11-20 03:01:18,261::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913305::DEBUG::2013-11-20 03:01:18,261::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913305::DEBUG::2013-11-20 03:01:18,262::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913305::DEBUG::2013-11-20 03:01:18,262::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913305::DEBUG::2013-11-20 03:01:18,263::task::495::TaskManager.Task::(_debug) Task 86d62929-c9ce-4472-9e77-ad90ab530cb9: ref 0 aborting False >Thread-15913305::INFO::2013-11-20 03:01:18,264::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:01:18,491::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 03:01:18,748::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:01:18,750::task::495::TaskManager.Task::(_debug) Task 370b38d4-6c5b-4214-8610-787f878ebd43: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:01:18,752::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`29ccad81-cbd0-418e-a77b-4fc4fff830f6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:01:18,753::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:01:18,753::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:01:18,754::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`29ccad81-cbd0-418e-a77b-4fc4fff830f6`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:01:18,755::task::495::TaskManager.Task::(_debug) Task 370b38d4-6c5b-4214-8610-787f878ebd43: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:01:18,756::task::495::TaskManager.Task::(_debug) Task 370b38d4-6c5b-4214-8610-787f878ebd43: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:01:18,756::task::495::TaskManager.Task::(_debug) Task 370b38d4-6c5b-4214-8610-787f878ebd43: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:01:18,757::task::495::TaskManager.Task::(_debug) Task 370b38d4-6c5b-4214-8610-787f878ebd43: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:01:18,758::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:01:18,758::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:01:18,759::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:01:18,759::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:01:18,760::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:01:18,760::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:01:18,761::task::495::TaskManager.Task::(_debug) Task 370b38d4-6c5b-4214-8610-787f878ebd43: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:01:18,762::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:01:19,015::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0739352 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:01:21,030::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:01:21,484::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0458684 s, 22.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:01:23,495::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:01:24,047::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0743449 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:01:26,062::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:01:26,586::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.073581 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15913309::DEBUG::2013-11-20 03:01:28,412::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913309::INFO::2013-11-20 03:01:28,413::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913309::DEBUG::2013-11-20 03:01:28,413::task::495::TaskManager.Task::(_debug) Task 988bb947-a53a-4536-ab31-ccff7b049521: moving from state init -> state preparing >Thread-15913309::DEBUG::2013-11-20 03:01:28,414::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913309::DEBUG::2013-11-20 03:01:28,414::task::495::TaskManager.Task::(_debug) Task 988bb947-a53a-4536-ab31-ccff7b049521: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913309::DEBUG::2013-11-20 03:01:28,414::task::495::TaskManager.Task::(_debug) Task 988bb947-a53a-4536-ab31-ccff7b049521: moving from state preparing -> state finished >Thread-15913309::DEBUG::2013-11-20 03:01:28,415::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913309::DEBUG::2013-11-20 03:01:28,415::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913309::DEBUG::2013-11-20 03:01:28,415::task::495::TaskManager.Task::(_debug) Task 988bb947-a53a-4536-ab31-ccff7b049521: ref 0 aborting False >Thread-15913309::INFO::2013-11-20 03:01:28,416::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913310::DEBUG::2013-11-20 03:01:28,420::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913310::INFO::2013-11-20 03:01:28,420::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913310::DEBUG::2013-11-20 03:01:28,421::task::495::TaskManager.Task::(_debug) Task 2a32a0ba-90a9-4654-9b57-5038c2dfc074: moving from state init -> state preparing >Thread-15913310::DEBUG::2013-11-20 03:01:28,422::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bc61e3b1-af72-422e-a273-46520a0924c2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913310::DEBUG::2013-11-20 03:01:28,422::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913310::DEBUG::2013-11-20 03:01:28,422::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913310::DEBUG::2013-11-20 03:01:28,423::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bc61e3b1-af72-422e-a273-46520a0924c2`::Granted request >Thread-15913310::DEBUG::2013-11-20 03:01:28,424::task::495::TaskManager.Task::(_debug) Task 2a32a0ba-90a9-4654-9b57-5038c2dfc074: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913310::DEBUG::2013-11-20 03:01:28,424::task::495::TaskManager.Task::(_debug) Task 2a32a0ba-90a9-4654-9b57-5038c2dfc074: ref 1 aborting False >Thread-15913310::WARNING::2013-11-20 03:01:28,425::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913310::DEBUG::2013-11-20 03:01:28,426::task::495::TaskManager.Task::(_debug) Task 2a32a0ba-90a9-4654-9b57-5038c2dfc074: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913310::DEBUG::2013-11-20 03:01:28,426::task::495::TaskManager.Task::(_debug) Task 2a32a0ba-90a9-4654-9b57-5038c2dfc074: moving from state preparing -> state finished >Thread-15913310::DEBUG::2013-11-20 03:01:28,427::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913310::DEBUG::2013-11-20 03:01:28,427::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913310::DEBUG::2013-11-20 03:01:28,427::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913310::DEBUG::2013-11-20 03:01:28,428::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913310::DEBUG::2013-11-20 03:01:28,428::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913310::DEBUG::2013-11-20 03:01:28,429::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913310::DEBUG::2013-11-20 03:01:28,429::task::495::TaskManager.Task::(_debug) Task 2a32a0ba-90a9-4654-9b57-5038c2dfc074: ref 0 aborting False >Thread-15913310::INFO::2013-11-20 03:01:28,429::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:01:28,601::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-42838::DEBUG::2013-11-20 03:01:28,813::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:01:29,221::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.130417 s, 7.9 MB/s\n'; <rc> = 0 >Thread-42838::DEBUG::2013-11-20 03:01:29,669::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:01:31,236::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913312::INFO::2013-11-20 03:01:31,567::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913312::DEBUG::2013-11-20 03:01:31,568::task::495::TaskManager.Task::(_debug) Task c263956b-ccee-4036-8050-0796f32433ec: moving from state init -> state preparing >Thread-15913312::DEBUG::2013-11-20 03:01:31,569::task::495::TaskManager.Task::(_debug) Task c263956b-ccee-4036-8050-0796f32433ec: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000324010848999', 'lastCheck': 1384916489.6738529, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0138931274414', 'lastCheck': 1384916486.7516749, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0014271736145', 'lastCheck': 1384916487.7431469, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00140190124512', 'lastCheck': 1384916486.741118, 'valid': True, 'code': 0}} >Thread-15913312::DEBUG::2013-11-20 03:01:31,570::task::495::TaskManager.Task::(_debug) Task c263956b-ccee-4036-8050-0796f32433ec: moving from state preparing -> state finished >Thread-15913312::DEBUG::2013-11-20 03:01:31,571::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913312::DEBUG::2013-11-20 03:01:31,571::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913312::DEBUG::2013-11-20 03:01:31,572::task::495::TaskManager.Task::(_debug) Task c263956b-ccee-4036-8050-0796f32433ec: ref 0 aborting False >Thread-15913312::INFO::2013-11-20 03:01:31,573::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000324010848999', 'lastCheck': 1384916489.6738529, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0138931274414', 'lastCheck': 1384916486.7516749, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0014271736145', 'lastCheck': 1384916487.7431469, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00140190124512', 'lastCheck': 1384916486.741118, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:01:31,778::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0778191 s, 13.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:01:33,792::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:01:34,328::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0828243 s, 12.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:01:36,350::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:01:36,896::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0765621 s, 13.4 MB/s\n'; <rc> = 0 >Thread-15913316::DEBUG::2013-11-20 03:01:38,599::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913316::INFO::2013-11-20 03:01:38,601::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913316::DEBUG::2013-11-20 03:01:38,602::task::495::TaskManager.Task::(_debug) Task e5665f26-f7ad-4f5c-bf6b-b25fa43a78c1: moving from state init -> state preparing >Thread-15913316::DEBUG::2013-11-20 03:01:38,603::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913316::DEBUG::2013-11-20 03:01:38,603::task::495::TaskManager.Task::(_debug) Task e5665f26-f7ad-4f5c-bf6b-b25fa43a78c1: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913316::DEBUG::2013-11-20 03:01:38,604::task::495::TaskManager.Task::(_debug) Task e5665f26-f7ad-4f5c-bf6b-b25fa43a78c1: moving from state preparing -> state finished >Thread-15913316::DEBUG::2013-11-20 03:01:38,604::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913316::DEBUG::2013-11-20 03:01:38,605::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913316::DEBUG::2013-11-20 03:01:38,606::task::495::TaskManager.Task::(_debug) Task e5665f26-f7ad-4f5c-bf6b-b25fa43a78c1: ref 0 aborting False >Thread-15913316::INFO::2013-11-20 03:01:38,606::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913317::DEBUG::2013-11-20 03:01:38,611::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913317::INFO::2013-11-20 03:01:38,612::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913317::DEBUG::2013-11-20 03:01:38,613::task::495::TaskManager.Task::(_debug) Task 5cf2116b-bf15-4a13-a3c5-e271ed39bdbb: moving from state init -> state preparing >Thread-15913317::DEBUG::2013-11-20 03:01:38,615::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bf9c529c-2835-423e-a96e-7b8f9317d12a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913317::DEBUG::2013-11-20 03:01:38,615::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913317::DEBUG::2013-11-20 03:01:38,616::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913317::DEBUG::2013-11-20 03:01:38,617::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bf9c529c-2835-423e-a96e-7b8f9317d12a`::Granted request >Thread-15913317::DEBUG::2013-11-20 03:01:38,618::task::495::TaskManager.Task::(_debug) Task 5cf2116b-bf15-4a13-a3c5-e271ed39bdbb: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913317::DEBUG::2013-11-20 03:01:38,618::task::495::TaskManager.Task::(_debug) Task 5cf2116b-bf15-4a13-a3c5-e271ed39bdbb: ref 1 aborting False >Thread-15913317::WARNING::2013-11-20 03:01:38,620::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913317::DEBUG::2013-11-20 03:01:38,621::task::495::TaskManager.Task::(_debug) Task 5cf2116b-bf15-4a13-a3c5-e271ed39bdbb: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913317::DEBUG::2013-11-20 03:01:38,622::task::495::TaskManager.Task::(_debug) Task 5cf2116b-bf15-4a13-a3c5-e271ed39bdbb: moving from state preparing -> state finished >Thread-15913317::DEBUG::2013-11-20 03:01:38,623::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913317::DEBUG::2013-11-20 03:01:38,623::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913317::DEBUG::2013-11-20 03:01:38,624::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913317::DEBUG::2013-11-20 03:01:38,625::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913317::DEBUG::2013-11-20 03:01:38,625::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913317::DEBUG::2013-11-20 03:01:38,625::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913317::DEBUG::2013-11-20 03:01:38,626::task::495::TaskManager.Task::(_debug) Task 5cf2116b-bf15-4a13-a3c5-e271ed39bdbb: ref 0 aborting False >Thread-15913317::INFO::2013-11-20 03:01:38,627::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:01:38,913::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:01:39,413::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0521656 s, 19.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:01:41,424::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:01:41,983::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0786517 s, 13.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:01:43,997::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:01:44,550::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0708651 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:01:46,566::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:01:47,103::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0734173 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15913320::INFO::2013-11-20 03:01:47,254::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913320::DEBUG::2013-11-20 03:01:47,254::task::495::TaskManager.Task::(_debug) Task 88fa7de7-6bc2-4817-911c-389dd459150d: moving from state init -> state preparing >Thread-15913320::DEBUG::2013-11-20 03:01:47,255::task::495::TaskManager.Task::(_debug) Task 88fa7de7-6bc2-4817-911c-389dd459150d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000402927398682', 'lastCheck': 1384916499.687809, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00165915489197', 'lastCheck': 1384916507.2403829, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00153803825378', 'lastCheck': 1384916497.997853, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00156402587891', 'lastCheck': 1384916507.236151, 'valid': True, 'code': 0}} >Thread-15913320::DEBUG::2013-11-20 03:01:47,256::task::495::TaskManager.Task::(_debug) Task 88fa7de7-6bc2-4817-911c-389dd459150d: moving from state preparing -> state finished >Thread-15913320::DEBUG::2013-11-20 03:01:47,257::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913320::DEBUG::2013-11-20 03:01:47,257::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913320::DEBUG::2013-11-20 03:01:47,258::task::495::TaskManager.Task::(_debug) Task 88fa7de7-6bc2-4817-911c-389dd459150d: ref 0 aborting False >Thread-15913320::INFO::2013-11-20 03:01:47,258::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000402927398682', 'lastCheck': 1384916499.687809, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00165915489197', 'lastCheck': 1384916507.2403829, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00153803825378', 'lastCheck': 1384916497.997853, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00156402587891', 'lastCheck': 1384916507.236151, 'valid': True, 'code': 0}} >Thread-15913322::DEBUG::2013-11-20 03:01:48,802::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913322::INFO::2013-11-20 03:01:48,803::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913322::DEBUG::2013-11-20 03:01:48,804::task::495::TaskManager.Task::(_debug) Task 8bae6e04-23e2-4a4e-9125-093111ed40fd: moving from state init -> state preparing >Thread-15913322::DEBUG::2013-11-20 03:01:48,805::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913322::DEBUG::2013-11-20 03:01:48,805::task::495::TaskManager.Task::(_debug) Task 8bae6e04-23e2-4a4e-9125-093111ed40fd: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913322::DEBUG::2013-11-20 03:01:48,806::task::495::TaskManager.Task::(_debug) Task 8bae6e04-23e2-4a4e-9125-093111ed40fd: moving from state preparing -> state finished >Thread-15913322::DEBUG::2013-11-20 03:01:48,806::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913322::DEBUG::2013-11-20 03:01:48,807::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913322::DEBUG::2013-11-20 03:01:48,807::task::495::TaskManager.Task::(_debug) Task 8bae6e04-23e2-4a4e-9125-093111ed40fd: ref 0 aborting False >Thread-15913322::INFO::2013-11-20 03:01:48,808::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913323::DEBUG::2013-11-20 03:01:48,855::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913323::INFO::2013-11-20 03:01:48,855::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913323::DEBUG::2013-11-20 03:01:48,856::task::495::TaskManager.Task::(_debug) Task d0ac0b8d-438d-4a66-aee2-37d23a9d21eb: moving from state init -> state preparing >Thread-15913323::DEBUG::2013-11-20 03:01:48,857::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f537be83-0193-4564-9d01-5ead0b3ad94a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913323::DEBUG::2013-11-20 03:01:48,858::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913323::DEBUG::2013-11-20 03:01:48,859::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913323::DEBUG::2013-11-20 03:01:48,859::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f537be83-0193-4564-9d01-5ead0b3ad94a`::Granted request >Thread-15913323::DEBUG::2013-11-20 03:01:48,860::task::495::TaskManager.Task::(_debug) Task d0ac0b8d-438d-4a66-aee2-37d23a9d21eb: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913323::DEBUG::2013-11-20 03:01:48,861::task::495::TaskManager.Task::(_debug) Task d0ac0b8d-438d-4a66-aee2-37d23a9d21eb: ref 1 aborting False >Thread-15913323::WARNING::2013-11-20 03:01:48,862::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913323::DEBUG::2013-11-20 03:01:48,863::task::495::TaskManager.Task::(_debug) Task d0ac0b8d-438d-4a66-aee2-37d23a9d21eb: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913323::DEBUG::2013-11-20 03:01:48,864::task::495::TaskManager.Task::(_debug) Task d0ac0b8d-438d-4a66-aee2-37d23a9d21eb: moving from state preparing -> state finished >Thread-15913323::DEBUG::2013-11-20 03:01:48,865::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913323::DEBUG::2013-11-20 03:01:48,865::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913323::DEBUG::2013-11-20 03:01:48,866::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913323::DEBUG::2013-11-20 03:01:48,867::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913323::DEBUG::2013-11-20 03:01:48,867::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913323::DEBUG::2013-11-20 03:01:48,868::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913323::DEBUG::2013-11-20 03:01:48,868::task::495::TaskManager.Task::(_debug) Task d0ac0b8d-438d-4a66-aee2-37d23a9d21eb: ref 0 aborting False >Thread-15913323::INFO::2013-11-20 03:01:48,869::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:01:49,118::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:01:49,642::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0741295 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:01:51,659::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:01:52,096::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0468533 s, 21.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:01:54,106::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:01:54,665::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.116353 s, 8.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:01:56,681::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:01:57,239::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0773553 s, 13.2 MB/s\n'; <rc> = 0 >Thread-15913327::DEBUG::2013-11-20 03:01:59,222::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913327::INFO::2013-11-20 03:01:59,224::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913327::DEBUG::2013-11-20 03:01:59,224::task::495::TaskManager.Task::(_debug) Task 68428e70-c2c9-423b-9a07-6d4d7c11b410: moving from state init -> state preparing >Thread-15913327::DEBUG::2013-11-20 03:01:59,226::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913327::DEBUG::2013-11-20 03:01:59,226::task::495::TaskManager.Task::(_debug) Task 68428e70-c2c9-423b-9a07-6d4d7c11b410: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913327::DEBUG::2013-11-20 03:01:59,227::task::495::TaskManager.Task::(_debug) Task 68428e70-c2c9-423b-9a07-6d4d7c11b410: moving from state preparing -> state finished >Thread-15913327::DEBUG::2013-11-20 03:01:59,227::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913327::DEBUG::2013-11-20 03:01:59,228::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913327::DEBUG::2013-11-20 03:01:59,228::task::495::TaskManager.Task::(_debug) Task 68428e70-c2c9-423b-9a07-6d4d7c11b410: ref 0 aborting False >Thread-15913327::INFO::2013-11-20 03:01:59,229::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913328::DEBUG::2013-11-20 03:01:59,234::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913328::INFO::2013-11-20 03:01:59,235::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913328::DEBUG::2013-11-20 03:01:59,235::task::495::TaskManager.Task::(_debug) Task d3a9e0c5-3929-4187-b246-c80e3b503db5: moving from state init -> state preparing >Thread-15913328::DEBUG::2013-11-20 03:01:59,237::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`26e31e22-4ff2-4785-a499-ff75b3422507`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913328::DEBUG::2013-11-20 03:01:59,237::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913328::DEBUG::2013-11-20 03:01:59,238::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913328::DEBUG::2013-11-20 03:01:59,238::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`26e31e22-4ff2-4785-a499-ff75b3422507`::Granted request >Thread-15913328::DEBUG::2013-11-20 03:01:59,239::task::495::TaskManager.Task::(_debug) Task d3a9e0c5-3929-4187-b246-c80e3b503db5: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913328::DEBUG::2013-11-20 03:01:59,240::task::495::TaskManager.Task::(_debug) Task d3a9e0c5-3929-4187-b246-c80e3b503db5: ref 1 aborting False >Thread-15913328::WARNING::2013-11-20 03:01:59,242::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913328::DEBUG::2013-11-20 03:01:59,243::task::495::TaskManager.Task::(_debug) Task d3a9e0c5-3929-4187-b246-c80e3b503db5: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913328::DEBUG::2013-11-20 03:01:59,244::task::495::TaskManager.Task::(_debug) Task d3a9e0c5-3929-4187-b246-c80e3b503db5: moving from state preparing -> state finished >Thread-15913328::DEBUG::2013-11-20 03:01:59,244::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913328::DEBUG::2013-11-20 03:01:59,245::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913328::DEBUG::2013-11-20 03:01:59,245::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913328::DEBUG::2013-11-20 03:01:59,246::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913328::DEBUG::2013-11-20 03:01:59,247::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913328::DEBUG::2013-11-20 03:01:59,247::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913328::DEBUG::2013-11-20 03:01:59,248::task::495::TaskManager.Task::(_debug) Task d3a9e0c5-3929-4187-b246-c80e3b503db5: ref 0 aborting False >Thread-15913328::INFO::2013-11-20 03:01:59,248::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:01:59,254::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:01:59,811::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.073665 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:02:01,826::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:02:02,350::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0732466 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15913330::INFO::2013-11-20 03:02:02,954::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913330::DEBUG::2013-11-20 03:02:02,955::task::495::TaskManager.Task::(_debug) Task f10ba8a0-dd57-4897-98f2-f70833f031c3: moving from state init -> state preparing >Thread-15913330::DEBUG::2013-11-20 03:02:02,956::task::495::TaskManager.Task::(_debug) Task f10ba8a0-dd57-4897-98f2-f70833f031c3: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00102996826172', 'lastCheck': 1384916519.9193771, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00149202346802', 'lastCheck': 1384916517.3620491, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00170803070068', 'lastCheck': 1384916518.3754909, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000462055206299', 'lastCheck': 1384916517.3563211, 'valid': True, 'code': 0}} >Thread-15913330::DEBUG::2013-11-20 03:02:02,957::task::495::TaskManager.Task::(_debug) Task f10ba8a0-dd57-4897-98f2-f70833f031c3: moving from state preparing -> state finished >Thread-15913330::DEBUG::2013-11-20 03:02:02,957::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913330::DEBUG::2013-11-20 03:02:02,958::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913330::DEBUG::2013-11-20 03:02:02,958::task::495::TaskManager.Task::(_debug) Task f10ba8a0-dd57-4897-98f2-f70833f031c3: ref 0 aborting False >Thread-15913330::INFO::2013-11-20 03:02:02,959::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00102996826172', 'lastCheck': 1384916519.9193771, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00149202346802', 'lastCheck': 1384916517.3620491, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00170803070068', 'lastCheck': 1384916518.3754909, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000462055206299', 'lastCheck': 1384916517.3563211, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:02:04,365::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:02:04,902::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0872997 s, 11.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:02:06,917::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:02:07,471::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0875833 s, 11.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:02:09,486::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913334::DEBUG::2013-11-20 03:02:09,734::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913334::INFO::2013-11-20 03:02:09,745::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913334::DEBUG::2013-11-20 03:02:09,746::task::495::TaskManager.Task::(_debug) Task 7ac88b51-87e0-47e6-a756-e3f541ee5149: moving from state init -> state preparing >Thread-15913334::DEBUG::2013-11-20 03:02:09,748::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913334::DEBUG::2013-11-20 03:02:09,748::task::495::TaskManager.Task::(_debug) Task 7ac88b51-87e0-47e6-a756-e3f541ee5149: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913334::DEBUG::2013-11-20 03:02:09,749::task::495::TaskManager.Task::(_debug) Task 7ac88b51-87e0-47e6-a756-e3f541ee5149: moving from state preparing -> state finished >Thread-15913334::DEBUG::2013-11-20 03:02:09,750::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913334::DEBUG::2013-11-20 03:02:09,751::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913334::DEBUG::2013-11-20 03:02:09,752::task::495::TaskManager.Task::(_debug) Task 7ac88b51-87e0-47e6-a756-e3f541ee5149: ref 0 aborting False >Thread-15913334::INFO::2013-11-20 03:02:09,752::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913335::DEBUG::2013-11-20 03:02:09,758::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913335::INFO::2013-11-20 03:02:09,760::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913335::DEBUG::2013-11-20 03:02:09,760::task::495::TaskManager.Task::(_debug) Task 3e4d6625-cdaa-4ac9-b3f7-37155235a7fa: moving from state init -> state preparing >Thread-15913335::DEBUG::2013-11-20 03:02:09,762::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e7a25d4f-05f0-4b39-8f5d-b6c20582b5a7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913335::DEBUG::2013-11-20 03:02:09,763::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913335::DEBUG::2013-11-20 03:02:09,764::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913335::DEBUG::2013-11-20 03:02:09,764::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e7a25d4f-05f0-4b39-8f5d-b6c20582b5a7`::Granted request >Thread-15913335::DEBUG::2013-11-20 03:02:09,765::task::495::TaskManager.Task::(_debug) Task 3e4d6625-cdaa-4ac9-b3f7-37155235a7fa: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913335::DEBUG::2013-11-20 03:02:09,766::task::495::TaskManager.Task::(_debug) Task 3e4d6625-cdaa-4ac9-b3f7-37155235a7fa: ref 1 aborting False >Thread-15913335::WARNING::2013-11-20 03:02:09,768::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913335::DEBUG::2013-11-20 03:02:09,769::task::495::TaskManager.Task::(_debug) Task 3e4d6625-cdaa-4ac9-b3f7-37155235a7fa: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913335::DEBUG::2013-11-20 03:02:09,770::task::495::TaskManager.Task::(_debug) Task 3e4d6625-cdaa-4ac9-b3f7-37155235a7fa: moving from state preparing -> state finished >Thread-15913335::DEBUG::2013-11-20 03:02:09,771::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913335::DEBUG::2013-11-20 03:02:09,772::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913335::DEBUG::2013-11-20 03:02:09,772::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913335::DEBUG::2013-11-20 03:02:09,773::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913335::DEBUG::2013-11-20 03:02:09,774::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913335::DEBUG::2013-11-20 03:02:09,774::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913335::DEBUG::2013-11-20 03:02:09,775::task::495::TaskManager.Task::(_debug) Task 3e4d6625-cdaa-4ac9-b3f7-37155235a7fa: ref 0 aborting False >Thread-15913335::INFO::2013-11-20 03:02:09,776::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:02:10,058::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0767573 s, 13.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:02:12,074::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:02:12,577::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0624024 s, 16.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:02:14,593::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:02:15,122::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.06921 s, 14.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:02:17,138::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:02:17,665::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0709185 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15913338::INFO::2013-11-20 03:02:18,335::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913338::DEBUG::2013-11-20 03:02:18,336::task::495::TaskManager.Task::(_debug) Task 588a38d2-8b8b-4542-a007-2754c4717812: moving from state init -> state preparing >Thread-15913338::DEBUG::2013-11-20 03:02:18,337::task::495::TaskManager.Task::(_debug) Task 588a38d2-8b8b-4542-a007-2754c4717812: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000300168991089', 'lastCheck': 1384916530.1445191, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152683258057', 'lastCheck': 1384916537.8342071, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00139188766479', 'lastCheck': 1384916528.594934, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00247287750244', 'lastCheck': 1384916537.8298481, 'valid': True, 'code': 0}} >Thread-15913338::DEBUG::2013-11-20 03:02:18,338::task::495::TaskManager.Task::(_debug) Task 588a38d2-8b8b-4542-a007-2754c4717812: moving from state preparing -> state finished >Thread-15913338::DEBUG::2013-11-20 03:02:18,338::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913338::DEBUG::2013-11-20 03:02:18,339::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913338::DEBUG::2013-11-20 03:02:18,340::task::495::TaskManager.Task::(_debug) Task 588a38d2-8b8b-4542-a007-2754c4717812: ref 0 aborting False >Thread-15913338::INFO::2013-11-20 03:02:18,340::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000300168991089', 'lastCheck': 1384916530.1445191, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00152683258057', 'lastCheck': 1384916537.8342071, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00139188766479', 'lastCheck': 1384916528.594934, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00247287750244', 'lastCheck': 1384916537.8298481, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:02:19,680::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 03:02:19,914::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15913340::DEBUG::2013-11-20 03:02:19,917::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912392::DEBUG::2013-11-20 03:02:19,918::task::495::TaskManager.Task::(_debug) Task 1e3154f0-876e-4480-892d-8ea958176425: moving from state init -> state preparing >Thread-15913340::INFO::2013-11-20 03:02:19,925::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912392::DEBUG::2013-11-20 03:02:19,927::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`894e2ff8-356e-4309-9851-bf876d2521e7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913340::DEBUG::2013-11-20 03:02:19,928::task::495::TaskManager.Task::(_debug) Task 39b7bdc7-2092-47e1-8ece-a99300138c42: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:02:19,928::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913340::DEBUG::2013-11-20 03:02:19,929::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913340::DEBUG::2013-11-20 03:02:19,930::task::495::TaskManager.Task::(_debug) Task 39b7bdc7-2092-47e1-8ece-a99300138c42: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912392::DEBUG::2013-11-20 03:02:19,931::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913340::DEBUG::2013-11-20 03:02:19,931::task::495::TaskManager.Task::(_debug) Task 39b7bdc7-2092-47e1-8ece-a99300138c42: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:02:19,932::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`894e2ff8-356e-4309-9851-bf876d2521e7`::Granted request >Thread-15913340::DEBUG::2013-11-20 03:02:19,932::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912392::DEBUG::2013-11-20 03:02:19,933::task::495::TaskManager.Task::(_debug) Task 1e3154f0-876e-4480-892d-8ea958176425: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913340::DEBUG::2013-11-20 03:02:19,933::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:02:19,934::task::495::TaskManager.Task::(_debug) Task 1e3154f0-876e-4480-892d-8ea958176425: ref 1 aborting False >Thread-15913340::DEBUG::2013-11-20 03:02:19,934::task::495::TaskManager.Task::(_debug) Task 39b7bdc7-2092-47e1-8ece-a99300138c42: ref 0 aborting False >Thread-15912392::DEBUG::2013-11-20 03:02:19,935::task::495::TaskManager.Task::(_debug) Task 1e3154f0-876e-4480-892d-8ea958176425: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913340::INFO::2013-11-20 03:02:19,936::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912392::DEBUG::2013-11-20 03:02:19,936::task::495::TaskManager.Task::(_debug) Task 1e3154f0-876e-4480-892d-8ea958176425: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:02:19,937::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:02:19,938::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:02:19,939::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:02:19,939::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:02:19,940::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:02:19,940::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913341::DEBUG::2013-11-20 03:02:19,941::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912392::DEBUG::2013-11-20 03:02:19,942::task::495::TaskManager.Task::(_debug) Task 1e3154f0-876e-4480-892d-8ea958176425: ref 0 aborting False >Thread-15913341::INFO::2013-11-20 03:02:19,942::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15912392::INFO::2013-11-20 03:02:19,943::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913341::DEBUG::2013-11-20 03:02:19,943::task::495::TaskManager.Task::(_debug) Task c7020af0-9c5e-42d8-8138-d147408e70c4: moving from state init -> state preparing >Thread-15913341::DEBUG::2013-11-20 03:02:19,945::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`8e57cb1a-db32-4bfc-abd4-4e8e2d67a70e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913341::DEBUG::2013-11-20 03:02:19,945::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913341::DEBUG::2013-11-20 03:02:19,946::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913341::DEBUG::2013-11-20 03:02:19,947::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`8e57cb1a-db32-4bfc-abd4-4e8e2d67a70e`::Granted request >Thread-15913341::DEBUG::2013-11-20 03:02:19,949::task::495::TaskManager.Task::(_debug) Task c7020af0-9c5e-42d8-8138-d147408e70c4: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913341::DEBUG::2013-11-20 03:02:19,949::task::495::TaskManager.Task::(_debug) Task c7020af0-9c5e-42d8-8138-d147408e70c4: ref 1 aborting False >Thread-15913341::WARNING::2013-11-20 03:02:19,951::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913341::DEBUG::2013-11-20 03:02:19,952::task::495::TaskManager.Task::(_debug) Task c7020af0-9c5e-42d8-8138-d147408e70c4: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913341::DEBUG::2013-11-20 03:02:19,953::task::495::TaskManager.Task::(_debug) Task c7020af0-9c5e-42d8-8138-d147408e70c4: moving from state preparing -> state finished >Thread-15913341::DEBUG::2013-11-20 03:02:19,954::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913341::DEBUG::2013-11-20 03:02:19,954::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913341::DEBUG::2013-11-20 03:02:19,955::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913341::DEBUG::2013-11-20 03:02:19,956::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913341::DEBUG::2013-11-20 03:02:19,957::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913341::DEBUG::2013-11-20 03:02:19,957::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913341::DEBUG::2013-11-20 03:02:19,958::task::495::TaskManager.Task::(_debug) Task c7020af0-9c5e-42d8-8138-d147408e70c4: ref 0 aborting False >Thread-15913341::INFO::2013-11-20 03:02:19,959::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:02:20,200::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0718925 s, 14.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:02:22,215::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:02:22,706::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.070126 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:02:24,722::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:02:25,265::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0766892 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:02:27,281::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:02:27,846::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0743702 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:02:29,861::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913345::DEBUG::2013-11-20 03:02:30,108::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913345::INFO::2013-11-20 03:02:30,110::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913345::DEBUG::2013-11-20 03:02:30,111::task::495::TaskManager.Task::(_debug) Task 9d264c69-d678-400d-a9a0-ae65cead9815: moving from state init -> state preparing >Thread-15913345::DEBUG::2013-11-20 03:02:30,112::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913345::DEBUG::2013-11-20 03:02:30,113::task::495::TaskManager.Task::(_debug) Task 9d264c69-d678-400d-a9a0-ae65cead9815: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913345::DEBUG::2013-11-20 03:02:30,113::task::495::TaskManager.Task::(_debug) Task 9d264c69-d678-400d-a9a0-ae65cead9815: moving from state preparing -> state finished >Thread-15913345::DEBUG::2013-11-20 03:02:30,114::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913345::DEBUG::2013-11-20 03:02:30,114::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913345::DEBUG::2013-11-20 03:02:30,115::task::495::TaskManager.Task::(_debug) Task 9d264c69-d678-400d-a9a0-ae65cead9815: ref 0 aborting False >Thread-15913345::INFO::2013-11-20 03:02:30,116::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913346::DEBUG::2013-11-20 03:02:30,121::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913346::INFO::2013-11-20 03:02:30,122::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913346::DEBUG::2013-11-20 03:02:30,122::task::495::TaskManager.Task::(_debug) Task dd3a3077-e55a-4e99-b9f6-19dd36445482: moving from state init -> state preparing >Thread-15913346::DEBUG::2013-11-20 03:02:30,124::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`10a46d7b-2c18-4dc2-bca8-5234d21e8db8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913346::DEBUG::2013-11-20 03:02:30,124::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913346::DEBUG::2013-11-20 03:02:30,125::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913346::DEBUG::2013-11-20 03:02:30,126::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`10a46d7b-2c18-4dc2-bca8-5234d21e8db8`::Granted request >Thread-15913346::DEBUG::2013-11-20 03:02:30,127::task::495::TaskManager.Task::(_debug) Task dd3a3077-e55a-4e99-b9f6-19dd36445482: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913346::DEBUG::2013-11-20 03:02:30,128::task::495::TaskManager.Task::(_debug) Task dd3a3077-e55a-4e99-b9f6-19dd36445482: ref 1 aborting False >Thread-15913346::WARNING::2013-11-20 03:02:30,130::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913346::DEBUG::2013-11-20 03:02:30,131::task::495::TaskManager.Task::(_debug) Task dd3a3077-e55a-4e99-b9f6-19dd36445482: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913346::DEBUG::2013-11-20 03:02:30,132::task::495::TaskManager.Task::(_debug) Task dd3a3077-e55a-4e99-b9f6-19dd36445482: moving from state preparing -> state finished >Thread-15913346::DEBUG::2013-11-20 03:02:30,133::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913346::DEBUG::2013-11-20 03:02:30,133::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913346::DEBUG::2013-11-20 03:02:30,134::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913346::DEBUG::2013-11-20 03:02:30,135::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913346::DEBUG::2013-11-20 03:02:30,135::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913346::DEBUG::2013-11-20 03:02:30,136::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913346::DEBUG::2013-11-20 03:02:30,136::task::495::TaskManager.Task::(_debug) Task dd3a3077-e55a-4e99-b9f6-19dd36445482: ref 0 aborting False >Thread-15913346::INFO::2013-11-20 03:02:30,137::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:02:30,395::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0749153 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:02:32,410::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:02:32,955::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0737981 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15913348::INFO::2013-11-20 03:02:33,640::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913348::DEBUG::2013-11-20 03:02:33,641::task::495::TaskManager.Task::(_debug) Task 4d57c18a-de50-427c-b959-102cf5f53583: moving from state init -> state preparing >Thread-15913348::DEBUG::2013-11-20 03:02:33,642::task::495::TaskManager.Task::(_debug) Task 4d57c18a-de50-427c-b959-102cf5f53583: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000504970550537', 'lastCheck': 1384916550.5359321, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00235915184021', 'lastCheck': 1384916547.978622, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00185799598694', 'lastCheck': 1384916548.966913, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000635862350464', 'lastCheck': 1384916547.9574921, 'valid': True, 'code': 0}} >Thread-15913348::DEBUG::2013-11-20 03:02:33,642::task::495::TaskManager.Task::(_debug) Task 4d57c18a-de50-427c-b959-102cf5f53583: moving from state preparing -> state finished >Thread-15913348::DEBUG::2013-11-20 03:02:33,643::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913348::DEBUG::2013-11-20 03:02:33,643::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913348::DEBUG::2013-11-20 03:02:33,644::task::495::TaskManager.Task::(_debug) Task 4d57c18a-de50-427c-b959-102cf5f53583: ref 0 aborting False >Thread-15913348::INFO::2013-11-20 03:02:33,645::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000504970550537', 'lastCheck': 1384916550.5359321, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00235915184021', 'lastCheck': 1384916547.978622, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00185799598694', 'lastCheck': 1384916548.966913, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000635862350464', 'lastCheck': 1384916547.9574921, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:02:34,970::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:02:35,495::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.079021 s, 13.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:02:37,509::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:02:38,071::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0739297 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:02:40,087::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913352::DEBUG::2013-11-20 03:02:40,416::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913352::INFO::2013-11-20 03:02:40,418::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913352::DEBUG::2013-11-20 03:02:40,419::task::495::TaskManager.Task::(_debug) Task fb3eb998-0281-49d6-80fd-fa08693e565f: moving from state init -> state preparing >Thread-15913352::DEBUG::2013-11-20 03:02:40,420::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913352::DEBUG::2013-11-20 03:02:40,420::task::495::TaskManager.Task::(_debug) Task fb3eb998-0281-49d6-80fd-fa08693e565f: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913352::DEBUG::2013-11-20 03:02:40,421::task::495::TaskManager.Task::(_debug) Task fb3eb998-0281-49d6-80fd-fa08693e565f: moving from state preparing -> state finished >Thread-15913352::DEBUG::2013-11-20 03:02:40,422::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913352::DEBUG::2013-11-20 03:02:40,422::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913352::DEBUG::2013-11-20 03:02:40,423::task::495::TaskManager.Task::(_debug) Task fb3eb998-0281-49d6-80fd-fa08693e565f: ref 0 aborting False >Thread-15913352::INFO::2013-11-20 03:02:40,425::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913353::DEBUG::2013-11-20 03:02:40,430::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913353::INFO::2013-11-20 03:02:40,431::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913353::DEBUG::2013-11-20 03:02:40,433::task::495::TaskManager.Task::(_debug) Task 0fc5d766-75f4-4b33-9738-7089ca82430a: moving from state init -> state preparing >Thread-15913353::DEBUG::2013-11-20 03:02:40,436::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`7c0170fe-7812-4d5a-859c-ceb53044af91`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913353::DEBUG::2013-11-20 03:02:40,438::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913353::DEBUG::2013-11-20 03:02:40,440::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913353::DEBUG::2013-11-20 03:02:40,446::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`7c0170fe-7812-4d5a-859c-ceb53044af91`::Granted request >Thread-15913353::DEBUG::2013-11-20 03:02:40,450::task::495::TaskManager.Task::(_debug) Task 0fc5d766-75f4-4b33-9738-7089ca82430a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913353::DEBUG::2013-11-20 03:02:40,451::task::495::TaskManager.Task::(_debug) Task 0fc5d766-75f4-4b33-9738-7089ca82430a: ref 1 aborting False >Thread-15913353::WARNING::2013-11-20 03:02:40,454::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913353::DEBUG::2013-11-20 03:02:40,455::task::495::TaskManager.Task::(_debug) Task 0fc5d766-75f4-4b33-9738-7089ca82430a: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913353::DEBUG::2013-11-20 03:02:40,456::task::495::TaskManager.Task::(_debug) Task 0fc5d766-75f4-4b33-9738-7089ca82430a: moving from state preparing -> state finished >Thread-15913353::DEBUG::2013-11-20 03:02:40,456::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913353::DEBUG::2013-11-20 03:02:40,457::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913353::DEBUG::2013-11-20 03:02:40,457::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913353::DEBUG::2013-11-20 03:02:40,458::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913353::DEBUG::2013-11-20 03:02:40,459::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913353::DEBUG::2013-11-20 03:02:40,459::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913353::DEBUG::2013-11-20 03:02:40,460::task::495::TaskManager.Task::(_debug) Task 0fc5d766-75f4-4b33-9738-7089ca82430a: ref 0 aborting False >Thread-15913353::INFO::2013-11-20 03:02:40,460::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:02:40,630::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0731539 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:02:42,645::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:02:43,269::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.080722 s, 12.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:02:45,282::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:02:45,834::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0718047 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:02:47,849::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:02:48,356::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0624285 s, 16.4 MB/s\n'; <rc> = 0 >Thread-15913356::INFO::2013-11-20 03:02:49,394::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913356::DEBUG::2013-11-20 03:02:49,395::task::495::TaskManager.Task::(_debug) Task 643c0aad-cbf0-4235-9b78-32c2e56cf1c4: moving from state init -> state preparing >Thread-15913356::DEBUG::2013-11-20 03:02:49,396::task::495::TaskManager.Task::(_debug) Task 643c0aad-cbf0-4235-9b78-32c2e56cf1c4: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000452041625977', 'lastCheck': 1384916560.7507241, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139403343201', 'lastCheck': 1384916568.5079219, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00128793716431', 'lastCheck': 1384916559.195143, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000478982925415', 'lastCheck': 1384916568.5010929, 'valid': True, 'code': 0}} >Thread-15913356::DEBUG::2013-11-20 03:02:49,397::task::495::TaskManager.Task::(_debug) Task 643c0aad-cbf0-4235-9b78-32c2e56cf1c4: moving from state preparing -> state finished >Thread-15913356::DEBUG::2013-11-20 03:02:49,397::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913356::DEBUG::2013-11-20 03:02:49,398::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913356::DEBUG::2013-11-20 03:02:49,398::task::495::TaskManager.Task::(_debug) Task 643c0aad-cbf0-4235-9b78-32c2e56cf1c4: ref 0 aborting False >Thread-15913356::INFO::2013-11-20 03:02:49,399::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000452041625977', 'lastCheck': 1384916560.7507241, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139403343201', 'lastCheck': 1384916568.5079219, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00128793716431', 'lastCheck': 1384916559.195143, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000478982925415', 'lastCheck': 1384916568.5010929, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:02:50,372::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913358::DEBUG::2013-11-20 03:02:50,635::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913358::INFO::2013-11-20 03:02:50,637::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913358::DEBUG::2013-11-20 03:02:50,638::task::495::TaskManager.Task::(_debug) Task c2ca3987-b25c-41b1-b7eb-db3f59d8c9be: moving from state init -> state preparing >Thread-15913358::DEBUG::2013-11-20 03:02:50,639::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913358::DEBUG::2013-11-20 03:02:50,640::task::495::TaskManager.Task::(_debug) Task c2ca3987-b25c-41b1-b7eb-db3f59d8c9be: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913358::DEBUG::2013-11-20 03:02:50,641::task::495::TaskManager.Task::(_debug) Task c2ca3987-b25c-41b1-b7eb-db3f59d8c9be: moving from state preparing -> state finished >Thread-15913358::DEBUG::2013-11-20 03:02:50,642::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913358::DEBUG::2013-11-20 03:02:50,643::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913358::DEBUG::2013-11-20 03:02:50,643::task::495::TaskManager.Task::(_debug) Task c2ca3987-b25c-41b1-b7eb-db3f59d8c9be: ref 0 aborting False >Thread-15913358::INFO::2013-11-20 03:02:50,644::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913359::DEBUG::2013-11-20 03:02:50,652::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913359::INFO::2013-11-20 03:02:50,660::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913359::DEBUG::2013-11-20 03:02:50,666::task::495::TaskManager.Task::(_debug) Task 11caa5da-42ea-4244-bfcf-afab23af8d30: moving from state init -> state preparing >Thread-15913359::DEBUG::2013-11-20 03:02:50,675::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d9b0411e-85d6-4ba9-8a86-814673e824ff`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913359::DEBUG::2013-11-20 03:02:50,675::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913359::DEBUG::2013-11-20 03:02:50,676::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913359::DEBUG::2013-11-20 03:02:50,677::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d9b0411e-85d6-4ba9-8a86-814673e824ff`::Granted request >Thread-15913359::DEBUG::2013-11-20 03:02:50,678::task::495::TaskManager.Task::(_debug) Task 11caa5da-42ea-4244-bfcf-afab23af8d30: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913359::DEBUG::2013-11-20 03:02:50,679::task::495::TaskManager.Task::(_debug) Task 11caa5da-42ea-4244-bfcf-afab23af8d30: ref 1 aborting False >Thread-15913359::WARNING::2013-11-20 03:02:50,681::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913359::DEBUG::2013-11-20 03:02:50,682::task::495::TaskManager.Task::(_debug) Task 11caa5da-42ea-4244-bfcf-afab23af8d30: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913359::DEBUG::2013-11-20 03:02:50,683::task::495::TaskManager.Task::(_debug) Task 11caa5da-42ea-4244-bfcf-afab23af8d30: moving from state preparing -> state finished >Thread-15913359::DEBUG::2013-11-20 03:02:50,684::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913359::DEBUG::2013-11-20 03:02:50,684::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913359::DEBUG::2013-11-20 03:02:50,685::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913359::DEBUG::2013-11-20 03:02:50,686::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913359::DEBUG::2013-11-20 03:02:50,686::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913359::DEBUG::2013-11-20 03:02:50,687::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913359::DEBUG::2013-11-20 03:02:50,688::task::495::TaskManager.Task::(_debug) Task 11caa5da-42ea-4244-bfcf-afab23af8d30: ref 0 aborting False >Thread-15913359::INFO::2013-11-20 03:02:50,688::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:02:50,908::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0742906 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:02:52,923::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:02:53,479::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0800357 s, 12.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:02:55,493::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:02:56,062::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0734466 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:02:58,077::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:02:58,596::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0661102 s, 15.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:03:00,609::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:01,158::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0759352 s, 13.5 MB/s\n'; <rc> = 0 >Thread-15913363::DEBUG::2013-11-20 03:03:01,327::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913363::INFO::2013-11-20 03:03:01,328::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913363::DEBUG::2013-11-20 03:03:01,329::task::495::TaskManager.Task::(_debug) Task 442c7635-5a5b-4f32-a72b-afe54febb92c: moving from state init -> state preparing >Thread-15913363::DEBUG::2013-11-20 03:03:01,330::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913363::DEBUG::2013-11-20 03:03:01,330::task::495::TaskManager.Task::(_debug) Task 442c7635-5a5b-4f32-a72b-afe54febb92c: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913363::DEBUG::2013-11-20 03:03:01,331::task::495::TaskManager.Task::(_debug) Task 442c7635-5a5b-4f32-a72b-afe54febb92c: moving from state preparing -> state finished >Thread-15913363::DEBUG::2013-11-20 03:03:01,332::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913363::DEBUG::2013-11-20 03:03:01,332::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913363::DEBUG::2013-11-20 03:03:01,333::task::495::TaskManager.Task::(_debug) Task 442c7635-5a5b-4f32-a72b-afe54febb92c: ref 0 aborting False >Thread-15913363::INFO::2013-11-20 03:03:01,334::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913364::DEBUG::2013-11-20 03:03:01,339::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913364::INFO::2013-11-20 03:03:01,340::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913364::DEBUG::2013-11-20 03:03:01,340::task::495::TaskManager.Task::(_debug) Task 376ab2a2-8c89-4427-bb67-66301b5e1749: moving from state init -> state preparing >Thread-15913364::DEBUG::2013-11-20 03:03:01,342::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`7845797a-d2d0-4a69-8f46-18db55562257`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913364::DEBUG::2013-11-20 03:03:01,342::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913364::DEBUG::2013-11-20 03:03:01,343::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913364::DEBUG::2013-11-20 03:03:01,344::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`7845797a-d2d0-4a69-8f46-18db55562257`::Granted request >Thread-15913364::DEBUG::2013-11-20 03:03:01,345::task::495::TaskManager.Task::(_debug) Task 376ab2a2-8c89-4427-bb67-66301b5e1749: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913364::DEBUG::2013-11-20 03:03:01,345::task::495::TaskManager.Task::(_debug) Task 376ab2a2-8c89-4427-bb67-66301b5e1749: ref 1 aborting False >Thread-15913364::WARNING::2013-11-20 03:03:01,347::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913364::DEBUG::2013-11-20 03:03:01,348::task::495::TaskManager.Task::(_debug) Task 376ab2a2-8c89-4427-bb67-66301b5e1749: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913364::DEBUG::2013-11-20 03:03:01,349::task::495::TaskManager.Task::(_debug) Task 376ab2a2-8c89-4427-bb67-66301b5e1749: moving from state preparing -> state finished >Thread-15913364::DEBUG::2013-11-20 03:03:01,350::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913364::DEBUG::2013-11-20 03:03:01,350::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913364::DEBUG::2013-11-20 03:03:01,351::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913364::DEBUG::2013-11-20 03:03:01,351::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913364::DEBUG::2013-11-20 03:03:01,351::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913364::DEBUG::2013-11-20 03:03:01,352::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913364::DEBUG::2013-11-20 03:03:01,352::task::495::TaskManager.Task::(_debug) Task 376ab2a2-8c89-4427-bb67-66301b5e1749: ref 0 aborting False >Thread-15913364::INFO::2013-11-20 03:03:01,352::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:03:03,174::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:03,753::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0741728 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15913366::INFO::2013-11-20 03:03:05,511::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913366::DEBUG::2013-11-20 03:03:05,512::task::495::TaskManager.Task::(_debug) Task a9c8710d-3e2d-4b8c-8bd7-3aa184979a35: moving from state init -> state preparing >Thread-15913366::DEBUG::2013-11-20 03:03:05,513::task::495::TaskManager.Task::(_debug) Task a9c8710d-3e2d-4b8c-8bd7-3aa184979a35: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000447988510132', 'lastCheck': 1384916581.2859671, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00149416923523', 'lastCheck': 1384916578.755456, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00164604187012', 'lastCheck': 1384916579.755542, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000443935394287', 'lastCheck': 1384916578.7491939, 'valid': True, 'code': 0}} >Thread-15913366::DEBUG::2013-11-20 03:03:05,513::task::495::TaskManager.Task::(_debug) Task a9c8710d-3e2d-4b8c-8bd7-3aa184979a35: moving from state preparing -> state finished >Thread-15913366::DEBUG::2013-11-20 03:03:05,514::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913366::DEBUG::2013-11-20 03:03:05,514::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913366::DEBUG::2013-11-20 03:03:05,515::task::495::TaskManager.Task::(_debug) Task a9c8710d-3e2d-4b8c-8bd7-3aa184979a35: ref 0 aborting False >Thread-15913366::INFO::2013-11-20 03:03:05,516::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000447988510132', 'lastCheck': 1384916581.2859671, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00149416923523', 'lastCheck': 1384916578.755456, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00164604187012', 'lastCheck': 1384916579.755542, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000443935394287', 'lastCheck': 1384916578.7491939, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:03:05,765::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:06,335::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0754488 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:03:08,351::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:08,864::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0730617 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:03:10,880::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:11,435::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0721546 s, 14.2 MB/s\n'; <rc> = 0 >Thread-15913369::DEBUG::2013-11-20 03:03:11,696::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913369::INFO::2013-11-20 03:03:11,698::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913369::DEBUG::2013-11-20 03:03:11,698::task::495::TaskManager.Task::(_debug) Task 38cd50f8-d71a-41a8-b78b-23868a799274: moving from state init -> state preparing >Thread-15913369::DEBUG::2013-11-20 03:03:11,699::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913369::DEBUG::2013-11-20 03:03:11,700::task::495::TaskManager.Task::(_debug) Task 38cd50f8-d71a-41a8-b78b-23868a799274: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913369::DEBUG::2013-11-20 03:03:11,700::task::495::TaskManager.Task::(_debug) Task 38cd50f8-d71a-41a8-b78b-23868a799274: moving from state preparing -> state finished >Thread-15913369::DEBUG::2013-11-20 03:03:11,701::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913369::DEBUG::2013-11-20 03:03:11,701::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913369::DEBUG::2013-11-20 03:03:11,702::task::495::TaskManager.Task::(_debug) Task 38cd50f8-d71a-41a8-b78b-23868a799274: ref 0 aborting False >Thread-15913369::INFO::2013-11-20 03:03:11,702::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913370::DEBUG::2013-11-20 03:03:11,707::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913370::INFO::2013-11-20 03:03:11,708::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913370::DEBUG::2013-11-20 03:03:11,708::task::495::TaskManager.Task::(_debug) Task 32a757c5-7bd5-4bbc-809a-91b3ac554dda: moving from state init -> state preparing >Thread-15913370::DEBUG::2013-11-20 03:03:11,710::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5d6caa6a-6f04-4adf-bbe7-6a72dedf8ddb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913370::DEBUG::2013-11-20 03:03:11,710::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913370::DEBUG::2013-11-20 03:03:11,711::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913370::DEBUG::2013-11-20 03:03:11,712::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5d6caa6a-6f04-4adf-bbe7-6a72dedf8ddb`::Granted request >Thread-15913370::DEBUG::2013-11-20 03:03:11,719::task::495::TaskManager.Task::(_debug) Task 32a757c5-7bd5-4bbc-809a-91b3ac554dda: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913370::DEBUG::2013-11-20 03:03:11,720::task::495::TaskManager.Task::(_debug) Task 32a757c5-7bd5-4bbc-809a-91b3ac554dda: ref 1 aborting False >Thread-15913370::WARNING::2013-11-20 03:03:11,722::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913370::DEBUG::2013-11-20 03:03:11,722::task::495::TaskManager.Task::(_debug) Task 32a757c5-7bd5-4bbc-809a-91b3ac554dda: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913370::DEBUG::2013-11-20 03:03:11,723::task::495::TaskManager.Task::(_debug) Task 32a757c5-7bd5-4bbc-809a-91b3ac554dda: moving from state preparing -> state finished >Thread-15913370::DEBUG::2013-11-20 03:03:11,724::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913370::DEBUG::2013-11-20 03:03:11,725::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913370::DEBUG::2013-11-20 03:03:11,725::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913370::DEBUG::2013-11-20 03:03:11,726::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913370::DEBUG::2013-11-20 03:03:11,727::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913370::DEBUG::2013-11-20 03:03:11,727::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913370::DEBUG::2013-11-20 03:03:11,728::task::495::TaskManager.Task::(_debug) Task 32a757c5-7bd5-4bbc-809a-91b3ac554dda: ref 0 aborting False >Thread-15913370::INFO::2013-11-20 03:03:11,728::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:03:13,451::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:13,982::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0728576 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:03:15,997::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:16,577::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0951199 s, 10.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:03:18,592::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:19,097::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0970488 s, 10.6 MB/s\n'; <rc> = 0 >Thread-15913374::INFO::2013-11-20 03:03:20,984::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913374::DEBUG::2013-11-20 03:03:20,985::task::495::TaskManager.Task::(_debug) Task 8784b357-707d-4530-99da-018d27cc94b1: moving from state init -> state preparing >Thread-15913374::DEBUG::2013-11-20 03:03:20,986::task::495::TaskManager.Task::(_debug) Task 8784b357-707d-4530-99da-018d27cc94b1: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00052809715271', 'lastCheck': 1384916591.5548351, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00161504745483', 'lastCheck': 1384916599.236433, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00185298919678', 'lastCheck': 1384916600.2357991, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000537872314453', 'lastCheck': 1384916599.2286971, 'valid': True, 'code': 0}} >Thread-15913374::DEBUG::2013-11-20 03:03:20,987::task::495::TaskManager.Task::(_debug) Task 8784b357-707d-4530-99da-018d27cc94b1: moving from state preparing -> state finished >Thread-15913374::DEBUG::2013-11-20 03:03:20,987::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913374::DEBUG::2013-11-20 03:03:20,988::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913374::DEBUG::2013-11-20 03:03:20,989::task::495::TaskManager.Task::(_debug) Task 8784b357-707d-4530-99da-018d27cc94b1: ref 0 aborting False >Thread-15913374::INFO::2013-11-20 03:03:20,989::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00052809715271', 'lastCheck': 1384916591.5548351, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00161504745483', 'lastCheck': 1384916599.236433, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00185298919678', 'lastCheck': 1384916600.2357991, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000537872314453', 'lastCheck': 1384916599.2286971, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:03:21,113::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 03:03:21,347::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:03:21,349::task::495::TaskManager.Task::(_debug) Task d2329a02-00dd-4670-8649-52245e3e8d6b: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:03:21,352::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`7dc61396-46d5-42cf-a5f2-ddd2bb2a7b76`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:03:21,353::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:03:21,354::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:03:21,355::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`7dc61396-46d5-42cf-a5f2-ddd2bb2a7b76`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:03:21,356::task::495::TaskManager.Task::(_debug) Task d2329a02-00dd-4670-8649-52245e3e8d6b: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:03:21,356::task::495::TaskManager.Task::(_debug) Task d2329a02-00dd-4670-8649-52245e3e8d6b: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:03:21,357::task::495::TaskManager.Task::(_debug) Task d2329a02-00dd-4670-8649-52245e3e8d6b: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:03:21,358::task::495::TaskManager.Task::(_debug) Task d2329a02-00dd-4670-8649-52245e3e8d6b: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:03:21,358::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:03:21,359::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:03:21,360::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:03:21,360::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:03:21,361::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:03:21,361::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:03:21,362::task::495::TaskManager.Task::(_debug) Task d2329a02-00dd-4670-8649-52245e3e8d6b: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:03:21,363::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:03:21,670::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0741755 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15913376::DEBUG::2013-11-20 03:03:22,052::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913376::INFO::2013-11-20 03:03:22,053::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913376::DEBUG::2013-11-20 03:03:22,053::task::495::TaskManager.Task::(_debug) Task a656acbf-25fb-4d70-a093-45e762d74c16: moving from state init -> state preparing >Thread-15913376::DEBUG::2013-11-20 03:03:22,054::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913376::DEBUG::2013-11-20 03:03:22,055::task::495::TaskManager.Task::(_debug) Task a656acbf-25fb-4d70-a093-45e762d74c16: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913376::DEBUG::2013-11-20 03:03:22,056::task::495::TaskManager.Task::(_debug) Task a656acbf-25fb-4d70-a093-45e762d74c16: moving from state preparing -> state finished >Thread-15913376::DEBUG::2013-11-20 03:03:22,057::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913376::DEBUG::2013-11-20 03:03:22,057::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913376::DEBUG::2013-11-20 03:03:22,058::task::495::TaskManager.Task::(_debug) Task a656acbf-25fb-4d70-a093-45e762d74c16: ref 0 aborting False >Thread-15913376::INFO::2013-11-20 03:03:22,058::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913377::DEBUG::2013-11-20 03:03:22,063::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913377::INFO::2013-11-20 03:03:22,064::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913377::DEBUG::2013-11-20 03:03:22,065::task::495::TaskManager.Task::(_debug) Task 6bc315dc-cb7f-4fe7-8ea2-1fa8041b3adf: moving from state init -> state preparing >Thread-15913377::DEBUG::2013-11-20 03:03:22,066::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`173faeaa-2fc8-4d98-bc39-d0402d950825`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913377::DEBUG::2013-11-20 03:03:22,067::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913377::DEBUG::2013-11-20 03:03:22,067::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913377::DEBUG::2013-11-20 03:03:22,068::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`173faeaa-2fc8-4d98-bc39-d0402d950825`::Granted request >Thread-15913377::DEBUG::2013-11-20 03:03:22,069::task::495::TaskManager.Task::(_debug) Task 6bc315dc-cb7f-4fe7-8ea2-1fa8041b3adf: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913377::DEBUG::2013-11-20 03:03:22,070::task::495::TaskManager.Task::(_debug) Task 6bc315dc-cb7f-4fe7-8ea2-1fa8041b3adf: ref 1 aborting False >Thread-15913377::WARNING::2013-11-20 03:03:22,071::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913377::DEBUG::2013-11-20 03:03:22,072::task::495::TaskManager.Task::(_debug) Task 6bc315dc-cb7f-4fe7-8ea2-1fa8041b3adf: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913377::DEBUG::2013-11-20 03:03:22,073::task::495::TaskManager.Task::(_debug) Task 6bc315dc-cb7f-4fe7-8ea2-1fa8041b3adf: moving from state preparing -> state finished >Thread-15913377::DEBUG::2013-11-20 03:03:22,074::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913377::DEBUG::2013-11-20 03:03:22,074::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913377::DEBUG::2013-11-20 03:03:22,075::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913377::DEBUG::2013-11-20 03:03:22,076::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913377::DEBUG::2013-11-20 03:03:22,076::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913377::DEBUG::2013-11-20 03:03:22,077::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913377::DEBUG::2013-11-20 03:03:22,077::task::495::TaskManager.Task::(_debug) Task 6bc315dc-cb7f-4fe7-8ea2-1fa8041b3adf: ref 0 aborting False >Thread-15913377::INFO::2013-11-20 03:03:22,078::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:03:23,685::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:24,233::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0798883 s, 12.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:03:26,247::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:26,771::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0679499 s, 15.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:03:28,781::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:29,274::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0797188 s, 12.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:03:31,289::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:31,832::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0775865 s, 13.2 MB/s\n'; <rc> = 0 >Thread-15913381::DEBUG::2013-11-20 03:03:32,250::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913381::INFO::2013-11-20 03:03:32,251::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913381::DEBUG::2013-11-20 03:03:32,252::task::495::TaskManager.Task::(_debug) Task da01bf5c-3263-4211-b4e3-be30dd12f5ab: moving from state init -> state preparing >Thread-15913381::DEBUG::2013-11-20 03:03:32,253::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913381::DEBUG::2013-11-20 03:03:32,253::task::495::TaskManager.Task::(_debug) Task da01bf5c-3263-4211-b4e3-be30dd12f5ab: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913381::DEBUG::2013-11-20 03:03:32,254::task::495::TaskManager.Task::(_debug) Task da01bf5c-3263-4211-b4e3-be30dd12f5ab: moving from state preparing -> state finished >Thread-15913381::DEBUG::2013-11-20 03:03:32,254::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913381::DEBUG::2013-11-20 03:03:32,255::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913381::DEBUG::2013-11-20 03:03:32,255::task::495::TaskManager.Task::(_debug) Task da01bf5c-3263-4211-b4e3-be30dd12f5ab: ref 0 aborting False >Thread-15913381::INFO::2013-11-20 03:03:32,256::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913382::DEBUG::2013-11-20 03:03:32,260::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913382::INFO::2013-11-20 03:03:32,261::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913382::DEBUG::2013-11-20 03:03:32,262::task::495::TaskManager.Task::(_debug) Task 83f3105b-b343-40be-807e-199f22f1a036: moving from state init -> state preparing >Thread-15913382::DEBUG::2013-11-20 03:03:32,263::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6f2f2903-74d5-45d7-8bc2-c32f6da09260`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913382::DEBUG::2013-11-20 03:03:32,263::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913382::DEBUG::2013-11-20 03:03:32,264::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913382::DEBUG::2013-11-20 03:03:32,265::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6f2f2903-74d5-45d7-8bc2-c32f6da09260`::Granted request >Thread-15913382::DEBUG::2013-11-20 03:03:32,266::task::495::TaskManager.Task::(_debug) Task 83f3105b-b343-40be-807e-199f22f1a036: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913382::DEBUG::2013-11-20 03:03:32,266::task::495::TaskManager.Task::(_debug) Task 83f3105b-b343-40be-807e-199f22f1a036: ref 1 aborting False >Thread-15913382::WARNING::2013-11-20 03:03:32,268::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913382::DEBUG::2013-11-20 03:03:32,269::task::495::TaskManager.Task::(_debug) Task 83f3105b-b343-40be-807e-199f22f1a036: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913382::DEBUG::2013-11-20 03:03:32,270::task::495::TaskManager.Task::(_debug) Task 83f3105b-b343-40be-807e-199f22f1a036: moving from state preparing -> state finished >Thread-15913382::DEBUG::2013-11-20 03:03:32,271::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913382::DEBUG::2013-11-20 03:03:32,271::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913382::DEBUG::2013-11-20 03:03:32,272::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913382::DEBUG::2013-11-20 03:03:32,272::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913382::DEBUG::2013-11-20 03:03:32,273::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913382::DEBUG::2013-11-20 03:03:32,273::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913382::DEBUG::2013-11-20 03:03:32,274::task::495::TaskManager.Task::(_debug) Task 83f3105b-b343-40be-807e-199f22f1a036: ref 0 aborting False >Thread-15913382::INFO::2013-11-20 03:03:32,275::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:03:33,847::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:34,376::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0749086 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15913384::INFO::2013-11-20 03:03:36,299::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913384::DEBUG::2013-11-20 03:03:36,299::task::495::TaskManager.Task::(_debug) Task b5bdfd84-9909-4214-98a1-2b7c52e4fe42: moving from state init -> state preparing >Thread-15913384::DEBUG::2013-11-20 03:03:36,312::task::495::TaskManager.Task::(_debug) Task b5bdfd84-9909-4214-98a1-2b7c52e4fe42: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000366926193237', 'lastCheck': 1384916612.0060439, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0022919178009', 'lastCheck': 1384916609.486022, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00154995918274', 'lastCheck': 1384916610.4844, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0022599697113', 'lastCheck': 1384916609.4818709, 'valid': True, 'code': 0}} >Thread-15913384::DEBUG::2013-11-20 03:03:36,313::task::495::TaskManager.Task::(_debug) Task b5bdfd84-9909-4214-98a1-2b7c52e4fe42: moving from state preparing -> state finished >Thread-15913384::DEBUG::2013-11-20 03:03:36,314::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913384::DEBUG::2013-11-20 03:03:36,314::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913384::DEBUG::2013-11-20 03:03:36,315::task::495::TaskManager.Task::(_debug) Task b5bdfd84-9909-4214-98a1-2b7c52e4fe42: ref 0 aborting False >Thread-15913384::INFO::2013-11-20 03:03:36,316::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000366926193237', 'lastCheck': 1384916612.0060439, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0022919178009', 'lastCheck': 1384916609.486022, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00154995918274', 'lastCheck': 1384916610.4844, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0022599697113', 'lastCheck': 1384916609.4818709, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:03:36,392::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:36,932::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0774215 s, 13.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:03:38,948::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:39,487::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0739553 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:03:41,504::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:42,108::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0754468 s, 13.6 MB/s\n'; <rc> = 0 >Thread-15913387::DEBUG::2013-11-20 03:03:42,430::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913387::INFO::2013-11-20 03:03:42,431::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913387::DEBUG::2013-11-20 03:03:42,432::task::495::TaskManager.Task::(_debug) Task c0cbe6b9-a3f1-4175-aab3-3fb10e366ea3: moving from state init -> state preparing >Thread-15913387::DEBUG::2013-11-20 03:03:42,433::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913387::DEBUG::2013-11-20 03:03:42,433::task::495::TaskManager.Task::(_debug) Task c0cbe6b9-a3f1-4175-aab3-3fb10e366ea3: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913387::DEBUG::2013-11-20 03:03:42,434::task::495::TaskManager.Task::(_debug) Task c0cbe6b9-a3f1-4175-aab3-3fb10e366ea3: moving from state preparing -> state finished >Thread-15913387::DEBUG::2013-11-20 03:03:42,434::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913387::DEBUG::2013-11-20 03:03:42,435::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913387::DEBUG::2013-11-20 03:03:42,435::task::495::TaskManager.Task::(_debug) Task c0cbe6b9-a3f1-4175-aab3-3fb10e366ea3: ref 0 aborting False >Thread-15913387::INFO::2013-11-20 03:03:42,436::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913388::DEBUG::2013-11-20 03:03:42,441::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913388::INFO::2013-11-20 03:03:42,442::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913388::DEBUG::2013-11-20 03:03:42,442::task::495::TaskManager.Task::(_debug) Task ce5b82f9-59cd-4821-bf85-de211d31482c: moving from state init -> state preparing >Thread-15913388::DEBUG::2013-11-20 03:03:42,444::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3c29621a-c04b-4c22-b200-50d3c05e0e4c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913388::DEBUG::2013-11-20 03:03:42,444::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913388::DEBUG::2013-11-20 03:03:42,445::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913388::DEBUG::2013-11-20 03:03:42,446::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3c29621a-c04b-4c22-b200-50d3c05e0e4c`::Granted request >Thread-15913388::DEBUG::2013-11-20 03:03:42,447::task::495::TaskManager.Task::(_debug) Task ce5b82f9-59cd-4821-bf85-de211d31482c: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913388::DEBUG::2013-11-20 03:03:42,447::task::495::TaskManager.Task::(_debug) Task ce5b82f9-59cd-4821-bf85-de211d31482c: ref 1 aborting False >Thread-15913388::WARNING::2013-11-20 03:03:42,449::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913388::DEBUG::2013-11-20 03:03:42,450::task::495::TaskManager.Task::(_debug) Task ce5b82f9-59cd-4821-bf85-de211d31482c: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913388::DEBUG::2013-11-20 03:03:42,451::task::495::TaskManager.Task::(_debug) Task ce5b82f9-59cd-4821-bf85-de211d31482c: moving from state preparing -> state finished >Thread-15913388::DEBUG::2013-11-20 03:03:42,451::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913388::DEBUG::2013-11-20 03:03:42,452::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913388::DEBUG::2013-11-20 03:03:42,453::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913388::DEBUG::2013-11-20 03:03:42,453::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913388::DEBUG::2013-11-20 03:03:42,454::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913388::DEBUG::2013-11-20 03:03:42,454::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913388::DEBUG::2013-11-20 03:03:42,455::task::495::TaskManager.Task::(_debug) Task ce5b82f9-59cd-4821-bf85-de211d31482c: ref 0 aborting False >Thread-15913388::INFO::2013-11-20 03:03:42,456::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:03:44,123::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:44,710::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0738376 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:03:46,725::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:47,251::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0741322 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:03:49,265::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:49,851::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0709328 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15913392::INFO::2013-11-20 03:03:51,613::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913392::DEBUG::2013-11-20 03:03:51,614::task::495::TaskManager.Task::(_debug) Task c6342e1c-f81f-4e15-9c13-9149f93f595d: moving from state init -> state preparing >Thread-15913392::DEBUG::2013-11-20 03:03:51,615::task::495::TaskManager.Task::(_debug) Task c6342e1c-f81f-4e15-9c13-9149f93f595d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00189280509949', 'lastCheck': 1384916622.1717269, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00123906135559', 'lastCheck': 1384916629.9370401, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0157740116119', 'lastCheck': 1384916630.950314, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0022611618042', 'lastCheck': 1384916629.9343081, 'valid': True, 'code': 0}} >Thread-15913392::DEBUG::2013-11-20 03:03:51,616::task::495::TaskManager.Task::(_debug) Task c6342e1c-f81f-4e15-9c13-9149f93f595d: moving from state preparing -> state finished >Thread-15913392::DEBUG::2013-11-20 03:03:51,617::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913392::DEBUG::2013-11-20 03:03:51,618::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913392::DEBUG::2013-11-20 03:03:51,619::task::495::TaskManager.Task::(_debug) Task c6342e1c-f81f-4e15-9c13-9149f93f595d: ref 0 aborting False >Thread-15913392::INFO::2013-11-20 03:03:51,620::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00189280509949', 'lastCheck': 1384916622.1717269, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00123906135559', 'lastCheck': 1384916629.9370401, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0157740116119', 'lastCheck': 1384916630.950314, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0022611618042', 'lastCheck': 1384916629.9343081, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:03:51,865::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:52,429::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0797454 s, 12.8 MB/s\n'; <rc> = 0 >Thread-15913394::DEBUG::2013-11-20 03:03:52,989::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913394::INFO::2013-11-20 03:03:52,990::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913394::DEBUG::2013-11-20 03:03:52,991::task::495::TaskManager.Task::(_debug) Task 79ba4b79-8d05-4c15-8e68-a82e0436c2ba: moving from state init -> state preparing >Thread-15913394::DEBUG::2013-11-20 03:03:52,992::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913394::DEBUG::2013-11-20 03:03:52,992::task::495::TaskManager.Task::(_debug) Task 79ba4b79-8d05-4c15-8e68-a82e0436c2ba: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913394::DEBUG::2013-11-20 03:03:52,993::task::495::TaskManager.Task::(_debug) Task 79ba4b79-8d05-4c15-8e68-a82e0436c2ba: moving from state preparing -> state finished >Thread-15913394::DEBUG::2013-11-20 03:03:52,994::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913394::DEBUG::2013-11-20 03:03:52,994::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913394::DEBUG::2013-11-20 03:03:52,995::task::495::TaskManager.Task::(_debug) Task 79ba4b79-8d05-4c15-8e68-a82e0436c2ba: ref 0 aborting False >Thread-15913394::INFO::2013-11-20 03:03:52,995::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913395::DEBUG::2013-11-20 03:03:53,000::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913395::INFO::2013-11-20 03:03:53,001::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913395::DEBUG::2013-11-20 03:03:53,001::task::495::TaskManager.Task::(_debug) Task ee59354d-f0b7-486c-9164-3aa36d84afd7: moving from state init -> state preparing >Thread-15913395::DEBUG::2013-11-20 03:03:53,003::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`13819e59-3283-486b-89ee-325e0af42be3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913395::DEBUG::2013-11-20 03:03:53,003::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913395::DEBUG::2013-11-20 03:03:53,004::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913395::DEBUG::2013-11-20 03:03:53,005::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`13819e59-3283-486b-89ee-325e0af42be3`::Granted request >Thread-15913395::DEBUG::2013-11-20 03:03:53,006::task::495::TaskManager.Task::(_debug) Task ee59354d-f0b7-486c-9164-3aa36d84afd7: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913395::DEBUG::2013-11-20 03:03:53,006::task::495::TaskManager.Task::(_debug) Task ee59354d-f0b7-486c-9164-3aa36d84afd7: ref 1 aborting False >Thread-15913395::WARNING::2013-11-20 03:03:53,008::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913395::DEBUG::2013-11-20 03:03:53,009::task::495::TaskManager.Task::(_debug) Task ee59354d-f0b7-486c-9164-3aa36d84afd7: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913395::DEBUG::2013-11-20 03:03:53,010::task::495::TaskManager.Task::(_debug) Task ee59354d-f0b7-486c-9164-3aa36d84afd7: moving from state preparing -> state finished >Thread-15913395::DEBUG::2013-11-20 03:03:53,011::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913395::DEBUG::2013-11-20 03:03:53,011::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913395::DEBUG::2013-11-20 03:03:53,012::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913395::DEBUG::2013-11-20 03:03:53,012::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913395::DEBUG::2013-11-20 03:03:53,013::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913395::DEBUG::2013-11-20 03:03:53,013::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913395::DEBUG::2013-11-20 03:03:53,014::task::495::TaskManager.Task::(_debug) Task ee59354d-f0b7-486c-9164-3aa36d84afd7: ref 0 aborting False >Thread-15913395::INFO::2013-11-20 03:03:53,015::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:03:54,445::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:54,942::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0465415 s, 22.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:03:56,952::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:03:57,488::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0753649 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:03:59,503::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:00,085::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.073482 s, 13.9 MB/s\n'; <rc> = 0 >Thread-42840::DEBUG::2013-11-20 03:04:00,179::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 03:04:01,004::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:04:02,100::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:02,619::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0746035 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15913399::DEBUG::2013-11-20 03:04:03,327::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913399::INFO::2013-11-20 03:04:03,328::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913399::DEBUG::2013-11-20 03:04:03,329::task::495::TaskManager.Task::(_debug) Task 6e7dd196-d84b-4858-afd3-001755ae5005: moving from state init -> state preparing >Thread-15913399::DEBUG::2013-11-20 03:04:03,329::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913399::DEBUG::2013-11-20 03:04:03,330::task::495::TaskManager.Task::(_debug) Task 6e7dd196-d84b-4858-afd3-001755ae5005: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913399::DEBUG::2013-11-20 03:04:03,331::task::495::TaskManager.Task::(_debug) Task 6e7dd196-d84b-4858-afd3-001755ae5005: moving from state preparing -> state finished >Thread-15913399::DEBUG::2013-11-20 03:04:03,331::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913399::DEBUG::2013-11-20 03:04:03,332::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913399::DEBUG::2013-11-20 03:04:03,332::task::495::TaskManager.Task::(_debug) Task 6e7dd196-d84b-4858-afd3-001755ae5005: ref 0 aborting False >Thread-15913399::INFO::2013-11-20 03:04:03,333::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913400::DEBUG::2013-11-20 03:04:03,338::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913400::INFO::2013-11-20 03:04:03,338::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913400::DEBUG::2013-11-20 03:04:03,339::task::495::TaskManager.Task::(_debug) Task fefd0323-d77a-4534-b156-0a1541a3700e: moving from state init -> state preparing >Thread-15913400::DEBUG::2013-11-20 03:04:03,340::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e208a94c-911a-4e96-860c-234fe7d57b7d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913400::DEBUG::2013-11-20 03:04:03,341::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913400::DEBUG::2013-11-20 03:04:03,342::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913400::DEBUG::2013-11-20 03:04:03,342::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e208a94c-911a-4e96-860c-234fe7d57b7d`::Granted request >Thread-15913400::DEBUG::2013-11-20 03:04:03,343::task::495::TaskManager.Task::(_debug) Task fefd0323-d77a-4534-b156-0a1541a3700e: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913400::DEBUG::2013-11-20 03:04:03,344::task::495::TaskManager.Task::(_debug) Task fefd0323-d77a-4534-b156-0a1541a3700e: ref 1 aborting False >Thread-15913400::WARNING::2013-11-20 03:04:03,346::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913400::DEBUG::2013-11-20 03:04:03,346::task::495::TaskManager.Task::(_debug) Task fefd0323-d77a-4534-b156-0a1541a3700e: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913400::DEBUG::2013-11-20 03:04:03,347::task::495::TaskManager.Task::(_debug) Task fefd0323-d77a-4534-b156-0a1541a3700e: moving from state preparing -> state finished >Thread-15913400::DEBUG::2013-11-20 03:04:03,348::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913400::DEBUG::2013-11-20 03:04:03,348::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913400::DEBUG::2013-11-20 03:04:03,349::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913400::DEBUG::2013-11-20 03:04:03,350::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913400::DEBUG::2013-11-20 03:04:03,351::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913400::DEBUG::2013-11-20 03:04:03,351::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913400::DEBUG::2013-11-20 03:04:03,352::task::495::TaskManager.Task::(_debug) Task fefd0323-d77a-4534-b156-0a1541a3700e: ref 0 aborting False >Thread-15913400::INFO::2013-11-20 03:04:03,352::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:04:04,635::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:05,183::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0744093 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:04:07,198::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913402::INFO::2013-11-20 03:04:07,419::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913402::DEBUG::2013-11-20 03:04:07,420::task::495::TaskManager.Task::(_debug) Task 9ff4dd5a-8d63-46d3-8038-7b334a53ba35: moving from state init -> state preparing >Thread-15913402::DEBUG::2013-11-20 03:04:07,421::task::495::TaskManager.Task::(_debug) Task 9ff4dd5a-8d63-46d3-8038-7b334a53ba35: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00126481056213', 'lastCheck': 1384916642.7752211, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00344204902649', 'lastCheck': 1384916640.1808009, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00153112411499', 'lastCheck': 1384916641.1822331, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00454092025757', 'lastCheck': 1384916641.011745, 'valid': True, 'code': 0}} >Thread-15913402::DEBUG::2013-11-20 03:04:07,422::task::495::TaskManager.Task::(_debug) Task 9ff4dd5a-8d63-46d3-8038-7b334a53ba35: moving from state preparing -> state finished >Thread-15913402::DEBUG::2013-11-20 03:04:07,423::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913402::DEBUG::2013-11-20 03:04:07,424::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913402::DEBUG::2013-11-20 03:04:07,424::task::495::TaskManager.Task::(_debug) Task 9ff4dd5a-8d63-46d3-8038-7b334a53ba35: ref 0 aborting False >Thread-15913402::INFO::2013-11-20 03:04:07,425::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00126481056213', 'lastCheck': 1384916642.7752211, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00344204902649', 'lastCheck': 1384916640.1808009, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00153112411499', 'lastCheck': 1384916641.1822331, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00454092025757', 'lastCheck': 1384916641.011745, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:04:07,771::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0947713 s, 10.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:04:09,786::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:10,275::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0611202 s, 16.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:04:12,290::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:12,825::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0657174 s, 15.6 MB/s\n'; <rc> = 0 >Thread-15913405::DEBUG::2013-11-20 03:04:13,700::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913405::INFO::2013-11-20 03:04:13,701::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913405::DEBUG::2013-11-20 03:04:13,702::task::495::TaskManager.Task::(_debug) Task 20a7be5b-00b4-4f08-9f80-16cb399f9480: moving from state init -> state preparing >Thread-15913405::DEBUG::2013-11-20 03:04:13,702::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913405::DEBUG::2013-11-20 03:04:13,703::task::495::TaskManager.Task::(_debug) Task 20a7be5b-00b4-4f08-9f80-16cb399f9480: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913405::DEBUG::2013-11-20 03:04:13,704::task::495::TaskManager.Task::(_debug) Task 20a7be5b-00b4-4f08-9f80-16cb399f9480: moving from state preparing -> state finished >Thread-15913405::DEBUG::2013-11-20 03:04:13,704::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913405::DEBUG::2013-11-20 03:04:13,705::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913405::DEBUG::2013-11-20 03:04:13,705::task::495::TaskManager.Task::(_debug) Task 20a7be5b-00b4-4f08-9f80-16cb399f9480: ref 0 aborting False >Thread-15913405::INFO::2013-11-20 03:04:13,706::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913406::DEBUG::2013-11-20 03:04:13,710::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913406::INFO::2013-11-20 03:04:13,711::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913406::DEBUG::2013-11-20 03:04:13,712::task::495::TaskManager.Task::(_debug) Task 631e5b47-e5b5-476e-892b-147f04afc351: moving from state init -> state preparing >Thread-15913406::DEBUG::2013-11-20 03:04:13,713::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`61d4cb4b-6335-43c2-8f84-3f232cc753aa`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913406::DEBUG::2013-11-20 03:04:13,713::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913406::DEBUG::2013-11-20 03:04:13,714::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913406::DEBUG::2013-11-20 03:04:13,715::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`61d4cb4b-6335-43c2-8f84-3f232cc753aa`::Granted request >Thread-15913406::DEBUG::2013-11-20 03:04:13,716::task::495::TaskManager.Task::(_debug) Task 631e5b47-e5b5-476e-892b-147f04afc351: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913406::DEBUG::2013-11-20 03:04:13,716::task::495::TaskManager.Task::(_debug) Task 631e5b47-e5b5-476e-892b-147f04afc351: ref 1 aborting False >Thread-15913406::WARNING::2013-11-20 03:04:13,718::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913406::DEBUG::2013-11-20 03:04:13,719::task::495::TaskManager.Task::(_debug) Task 631e5b47-e5b5-476e-892b-147f04afc351: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913406::DEBUG::2013-11-20 03:04:13,720::task::495::TaskManager.Task::(_debug) Task 631e5b47-e5b5-476e-892b-147f04afc351: moving from state preparing -> state finished >Thread-15913406::DEBUG::2013-11-20 03:04:13,721::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913406::DEBUG::2013-11-20 03:04:13,721::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913406::DEBUG::2013-11-20 03:04:13,722::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913406::DEBUG::2013-11-20 03:04:13,722::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913406::DEBUG::2013-11-20 03:04:13,723::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913406::DEBUG::2013-11-20 03:04:13,723::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913406::DEBUG::2013-11-20 03:04:13,724::task::495::TaskManager.Task::(_debug) Task 631e5b47-e5b5-476e-892b-147f04afc351: ref 0 aborting False >Thread-15913406::INFO::2013-11-20 03:04:13,725::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:04:14,840::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:15,360::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0729735 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:04:17,376::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:17,932::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0737291 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:04:19,947::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:20,482::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0825499 s, 12.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:04:22,498::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 03:04:22,734::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:04:22,744::task::495::TaskManager.Task::(_debug) Task 7ce8ddc9-ce64-4a96-b894-500477c7456f: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:04:22,748::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e6072185-471a-4b0b-b442-9aeeed619848`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:04:22,749::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:04:22,751::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:04:22,751::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e6072185-471a-4b0b-b442-9aeeed619848`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:04:22,753::task::495::TaskManager.Task::(_debug) Task 7ce8ddc9-ce64-4a96-b894-500477c7456f: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:04:22,754::task::495::TaskManager.Task::(_debug) Task 7ce8ddc9-ce64-4a96-b894-500477c7456f: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:04:22,755::task::495::TaskManager.Task::(_debug) Task 7ce8ddc9-ce64-4a96-b894-500477c7456f: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:04:22,755::task::495::TaskManager.Task::(_debug) Task 7ce8ddc9-ce64-4a96-b894-500477c7456f: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:04:22,756::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:04:22,757::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:04:22,758::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:04:22,759::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:04:22,759::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:04:22,760::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:04:22,761::task::495::TaskManager.Task::(_debug) Task 7ce8ddc9-ce64-4a96-b894-500477c7456f: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:04:22,762::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:04:23,031::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0648384 s, 15.8 MB/s\n'; <rc> = 0 >Thread-15913410::INFO::2013-11-20 03:04:23,180::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913410::DEBUG::2013-11-20 03:04:23,180::task::495::TaskManager.Task::(_debug) Task b8cf6f43-3ef1-4d15-a81a-53d116878a92: moving from state init -> state preparing >Thread-15913410::DEBUG::2013-11-20 03:04:23,181::task::495::TaskManager.Task::(_debug) Task b8cf6f43-3ef1-4d15-a81a-53d116878a92: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000411987304688', 'lastCheck': 1384916663.1632409, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00121688842773', 'lastCheck': 1384916660.6189511, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00131988525391', 'lastCheck': 1384916661.6298881, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000414848327637', 'lastCheck': 1384916661.157161, 'valid': True, 'code': 0}} >Thread-15913410::DEBUG::2013-11-20 03:04:23,182::task::495::TaskManager.Task::(_debug) Task b8cf6f43-3ef1-4d15-a81a-53d116878a92: moving from state preparing -> state finished >Thread-15913410::DEBUG::2013-11-20 03:04:23,183::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913410::DEBUG::2013-11-20 03:04:23,183::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913410::DEBUG::2013-11-20 03:04:23,184::task::495::TaskManager.Task::(_debug) Task b8cf6f43-3ef1-4d15-a81a-53d116878a92: ref 0 aborting False >Thread-15913410::INFO::2013-11-20 03:04:23,184::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000411987304688', 'lastCheck': 1384916663.1632409, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00121688842773', 'lastCheck': 1384916660.6189511, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00131988525391', 'lastCheck': 1384916661.6298881, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000414848327637', 'lastCheck': 1384916661.157161, 'valid': True, 'code': 0}} >Thread-15913412::DEBUG::2013-11-20 03:04:24,081::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913412::INFO::2013-11-20 03:04:24,082::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913412::DEBUG::2013-11-20 03:04:24,083::task::495::TaskManager.Task::(_debug) Task a789e2c7-9512-45d9-b178-14fec1a2d7a4: moving from state init -> state preparing >Thread-15913412::DEBUG::2013-11-20 03:04:24,084::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913412::DEBUG::2013-11-20 03:04:24,084::task::495::TaskManager.Task::(_debug) Task a789e2c7-9512-45d9-b178-14fec1a2d7a4: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913412::DEBUG::2013-11-20 03:04:24,085::task::495::TaskManager.Task::(_debug) Task a789e2c7-9512-45d9-b178-14fec1a2d7a4: moving from state preparing -> state finished >Thread-15913412::DEBUG::2013-11-20 03:04:24,085::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913412::DEBUG::2013-11-20 03:04:24,086::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913412::DEBUG::2013-11-20 03:04:24,086::task::495::TaskManager.Task::(_debug) Task a789e2c7-9512-45d9-b178-14fec1a2d7a4: ref 0 aborting False >Thread-15913412::INFO::2013-11-20 03:04:24,087::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913413::DEBUG::2013-11-20 03:04:24,091::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913413::INFO::2013-11-20 03:04:24,092::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913413::DEBUG::2013-11-20 03:04:24,093::task::495::TaskManager.Task::(_debug) Task 40db714c-d015-4e5e-ade0-6ea908894062: moving from state init -> state preparing >Thread-15913413::DEBUG::2013-11-20 03:04:24,094::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e18194cc-c6b9-403d-a670-d42807c13b43`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913413::DEBUG::2013-11-20 03:04:24,094::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913413::DEBUG::2013-11-20 03:04:24,095::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913413::DEBUG::2013-11-20 03:04:24,096::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e18194cc-c6b9-403d-a670-d42807c13b43`::Granted request >Thread-15913413::DEBUG::2013-11-20 03:04:24,096::task::495::TaskManager.Task::(_debug) Task 40db714c-d015-4e5e-ade0-6ea908894062: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913413::DEBUG::2013-11-20 03:04:24,097::task::495::TaskManager.Task::(_debug) Task 40db714c-d015-4e5e-ade0-6ea908894062: ref 1 aborting False >Thread-15913413::WARNING::2013-11-20 03:04:24,099::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913413::DEBUG::2013-11-20 03:04:24,100::task::495::TaskManager.Task::(_debug) Task 40db714c-d015-4e5e-ade0-6ea908894062: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913413::DEBUG::2013-11-20 03:04:24,101::task::495::TaskManager.Task::(_debug) Task 40db714c-d015-4e5e-ade0-6ea908894062: moving from state preparing -> state finished >Thread-15913413::DEBUG::2013-11-20 03:04:24,101::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913413::DEBUG::2013-11-20 03:04:24,102::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913413::DEBUG::2013-11-20 03:04:24,103::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913413::DEBUG::2013-11-20 03:04:24,103::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913413::DEBUG::2013-11-20 03:04:24,104::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913413::DEBUG::2013-11-20 03:04:24,104::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913413::DEBUG::2013-11-20 03:04:24,105::task::495::TaskManager.Task::(_debug) Task 40db714c-d015-4e5e-ade0-6ea908894062: ref 0 aborting False >Thread-15913413::INFO::2013-11-20 03:04:24,106::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:04:25,045::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:25,590::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0734838 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:04:27,604::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:28,145::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.074059 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:04:30,160::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:30,712::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0735521 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:04:32,727::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:33,251::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0747814 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15913417::DEBUG::2013-11-20 03:04:34,240::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913417::INFO::2013-11-20 03:04:34,242::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913417::DEBUG::2013-11-20 03:04:34,243::task::495::TaskManager.Task::(_debug) Task 60ba0124-a81d-44d6-8535-5f900a106447: moving from state init -> state preparing >Thread-15913417::DEBUG::2013-11-20 03:04:34,243::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913417::DEBUG::2013-11-20 03:04:34,244::task::495::TaskManager.Task::(_debug) Task 60ba0124-a81d-44d6-8535-5f900a106447: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913417::DEBUG::2013-11-20 03:04:34,245::task::495::TaskManager.Task::(_debug) Task 60ba0124-a81d-44d6-8535-5f900a106447: moving from state preparing -> state finished >Thread-15913417::DEBUG::2013-11-20 03:04:34,245::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913417::DEBUG::2013-11-20 03:04:34,246::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913417::DEBUG::2013-11-20 03:04:34,246::task::495::TaskManager.Task::(_debug) Task 60ba0124-a81d-44d6-8535-5f900a106447: ref 0 aborting False >Thread-15913417::INFO::2013-11-20 03:04:34,247::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913418::DEBUG::2013-11-20 03:04:34,252::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913418::INFO::2013-11-20 03:04:34,252::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913418::DEBUG::2013-11-20 03:04:34,253::task::495::TaskManager.Task::(_debug) Task 1b741d93-7fc4-4a22-b836-21f1e58ac960: moving from state init -> state preparing >Thread-15913418::DEBUG::2013-11-20 03:04:34,254::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`507fe5d5-4a97-48fb-a8bb-843280ccc3bd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913418::DEBUG::2013-11-20 03:04:34,255::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913418::DEBUG::2013-11-20 03:04:34,256::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913418::DEBUG::2013-11-20 03:04:34,256::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`507fe5d5-4a97-48fb-a8bb-843280ccc3bd`::Granted request >Thread-15913418::DEBUG::2013-11-20 03:04:34,257::task::495::TaskManager.Task::(_debug) Task 1b741d93-7fc4-4a22-b836-21f1e58ac960: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913418::DEBUG::2013-11-20 03:04:34,258::task::495::TaskManager.Task::(_debug) Task 1b741d93-7fc4-4a22-b836-21f1e58ac960: ref 1 aborting False >Thread-15913418::WARNING::2013-11-20 03:04:34,260::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913418::DEBUG::2013-11-20 03:04:34,261::task::495::TaskManager.Task::(_debug) Task 1b741d93-7fc4-4a22-b836-21f1e58ac960: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913418::DEBUG::2013-11-20 03:04:34,261::task::495::TaskManager.Task::(_debug) Task 1b741d93-7fc4-4a22-b836-21f1e58ac960: moving from state preparing -> state finished >Thread-15913418::DEBUG::2013-11-20 03:04:34,262::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913418::DEBUG::2013-11-20 03:04:34,263::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913418::DEBUG::2013-11-20 03:04:34,263::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913418::DEBUG::2013-11-20 03:04:34,264::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913418::DEBUG::2013-11-20 03:04:34,264::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913418::DEBUG::2013-11-20 03:04:34,265::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913418::DEBUG::2013-11-20 03:04:34,265::task::495::TaskManager.Task::(_debug) Task 1b741d93-7fc4-4a22-b836-21f1e58ac960: ref 0 aborting False >Thread-15913418::INFO::2013-11-20 03:04:34,266::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:04:35,264::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:35,817::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.073512 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:04:37,832::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:38,342::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0769311 s, 13.3 MB/s\n'; <rc> = 0 >Thread-15913420::INFO::2013-11-20 03:04:38,522::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913420::DEBUG::2013-11-20 03:04:38,523::task::495::TaskManager.Task::(_debug) Task 42d41144-bca7-4fd8-8e80-af6952184c77: moving from state init -> state preparing >Thread-15913420::DEBUG::2013-11-20 03:04:38,524::task::495::TaskManager.Task::(_debug) Task 42d41144-bca7-4fd8-8e80-af6952184c77: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000554084777832', 'lastCheck': 1384916673.4057069, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00281405448914', 'lastCheck': 1384916670.8374691, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00169396400452', 'lastCheck': 1384916671.8234911, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000450849533081', 'lastCheck': 1384916671.397594, 'valid': True, 'code': 0}} >Thread-15913420::DEBUG::2013-11-20 03:04:38,525::task::495::TaskManager.Task::(_debug) Task 42d41144-bca7-4fd8-8e80-af6952184c77: moving from state preparing -> state finished >Thread-15913420::DEBUG::2013-11-20 03:04:38,525::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913420::DEBUG::2013-11-20 03:04:38,526::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913420::DEBUG::2013-11-20 03:04:38,527::task::495::TaskManager.Task::(_debug) Task 42d41144-bca7-4fd8-8e80-af6952184c77: ref 0 aborting False >Thread-15913420::INFO::2013-11-20 03:04:38,527::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000554084777832', 'lastCheck': 1384916673.4057069, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00281405448914', 'lastCheck': 1384916670.8374691, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00169396400452', 'lastCheck': 1384916671.8234911, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000450849533081', 'lastCheck': 1384916671.397594, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:04:40,357::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:40,899::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0701291 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:04:42,915::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:43,413::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0528653 s, 19.4 MB/s\n'; <rc> = 0 >Thread-15913423::DEBUG::2013-11-20 03:04:44,406::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913423::INFO::2013-11-20 03:04:44,408::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913423::DEBUG::2013-11-20 03:04:44,409::task::495::TaskManager.Task::(_debug) Task 8ad3370f-ad8d-443e-b013-93e496298393: moving from state init -> state preparing >Thread-15913423::DEBUG::2013-11-20 03:04:44,409::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913423::DEBUG::2013-11-20 03:04:44,410::task::495::TaskManager.Task::(_debug) Task 8ad3370f-ad8d-443e-b013-93e496298393: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913423::DEBUG::2013-11-20 03:04:44,411::task::495::TaskManager.Task::(_debug) Task 8ad3370f-ad8d-443e-b013-93e496298393: moving from state preparing -> state finished >Thread-15913423::DEBUG::2013-11-20 03:04:44,411::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913423::DEBUG::2013-11-20 03:04:44,412::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913423::DEBUG::2013-11-20 03:04:44,412::task::495::TaskManager.Task::(_debug) Task 8ad3370f-ad8d-443e-b013-93e496298393: ref 0 aborting False >Thread-15913423::INFO::2013-11-20 03:04:44,413::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913424::DEBUG::2013-11-20 03:04:44,417::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913424::INFO::2013-11-20 03:04:44,418::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913424::DEBUG::2013-11-20 03:04:44,419::task::495::TaskManager.Task::(_debug) Task c9830739-0e42-4874-9439-2d920f5ea0ee: moving from state init -> state preparing >Thread-15913424::DEBUG::2013-11-20 03:04:44,420::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`7a046fed-ec53-4b4d-92f2-551f723c4c3b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913424::DEBUG::2013-11-20 03:04:44,421::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913424::DEBUG::2013-11-20 03:04:44,422::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913424::DEBUG::2013-11-20 03:04:44,422::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`7a046fed-ec53-4b4d-92f2-551f723c4c3b`::Granted request >Thread-15913424::DEBUG::2013-11-20 03:04:44,423::task::495::TaskManager.Task::(_debug) Task c9830739-0e42-4874-9439-2d920f5ea0ee: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913424::DEBUG::2013-11-20 03:04:44,424::task::495::TaskManager.Task::(_debug) Task c9830739-0e42-4874-9439-2d920f5ea0ee: ref 1 aborting False >Thread-15913424::WARNING::2013-11-20 03:04:44,426::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913424::DEBUG::2013-11-20 03:04:44,427::task::495::TaskManager.Task::(_debug) Task c9830739-0e42-4874-9439-2d920f5ea0ee: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913424::DEBUG::2013-11-20 03:04:44,427::task::495::TaskManager.Task::(_debug) Task c9830739-0e42-4874-9439-2d920f5ea0ee: moving from state preparing -> state finished >Thread-15913424::DEBUG::2013-11-20 03:04:44,428::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913424::DEBUG::2013-11-20 03:04:44,428::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913424::DEBUG::2013-11-20 03:04:44,429::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913424::DEBUG::2013-11-20 03:04:44,430::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913424::DEBUG::2013-11-20 03:04:44,430::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913424::DEBUG::2013-11-20 03:04:44,431::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913424::DEBUG::2013-11-20 03:04:44,431::task::495::TaskManager.Task::(_debug) Task c9830739-0e42-4874-9439-2d920f5ea0ee: ref 0 aborting False >Thread-15913424::INFO::2013-11-20 03:04:44,432::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:04:45,424::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:45,951::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0864471 s, 11.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:04:47,966::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:48,481::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0802927 s, 12.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:04:50,496::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:51,057::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0871984 s, 11.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:04:53,072::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:53,547::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.071315 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15913428::INFO::2013-11-20 03:04:53,778::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913428::DEBUG::2013-11-20 03:04:53,779::task::495::TaskManager.Task::(_debug) Task e7db8e07-07e1-4cc2-a2fd-3e826426053b: moving from state init -> state preparing >Thread-15913428::DEBUG::2013-11-20 03:04:53,780::task::495::TaskManager.Task::(_debug) Task e7db8e07-07e1-4cc2-a2fd-3e826426053b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000581026077271', 'lastCheck': 1384916693.7411571, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142788887024', 'lastCheck': 1384916691.179208, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0096869468689', 'lastCheck': 1384916692.1922121, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000582933425903', 'lastCheck': 1384916691.733341, 'valid': True, 'code': 0}} >Thread-15913428::DEBUG::2013-11-20 03:04:53,781::task::495::TaskManager.Task::(_debug) Task e7db8e07-07e1-4cc2-a2fd-3e826426053b: moving from state preparing -> state finished >Thread-15913428::DEBUG::2013-11-20 03:04:53,781::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913428::DEBUG::2013-11-20 03:04:53,782::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913428::DEBUG::2013-11-20 03:04:53,782::task::495::TaskManager.Task::(_debug) Task e7db8e07-07e1-4cc2-a2fd-3e826426053b: ref 0 aborting False >Thread-15913428::INFO::2013-11-20 03:04:53,783::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000581026077271', 'lastCheck': 1384916693.7411571, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142788887024', 'lastCheck': 1384916691.179208, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0096869468689', 'lastCheck': 1384916692.1922121, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000582933425903', 'lastCheck': 1384916691.733341, 'valid': True, 'code': 0}} >Thread-15913430::DEBUG::2013-11-20 03:04:54,789::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913430::INFO::2013-11-20 03:04:54,790::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913430::DEBUG::2013-11-20 03:04:54,790::task::495::TaskManager.Task::(_debug) Task 1199d935-15d6-49e8-af4a-8756b7fab69a: moving from state init -> state preparing >Thread-15913430::DEBUG::2013-11-20 03:04:54,791::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913430::DEBUG::2013-11-20 03:04:54,792::task::495::TaskManager.Task::(_debug) Task 1199d935-15d6-49e8-af4a-8756b7fab69a: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913430::DEBUG::2013-11-20 03:04:54,792::task::495::TaskManager.Task::(_debug) Task 1199d935-15d6-49e8-af4a-8756b7fab69a: moving from state preparing -> state finished >Thread-15913430::DEBUG::2013-11-20 03:04:54,793::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913430::DEBUG::2013-11-20 03:04:54,793::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913430::DEBUG::2013-11-20 03:04:54,794::task::495::TaskManager.Task::(_debug) Task 1199d935-15d6-49e8-af4a-8756b7fab69a: ref 0 aborting False >Thread-15913430::INFO::2013-11-20 03:04:54,795::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913431::DEBUG::2013-11-20 03:04:54,799::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913431::INFO::2013-11-20 03:04:54,800::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913431::DEBUG::2013-11-20 03:04:54,800::task::495::TaskManager.Task::(_debug) Task eb48afc5-5a9e-4c9c-b26f-c32e744d18ae: moving from state init -> state preparing >Thread-15913431::DEBUG::2013-11-20 03:04:54,802::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`66078b24-cbd6-4c10-8245-ee02ce1a4483`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913431::DEBUG::2013-11-20 03:04:54,803::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913431::DEBUG::2013-11-20 03:04:54,804::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913431::DEBUG::2013-11-20 03:04:54,805::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`66078b24-cbd6-4c10-8245-ee02ce1a4483`::Granted request >Thread-15913431::DEBUG::2013-11-20 03:04:54,807::task::495::TaskManager.Task::(_debug) Task eb48afc5-5a9e-4c9c-b26f-c32e744d18ae: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913431::DEBUG::2013-11-20 03:04:54,808::task::495::TaskManager.Task::(_debug) Task eb48afc5-5a9e-4c9c-b26f-c32e744d18ae: ref 1 aborting False >Thread-15913431::WARNING::2013-11-20 03:04:54,811::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913431::DEBUG::2013-11-20 03:04:54,813::task::495::TaskManager.Task::(_debug) Task eb48afc5-5a9e-4c9c-b26f-c32e744d18ae: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913431::DEBUG::2013-11-20 03:04:54,814::task::495::TaskManager.Task::(_debug) Task eb48afc5-5a9e-4c9c-b26f-c32e744d18ae: moving from state preparing -> state finished >Thread-15913431::DEBUG::2013-11-20 03:04:54,816::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913431::DEBUG::2013-11-20 03:04:54,817::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913431::DEBUG::2013-11-20 03:04:54,819::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913431::DEBUG::2013-11-20 03:04:54,820::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913431::DEBUG::2013-11-20 03:04:54,821::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913431::DEBUG::2013-11-20 03:04:54,822::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913431::DEBUG::2013-11-20 03:04:54,824::task::495::TaskManager.Task::(_debug) Task eb48afc5-5a9e-4c9c-b26f-c32e744d18ae: ref 0 aborting False >Thread-15913431::INFO::2013-11-20 03:04:54,825::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:04:55,563::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:56,091::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0704603 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:04:58,105::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:04:58,613::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0683611 s, 15.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:05:00,623::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:01,168::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0728484 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:05:03,183::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:03,714::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0733808 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15913435::DEBUG::2013-11-20 03:05:05,261::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913435::INFO::2013-11-20 03:05:05,263::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913435::DEBUG::2013-11-20 03:05:05,264::task::495::TaskManager.Task::(_debug) Task 25742598-5de9-4f5e-9273-fa0e7f429bb4: moving from state init -> state preparing >Thread-15913435::DEBUG::2013-11-20 03:05:05,264::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913435::DEBUG::2013-11-20 03:05:05,265::task::495::TaskManager.Task::(_debug) Task 25742598-5de9-4f5e-9273-fa0e7f429bb4: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913435::DEBUG::2013-11-20 03:05:05,266::task::495::TaskManager.Task::(_debug) Task 25742598-5de9-4f5e-9273-fa0e7f429bb4: moving from state preparing -> state finished >Thread-15913435::DEBUG::2013-11-20 03:05:05,266::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913435::DEBUG::2013-11-20 03:05:05,267::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913435::DEBUG::2013-11-20 03:05:05,267::task::495::TaskManager.Task::(_debug) Task 25742598-5de9-4f5e-9273-fa0e7f429bb4: ref 0 aborting False >Thread-15913435::INFO::2013-11-20 03:05:05,268::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913436::DEBUG::2013-11-20 03:05:05,276::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913436::INFO::2013-11-20 03:05:05,277::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913436::DEBUG::2013-11-20 03:05:05,278::task::495::TaskManager.Task::(_debug) Task 30f65d7d-351d-4d95-95f0-5b63eea0573f: moving from state init -> state preparing >Thread-15913436::DEBUG::2013-11-20 03:05:05,279::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`41f62e80-9468-4969-a90f-dc35195e4110`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913436::DEBUG::2013-11-20 03:05:05,280::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913436::DEBUG::2013-11-20 03:05:05,281::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913436::DEBUG::2013-11-20 03:05:05,281::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`41f62e80-9468-4969-a90f-dc35195e4110`::Granted request >Thread-15913436::DEBUG::2013-11-20 03:05:05,282::task::495::TaskManager.Task::(_debug) Task 30f65d7d-351d-4d95-95f0-5b63eea0573f: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913436::DEBUG::2013-11-20 03:05:05,283::task::495::TaskManager.Task::(_debug) Task 30f65d7d-351d-4d95-95f0-5b63eea0573f: ref 1 aborting False >Thread-15913436::WARNING::2013-11-20 03:05:05,285::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913436::DEBUG::2013-11-20 03:05:05,285::task::495::TaskManager.Task::(_debug) Task 30f65d7d-351d-4d95-95f0-5b63eea0573f: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913436::DEBUG::2013-11-20 03:05:05,286::task::495::TaskManager.Task::(_debug) Task 30f65d7d-351d-4d95-95f0-5b63eea0573f: moving from state preparing -> state finished >Thread-15913436::DEBUG::2013-11-20 03:05:05,287::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913436::DEBUG::2013-11-20 03:05:05,287::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913436::DEBUG::2013-11-20 03:05:05,288::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913436::DEBUG::2013-11-20 03:05:05,289::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913436::DEBUG::2013-11-20 03:05:05,289::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913436::DEBUG::2013-11-20 03:05:05,290::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913436::DEBUG::2013-11-20 03:05:05,290::task::495::TaskManager.Task::(_debug) Task 30f65d7d-351d-4d95-95f0-5b63eea0573f: ref 0 aborting False >Thread-15913436::INFO::2013-11-20 03:05:05,291::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:05:05,728::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:06,253::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0768855 s, 13.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:05:08,268::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:08,803::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0751446 s, 13.6 MB/s\n'; <rc> = 0 >Thread-15913438::INFO::2013-11-20 03:05:09,454::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913438::DEBUG::2013-11-20 03:05:09,455::task::495::TaskManager.Task::(_debug) Task bd0bc79d-f044-4498-a5a5-b3c2a7bcfd03: moving from state init -> state preparing >Thread-15913438::DEBUG::2013-11-20 03:05:09,455::task::495::TaskManager.Task::(_debug) Task bd0bc79d-f044-4498-a5a5-b3c2a7bcfd03: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000415802001953', 'lastCheck': 1384916703.865972, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00167083740234', 'lastCheck': 1384916701.332516, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00159215927124', 'lastCheck': 1384916702.3446119, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000380992889404', 'lastCheck': 1384916701.8539801, 'valid': True, 'code': 0}} >Thread-15913438::DEBUG::2013-11-20 03:05:09,456::task::495::TaskManager.Task::(_debug) Task bd0bc79d-f044-4498-a5a5-b3c2a7bcfd03: moving from state preparing -> state finished >Thread-15913438::DEBUG::2013-11-20 03:05:09,457::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913438::DEBUG::2013-11-20 03:05:09,457::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913438::DEBUG::2013-11-20 03:05:09,458::task::495::TaskManager.Task::(_debug) Task bd0bc79d-f044-4498-a5a5-b3c2a7bcfd03: ref 0 aborting False >Thread-15913438::INFO::2013-11-20 03:05:09,458::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000415802001953', 'lastCheck': 1384916703.865972, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00167083740234', 'lastCheck': 1384916701.332516, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00159215927124', 'lastCheck': 1384916702.3446119, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000380992889404', 'lastCheck': 1384916701.8539801, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:05:10,819::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:11,353::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0694019 s, 14.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:05:13,368::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:13,916::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0862804 s, 11.9 MB/s\n'; <rc> = 0 >Thread-15913441::DEBUG::2013-11-20 03:05:15,764::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913441::INFO::2013-11-20 03:05:15,765::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913441::DEBUG::2013-11-20 03:05:15,766::task::495::TaskManager.Task::(_debug) Task eff70ab2-31eb-4eda-9a54-653827fdc925: moving from state init -> state preparing >Thread-15913441::DEBUG::2013-11-20 03:05:15,766::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913441::DEBUG::2013-11-20 03:05:15,767::task::495::TaskManager.Task::(_debug) Task eff70ab2-31eb-4eda-9a54-653827fdc925: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913441::DEBUG::2013-11-20 03:05:15,767::task::495::TaskManager.Task::(_debug) Task eff70ab2-31eb-4eda-9a54-653827fdc925: moving from state preparing -> state finished >Thread-15913441::DEBUG::2013-11-20 03:05:15,768::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913441::DEBUG::2013-11-20 03:05:15,769::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913441::DEBUG::2013-11-20 03:05:15,769::task::495::TaskManager.Task::(_debug) Task eff70ab2-31eb-4eda-9a54-653827fdc925: ref 0 aborting False >Thread-15913441::INFO::2013-11-20 03:05:15,770::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913442::DEBUG::2013-11-20 03:05:15,774::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913442::INFO::2013-11-20 03:05:15,775::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913442::DEBUG::2013-11-20 03:05:15,776::task::495::TaskManager.Task::(_debug) Task b1c09bdf-c14d-408d-aa4f-94c869640ede: moving from state init -> state preparing >Thread-15913442::DEBUG::2013-11-20 03:05:15,777::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a1cfadab-b702-4377-96ff-d12a0d7b1bdf`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913442::DEBUG::2013-11-20 03:05:15,778::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913442::DEBUG::2013-11-20 03:05:15,778::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913442::DEBUG::2013-11-20 03:05:15,779::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a1cfadab-b702-4377-96ff-d12a0d7b1bdf`::Granted request >Thread-15913442::DEBUG::2013-11-20 03:05:15,780::task::495::TaskManager.Task::(_debug) Task b1c09bdf-c14d-408d-aa4f-94c869640ede: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913442::DEBUG::2013-11-20 03:05:15,780::task::495::TaskManager.Task::(_debug) Task b1c09bdf-c14d-408d-aa4f-94c869640ede: ref 1 aborting False >Thread-15913442::WARNING::2013-11-20 03:05:15,782::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913442::DEBUG::2013-11-20 03:05:15,783::task::495::TaskManager.Task::(_debug) Task b1c09bdf-c14d-408d-aa4f-94c869640ede: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913442::DEBUG::2013-11-20 03:05:15,784::task::495::TaskManager.Task::(_debug) Task b1c09bdf-c14d-408d-aa4f-94c869640ede: moving from state preparing -> state finished >Thread-15913442::DEBUG::2013-11-20 03:05:15,785::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913442::DEBUG::2013-11-20 03:05:15,785::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913442::DEBUG::2013-11-20 03:05:15,786::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913442::DEBUG::2013-11-20 03:05:15,786::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913442::DEBUG::2013-11-20 03:05:15,787::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913442::DEBUG::2013-11-20 03:05:15,787::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913442::DEBUG::2013-11-20 03:05:15,788::task::495::TaskManager.Task::(_debug) Task b1c09bdf-c14d-408d-aa4f-94c869640ede: ref 0 aborting False >Thread-15913442::INFO::2013-11-20 03:05:15,789::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:05:15,932::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:16,417::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0710237 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:05:18,427::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:18,964::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0826974 s, 12.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:05:20,980::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:21,500::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.083333 s, 12.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:05:23,516::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 03:05:23,753::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:05:23,755::task::495::TaskManager.Task::(_debug) Task 609b8572-61c6-47f7-ae6b-8bf533c4bf65: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:05:23,764::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`6e821646-d1b4-4659-aca6-62361878de78`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:05:23,765::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:05:23,766::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:05:23,766::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`6e821646-d1b4-4659-aca6-62361878de78`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:05:23,768::task::495::TaskManager.Task::(_debug) Task 609b8572-61c6-47f7-ae6b-8bf533c4bf65: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:05:23,768::task::495::TaskManager.Task::(_debug) Task 609b8572-61c6-47f7-ae6b-8bf533c4bf65: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:05:23,769::task::495::TaskManager.Task::(_debug) Task 609b8572-61c6-47f7-ae6b-8bf533c4bf65: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:05:23,769::task::495::TaskManager.Task::(_debug) Task 609b8572-61c6-47f7-ae6b-8bf533c4bf65: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:05:23,770::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:05:23,771::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:05:23,771::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:05:23,772::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:05:23,772::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:05:23,773::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:05:23,774::task::495::TaskManager.Task::(_debug) Task 609b8572-61c6-47f7-ae6b-8bf533c4bf65: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:05:23,774::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:05:24,097::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100523 s, 10.2 MB/s\n'; <rc> = 0 >Thread-15913446::INFO::2013-11-20 03:05:25,183::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913446::DEBUG::2013-11-20 03:05:25,184::task::495::TaskManager.Task::(_debug) Task 27ead4e8-90b4-464d-ba50-b841c594e44d: moving from state init -> state preparing >Thread-15913446::DEBUG::2013-11-20 03:05:25,184::task::495::TaskManager.Task::(_debug) Task 27ead4e8-90b4-464d-ba50-b841c594e44d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000411033630371', 'lastCheck': 1384916724.1854701, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00159096717834', 'lastCheck': 1384916721.6453769, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00164198875427', 'lastCheck': 1384916722.6457801, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000447034835815', 'lastCheck': 1384916722.178664, 'valid': True, 'code': 0}} >Thread-15913446::DEBUG::2013-11-20 03:05:25,185::task::495::TaskManager.Task::(_debug) Task 27ead4e8-90b4-464d-ba50-b841c594e44d: moving from state preparing -> state finished >Thread-15913446::DEBUG::2013-11-20 03:05:25,186::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913446::DEBUG::2013-11-20 03:05:25,186::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913446::DEBUG::2013-11-20 03:05:25,187::task::495::TaskManager.Task::(_debug) Task 27ead4e8-90b4-464d-ba50-b841c594e44d: ref 0 aborting False >Thread-15913446::INFO::2013-11-20 03:05:25,188::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000411033630371', 'lastCheck': 1384916724.1854701, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00159096717834', 'lastCheck': 1384916721.6453769, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00164198875427', 'lastCheck': 1384916722.6457801, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000447034835815', 'lastCheck': 1384916722.178664, 'valid': True, 'code': 0}} >Thread-15913448::DEBUG::2013-11-20 03:05:25,872::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913448::INFO::2013-11-20 03:05:25,874::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913448::DEBUG::2013-11-20 03:05:25,874::task::495::TaskManager.Task::(_debug) Task 1d4e2abf-7682-47a3-b404-7286efdafda4: moving from state init -> state preparing >Thread-15913448::DEBUG::2013-11-20 03:05:25,875::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913448::DEBUG::2013-11-20 03:05:25,875::task::495::TaskManager.Task::(_debug) Task 1d4e2abf-7682-47a3-b404-7286efdafda4: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913448::DEBUG::2013-11-20 03:05:25,876::task::495::TaskManager.Task::(_debug) Task 1d4e2abf-7682-47a3-b404-7286efdafda4: moving from state preparing -> state finished >Thread-15913448::DEBUG::2013-11-20 03:05:25,877::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913448::DEBUG::2013-11-20 03:05:25,877::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913448::DEBUG::2013-11-20 03:05:25,878::task::495::TaskManager.Task::(_debug) Task 1d4e2abf-7682-47a3-b404-7286efdafda4: ref 0 aborting False >Thread-15913448::INFO::2013-11-20 03:05:25,878::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913449::DEBUG::2013-11-20 03:05:25,883::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913449::INFO::2013-11-20 03:05:25,884::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913449::DEBUG::2013-11-20 03:05:25,884::task::495::TaskManager.Task::(_debug) Task fbaaf9ec-a99e-4a90-8648-ce5addac640d: moving from state init -> state preparing >Thread-15913449::DEBUG::2013-11-20 03:05:25,885::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5c9d9f1b-ab04-4230-b2e1-1e709d5937d8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913449::DEBUG::2013-11-20 03:05:25,886::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913449::DEBUG::2013-11-20 03:05:25,887::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913449::DEBUG::2013-11-20 03:05:25,887::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5c9d9f1b-ab04-4230-b2e1-1e709d5937d8`::Granted request >Thread-15913449::DEBUG::2013-11-20 03:05:25,888::task::495::TaskManager.Task::(_debug) Task fbaaf9ec-a99e-4a90-8648-ce5addac640d: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913449::DEBUG::2013-11-20 03:05:25,889::task::495::TaskManager.Task::(_debug) Task fbaaf9ec-a99e-4a90-8648-ce5addac640d: ref 1 aborting False >Thread-15913449::WARNING::2013-11-20 03:05:25,891::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913449::DEBUG::2013-11-20 03:05:25,891::task::495::TaskManager.Task::(_debug) Task fbaaf9ec-a99e-4a90-8648-ce5addac640d: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913449::DEBUG::2013-11-20 03:05:25,892::task::495::TaskManager.Task::(_debug) Task fbaaf9ec-a99e-4a90-8648-ce5addac640d: moving from state preparing -> state finished >Thread-15913449::DEBUG::2013-11-20 03:05:25,893::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913449::DEBUG::2013-11-20 03:05:25,893::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913449::DEBUG::2013-11-20 03:05:25,894::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913449::DEBUG::2013-11-20 03:05:25,895::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913449::DEBUG::2013-11-20 03:05:25,895::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913449::DEBUG::2013-11-20 03:05:25,896::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913449::DEBUG::2013-11-20 03:05:25,896::task::495::TaskManager.Task::(_debug) Task fbaaf9ec-a99e-4a90-8648-ce5addac640d: ref 0 aborting False >Thread-15913449::INFO::2013-11-20 03:05:25,897::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:05:26,108::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:26,629::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0707103 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:05:28,644::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:29,160::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0546376 s, 18.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:05:31,171::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:31,717::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0749569 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:05:33,732::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:34,231::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0587828 s, 17.4 MB/s\n'; <rc> = 0 >Thread-15913453::DEBUG::2013-11-20 03:05:36,015::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913453::INFO::2013-11-20 03:05:36,017::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913453::DEBUG::2013-11-20 03:05:36,018::task::495::TaskManager.Task::(_debug) Task dba732b3-a64d-4464-8548-1c56c8d6b86a: moving from state init -> state preparing >Thread-15913453::DEBUG::2013-11-20 03:05:36,019::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913453::DEBUG::2013-11-20 03:05:36,019::task::495::TaskManager.Task::(_debug) Task dba732b3-a64d-4464-8548-1c56c8d6b86a: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913453::DEBUG::2013-11-20 03:05:36,020::task::495::TaskManager.Task::(_debug) Task dba732b3-a64d-4464-8548-1c56c8d6b86a: moving from state preparing -> state finished >Thread-15913453::DEBUG::2013-11-20 03:05:36,021::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913453::DEBUG::2013-11-20 03:05:36,021::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913453::DEBUG::2013-11-20 03:05:36,022::task::495::TaskManager.Task::(_debug) Task dba732b3-a64d-4464-8548-1c56c8d6b86a: ref 0 aborting False >Thread-15913453::INFO::2013-11-20 03:05:36,022::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913454::DEBUG::2013-11-20 03:05:36,027::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913454::INFO::2013-11-20 03:05:36,028::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913454::DEBUG::2013-11-20 03:05:36,028::task::495::TaskManager.Task::(_debug) Task 1f330e1d-18a3-41d5-afd5-b4e91f663076: moving from state init -> state preparing >Thread-15913454::DEBUG::2013-11-20 03:05:36,030::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b62667df-4ba9-4d10-a1de-08095b117c52`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913454::DEBUG::2013-11-20 03:05:36,030::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913454::DEBUG::2013-11-20 03:05:36,031::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913454::DEBUG::2013-11-20 03:05:36,032::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b62667df-4ba9-4d10-a1de-08095b117c52`::Granted request >Thread-15913454::DEBUG::2013-11-20 03:05:36,033::task::495::TaskManager.Task::(_debug) Task 1f330e1d-18a3-41d5-afd5-b4e91f663076: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913454::DEBUG::2013-11-20 03:05:36,033::task::495::TaskManager.Task::(_debug) Task 1f330e1d-18a3-41d5-afd5-b4e91f663076: ref 1 aborting False >Thread-15913454::WARNING::2013-11-20 03:05:36,035::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913454::DEBUG::2013-11-20 03:05:36,036::task::495::TaskManager.Task::(_debug) Task 1f330e1d-18a3-41d5-afd5-b4e91f663076: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913454::DEBUG::2013-11-20 03:05:36,037::task::495::TaskManager.Task::(_debug) Task 1f330e1d-18a3-41d5-afd5-b4e91f663076: moving from state preparing -> state finished >Thread-15913454::DEBUG::2013-11-20 03:05:36,037::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913454::DEBUG::2013-11-20 03:05:36,038::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913454::DEBUG::2013-11-20 03:05:36,039::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913454::DEBUG::2013-11-20 03:05:36,039::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913454::DEBUG::2013-11-20 03:05:36,040::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913454::DEBUG::2013-11-20 03:05:36,040::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913454::DEBUG::2013-11-20 03:05:36,041::task::495::TaskManager.Task::(_debug) Task 1f330e1d-18a3-41d5-afd5-b4e91f663076: ref 0 aborting False >Thread-15913454::INFO::2013-11-20 03:05:36,041::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:05:36,246::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:36,818::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0825946 s, 12.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:05:38,833::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:39,382::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0769691 s, 13.3 MB/s\n'; <rc> = 0 >Thread-15913456::INFO::2013-11-20 03:05:40,485::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913456::DEBUG::2013-11-20 03:05:40,486::task::495::TaskManager.Task::(_debug) Task 26575f63-fa9d-475a-8094-a799ebae4d65: moving from state init -> state preparing >Thread-15913456::DEBUG::2013-11-20 03:05:40,487::task::495::TaskManager.Task::(_debug) Task 26575f63-fa9d-475a-8094-a799ebae4d65: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000312805175781', 'lastCheck': 1384916734.4314971, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00180983543396', 'lastCheck': 1384916731.899389, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00146317481995', 'lastCheck': 1384916732.8943069, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000438213348389', 'lastCheck': 1384916732.404979, 'valid': True, 'code': 0}} >Thread-15913456::DEBUG::2013-11-20 03:05:40,487::task::495::TaskManager.Task::(_debug) Task 26575f63-fa9d-475a-8094-a799ebae4d65: moving from state preparing -> state finished >Thread-15913456::DEBUG::2013-11-20 03:05:40,488::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913456::DEBUG::2013-11-20 03:05:40,489::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913456::DEBUG::2013-11-20 03:05:40,489::task::495::TaskManager.Task::(_debug) Task 26575f63-fa9d-475a-8094-a799ebae4d65: ref 0 aborting False >Thread-15913456::INFO::2013-11-20 03:05:40,490::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000312805175781', 'lastCheck': 1384916734.4314971, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00180983543396', 'lastCheck': 1384916731.899389, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00146317481995', 'lastCheck': 1384916732.8943069, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000438213348389', 'lastCheck': 1384916732.404979, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:05:41,397::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:41,968::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0792528 s, 12.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:05:43,980::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:44,597::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0915021 s, 11.2 MB/s\n'; <rc> = 0 >Thread-15913459::DEBUG::2013-11-20 03:05:46,225::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913459::INFO::2013-11-20 03:05:46,226::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913459::DEBUG::2013-11-20 03:05:46,227::task::495::TaskManager.Task::(_debug) Task 7c168868-a407-4a5b-88d5-52196e2cf631: moving from state init -> state preparing >Thread-15913459::DEBUG::2013-11-20 03:05:46,228::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913459::DEBUG::2013-11-20 03:05:46,228::task::495::TaskManager.Task::(_debug) Task 7c168868-a407-4a5b-88d5-52196e2cf631: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913459::DEBUG::2013-11-20 03:05:46,229::task::495::TaskManager.Task::(_debug) Task 7c168868-a407-4a5b-88d5-52196e2cf631: moving from state preparing -> state finished >Thread-15913459::DEBUG::2013-11-20 03:05:46,229::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913459::DEBUG::2013-11-20 03:05:46,230::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913459::DEBUG::2013-11-20 03:05:46,230::task::495::TaskManager.Task::(_debug) Task 7c168868-a407-4a5b-88d5-52196e2cf631: ref 0 aborting False >Thread-15913459::INFO::2013-11-20 03:05:46,231::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913460::DEBUG::2013-11-20 03:05:46,236::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913460::INFO::2013-11-20 03:05:46,236::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913460::DEBUG::2013-11-20 03:05:46,237::task::495::TaskManager.Task::(_debug) Task 2128abb8-1c96-4e5a-84be-8e39cf093eaa: moving from state init -> state preparing >Thread-15913460::DEBUG::2013-11-20 03:05:46,238::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`83f77196-2a55-46c9-9b31-eed1ce997888`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913460::DEBUG::2013-11-20 03:05:46,239::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913460::DEBUG::2013-11-20 03:05:46,240::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913460::DEBUG::2013-11-20 03:05:46,240::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`83f77196-2a55-46c9-9b31-eed1ce997888`::Granted request >Thread-15913460::DEBUG::2013-11-20 03:05:46,241::task::495::TaskManager.Task::(_debug) Task 2128abb8-1c96-4e5a-84be-8e39cf093eaa: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913460::DEBUG::2013-11-20 03:05:46,242::task::495::TaskManager.Task::(_debug) Task 2128abb8-1c96-4e5a-84be-8e39cf093eaa: ref 1 aborting False >Thread-15913460::WARNING::2013-11-20 03:05:46,244::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913460::DEBUG::2013-11-20 03:05:46,244::task::495::TaskManager.Task::(_debug) Task 2128abb8-1c96-4e5a-84be-8e39cf093eaa: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913460::DEBUG::2013-11-20 03:05:46,245::task::495::TaskManager.Task::(_debug) Task 2128abb8-1c96-4e5a-84be-8e39cf093eaa: moving from state preparing -> state finished >Thread-15913460::DEBUG::2013-11-20 03:05:46,246::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913460::DEBUG::2013-11-20 03:05:46,246::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913460::DEBUG::2013-11-20 03:05:46,247::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913460::DEBUG::2013-11-20 03:05:46,248::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913460::DEBUG::2013-11-20 03:05:46,248::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913460::DEBUG::2013-11-20 03:05:46,249::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913460::DEBUG::2013-11-20 03:05:46,249::task::495::TaskManager.Task::(_debug) Task 2128abb8-1c96-4e5a-84be-8e39cf093eaa: ref 0 aborting False >Thread-15913460::INFO::2013-11-20 03:05:46,250::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:05:46,613::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:47,094::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.057228 s, 17.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:05:49,108::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:49,622::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0775906 s, 13.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:05:51,637::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:52,171::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0706431 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:05:54,186::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:54,718::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0757148 s, 13.5 MB/s\n'; <rc> = 0 >Thread-15913464::INFO::2013-11-20 03:05:55,916::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913464::DEBUG::2013-11-20 03:05:55,917::task::495::TaskManager.Task::(_debug) Task 2113437d-98d1-42a7-9bd1-bc3377c93ab4: moving from state init -> state preparing >Thread-15913464::DEBUG::2013-11-20 03:05:55,918::task::495::TaskManager.Task::(_debug) Task 2113437d-98d1-42a7-9bd1-bc3377c93ab4: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000388860702515', 'lastCheck': 1384916754.8622799, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00149893760681', 'lastCheck': 1384916752.3143871, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0122559070587', 'lastCheck': 1384916753.325098, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000472068786621', 'lastCheck': 1384916752.8574569, 'valid': True, 'code': 0}} >Thread-15913464::DEBUG::2013-11-20 03:05:55,919::task::495::TaskManager.Task::(_debug) Task 2113437d-98d1-42a7-9bd1-bc3377c93ab4: moving from state preparing -> state finished >Thread-15913464::DEBUG::2013-11-20 03:05:55,919::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913464::DEBUG::2013-11-20 03:05:55,920::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913464::DEBUG::2013-11-20 03:05:55,921::task::495::TaskManager.Task::(_debug) Task 2113437d-98d1-42a7-9bd1-bc3377c93ab4: ref 0 aborting False >Thread-15913464::INFO::2013-11-20 03:05:55,922::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000388860702515', 'lastCheck': 1384916754.8622799, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00149893760681', 'lastCheck': 1384916752.3143871, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0122559070587', 'lastCheck': 1384916753.325098, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000472068786621', 'lastCheck': 1384916752.8574569, 'valid': True, 'code': 0}} >Thread-15913466::DEBUG::2013-11-20 03:05:56,699::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913466::INFO::2013-11-20 03:05:56,700::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913466::DEBUG::2013-11-20 03:05:56,709::task::495::TaskManager.Task::(_debug) Task 6d17db36-b18e-47ca-8e08-e2aba1c8d71e: moving from state init -> state preparing >Thread-15913466::DEBUG::2013-11-20 03:05:56,710::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913466::DEBUG::2013-11-20 03:05:56,711::task::495::TaskManager.Task::(_debug) Task 6d17db36-b18e-47ca-8e08-e2aba1c8d71e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913466::DEBUG::2013-11-20 03:05:56,712::task::495::TaskManager.Task::(_debug) Task 6d17db36-b18e-47ca-8e08-e2aba1c8d71e: moving from state preparing -> state finished >Thread-15913466::DEBUG::2013-11-20 03:05:56,713::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913466::DEBUG::2013-11-20 03:05:56,714::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913466::DEBUG::2013-11-20 03:05:56,715::task::495::TaskManager.Task::(_debug) Task 6d17db36-b18e-47ca-8e08-e2aba1c8d71e: ref 0 aborting False >Thread-15913466::INFO::2013-11-20 03:05:56,716::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913467::DEBUG::2013-11-20 03:05:56,722::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913467::INFO::2013-11-20 03:05:56,724::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913467::DEBUG::2013-11-20 03:05:56,724::task::495::TaskManager.Task::(_debug) Task 1e4e6b5a-6898-493f-870b-c17d14ce3b57: moving from state init -> state preparing >Thread-15913467::DEBUG::2013-11-20 03:05:56,726::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`dc258d50-fa5c-40a3-8ef4-f86edd11b622`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913467::DEBUG::2013-11-20 03:05:56,727::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913467::DEBUG::2013-11-20 03:05:56,729::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913467::DEBUG::2013-11-20 03:05:56,729::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`dc258d50-fa5c-40a3-8ef4-f86edd11b622`::Granted request >Dummy-15912651::DEBUG::2013-11-20 03:05:56,738::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913467::DEBUG::2013-11-20 03:05:56,972::task::495::TaskManager.Task::(_debug) Task 1e4e6b5a-6898-493f-870b-c17d14ce3b57: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913467::DEBUG::2013-11-20 03:05:56,974::task::495::TaskManager.Task::(_debug) Task 1e4e6b5a-6898-493f-870b-c17d14ce3b57: ref 1 aborting False >Thread-15913467::WARNING::2013-11-20 03:05:56,976::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913467::DEBUG::2013-11-20 03:05:56,977::task::495::TaskManager.Task::(_debug) Task 1e4e6b5a-6898-493f-870b-c17d14ce3b57: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913467::DEBUG::2013-11-20 03:05:56,977::task::495::TaskManager.Task::(_debug) Task 1e4e6b5a-6898-493f-870b-c17d14ce3b57: moving from state preparing -> state finished >Thread-15913467::DEBUG::2013-11-20 03:05:56,978::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913467::DEBUG::2013-11-20 03:05:56,978::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913467::DEBUG::2013-11-20 03:05:56,979::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913467::DEBUG::2013-11-20 03:05:56,979::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913467::DEBUG::2013-11-20 03:05:56,979::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913467::DEBUG::2013-11-20 03:05:56,980::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913467::DEBUG::2013-11-20 03:05:56,980::task::495::TaskManager.Task::(_debug) Task 1e4e6b5a-6898-493f-870b-c17d14ce3b57: ref 0 aborting False >Thread-15913467::INFO::2013-11-20 03:05:56,981::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:05:57,267::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0735358 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:05:59,282::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:05:59,823::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.074823 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:06:01,838::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:02,375::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0777546 s, 13.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:06:04,390::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:05,017::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0836346 s, 12.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:06:07,037::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:07,583::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0777265 s, 13.2 MB/s\n'; <rc> = 0 >Thread-15913471::DEBUG::2013-11-20 03:06:07,766::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913471::INFO::2013-11-20 03:06:07,768::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913471::DEBUG::2013-11-20 03:06:07,769::task::495::TaskManager.Task::(_debug) Task dce53b82-ab73-4cc9-9e84-b5ba207e957b: moving from state init -> state preparing >Thread-15913471::DEBUG::2013-11-20 03:06:07,770::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913471::DEBUG::2013-11-20 03:06:07,770::task::495::TaskManager.Task::(_debug) Task dce53b82-ab73-4cc9-9e84-b5ba207e957b: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913471::DEBUG::2013-11-20 03:06:07,771::task::495::TaskManager.Task::(_debug) Task dce53b82-ab73-4cc9-9e84-b5ba207e957b: moving from state preparing -> state finished >Thread-15913471::DEBUG::2013-11-20 03:06:07,771::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913471::DEBUG::2013-11-20 03:06:07,772::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913471::DEBUG::2013-11-20 03:06:07,772::task::495::TaskManager.Task::(_debug) Task dce53b82-ab73-4cc9-9e84-b5ba207e957b: ref 0 aborting False >Thread-15913471::INFO::2013-11-20 03:06:07,773::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913472::DEBUG::2013-11-20 03:06:07,777::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913472::INFO::2013-11-20 03:06:07,778::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913472::DEBUG::2013-11-20 03:06:07,779::task::495::TaskManager.Task::(_debug) Task b9c79cd2-a4d0-4a2f-9111-aa9179e550d0: moving from state init -> state preparing >Thread-15913472::DEBUG::2013-11-20 03:06:07,780::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`dfb8855a-5c03-45f6-885f-36c335a1ea6e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913472::DEBUG::2013-11-20 03:06:07,781::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913472::DEBUG::2013-11-20 03:06:07,782::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913472::DEBUG::2013-11-20 03:06:07,782::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`dfb8855a-5c03-45f6-885f-36c335a1ea6e`::Granted request >Thread-15913472::DEBUG::2013-11-20 03:06:07,783::task::495::TaskManager.Task::(_debug) Task b9c79cd2-a4d0-4a2f-9111-aa9179e550d0: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913472::DEBUG::2013-11-20 03:06:07,784::task::495::TaskManager.Task::(_debug) Task b9c79cd2-a4d0-4a2f-9111-aa9179e550d0: ref 1 aborting False >Thread-15913472::WARNING::2013-11-20 03:06:07,786::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913472::DEBUG::2013-11-20 03:06:07,786::task::495::TaskManager.Task::(_debug) Task b9c79cd2-a4d0-4a2f-9111-aa9179e550d0: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913472::DEBUG::2013-11-20 03:06:07,787::task::495::TaskManager.Task::(_debug) Task b9c79cd2-a4d0-4a2f-9111-aa9179e550d0: moving from state preparing -> state finished >Thread-15913472::DEBUG::2013-11-20 03:06:07,788::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913472::DEBUG::2013-11-20 03:06:07,788::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913472::DEBUG::2013-11-20 03:06:07,789::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913472::DEBUG::2013-11-20 03:06:07,790::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913472::DEBUG::2013-11-20 03:06:07,790::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913472::DEBUG::2013-11-20 03:06:07,791::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913472::DEBUG::2013-11-20 03:06:07,791::task::495::TaskManager.Task::(_debug) Task b9c79cd2-a4d0-4a2f-9111-aa9179e550d0: ref 0 aborting False >Thread-15913472::INFO::2013-11-20 03:06:07,792::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:06:09,599::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:10,090::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0757982 s, 13.5 MB/s\n'; <rc> = 0 >Thread-15913474::INFO::2013-11-20 03:06:11,332::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913474::DEBUG::2013-11-20 03:06:11,333::task::495::TaskManager.Task::(_debug) Task 64a28809-dd82-4689-98f9-9a51da88fe19: moving from state init -> state preparing >Thread-15913474::DEBUG::2013-11-20 03:06:11,334::task::495::TaskManager.Task::(_debug) Task 64a28809-dd82-4689-98f9-9a51da88fe19: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000549077987671', 'lastCheck': 1384916765.0667751, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00141906738281', 'lastCheck': 1384916762.5094309, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00165200233459', 'lastCheck': 1384916763.5119669, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000478982925415', 'lastCheck': 1384916763.0655949, 'valid': True, 'code': 0}} >Thread-15913474::DEBUG::2013-11-20 03:06:11,335::task::495::TaskManager.Task::(_debug) Task 64a28809-dd82-4689-98f9-9a51da88fe19: moving from state preparing -> state finished >Thread-15913474::DEBUG::2013-11-20 03:06:11,335::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913474::DEBUG::2013-11-20 03:06:11,336::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913474::DEBUG::2013-11-20 03:06:11,337::task::495::TaskManager.Task::(_debug) Task 64a28809-dd82-4689-98f9-9a51da88fe19: ref 0 aborting False >Thread-15913474::INFO::2013-11-20 03:06:11,337::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000549077987671', 'lastCheck': 1384916765.0667751, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00141906738281', 'lastCheck': 1384916762.5094309, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00165200233459', 'lastCheck': 1384916763.5119669, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000478982925415', 'lastCheck': 1384916763.0655949, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:06:12,103::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:12,633::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0702525 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:06:14,649::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:15,214::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0991997 s, 10.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:06:17,229::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:17,742::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0732949 s, 14.0 MB/s\n'; <rc> = 0 >Thread-15913478::DEBUG::2013-11-20 03:06:18,004::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913478::INFO::2013-11-20 03:06:18,006::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913478::DEBUG::2013-11-20 03:06:18,007::task::495::TaskManager.Task::(_debug) Task a18f6b27-2be2-4200-bac8-36792255652b: moving from state init -> state preparing >Thread-15913478::DEBUG::2013-11-20 03:06:18,008::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913478::DEBUG::2013-11-20 03:06:18,008::task::495::TaskManager.Task::(_debug) Task a18f6b27-2be2-4200-bac8-36792255652b: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913478::DEBUG::2013-11-20 03:06:18,009::task::495::TaskManager.Task::(_debug) Task a18f6b27-2be2-4200-bac8-36792255652b: moving from state preparing -> state finished >Thread-15913478::DEBUG::2013-11-20 03:06:18,010::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913478::DEBUG::2013-11-20 03:06:18,010::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913478::DEBUG::2013-11-20 03:06:18,011::task::495::TaskManager.Task::(_debug) Task a18f6b27-2be2-4200-bac8-36792255652b: ref 0 aborting False >Thread-15913478::INFO::2013-11-20 03:06:18,011::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913479::DEBUG::2013-11-20 03:06:18,016::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913479::INFO::2013-11-20 03:06:18,017::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913479::DEBUG::2013-11-20 03:06:18,017::task::495::TaskManager.Task::(_debug) Task 2793af67-ee53-460a-b38e-9d75ed2a1c77: moving from state init -> state preparing >Thread-15913479::DEBUG::2013-11-20 03:06:18,018::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1147906c-7da3-487f-b300-762cc336579c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913479::DEBUG::2013-11-20 03:06:18,019::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913479::DEBUG::2013-11-20 03:06:18,020::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913479::DEBUG::2013-11-20 03:06:18,021::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1147906c-7da3-487f-b300-762cc336579c`::Granted request >Thread-15913479::DEBUG::2013-11-20 03:06:18,022::task::495::TaskManager.Task::(_debug) Task 2793af67-ee53-460a-b38e-9d75ed2a1c77: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913479::DEBUG::2013-11-20 03:06:18,022::task::495::TaskManager.Task::(_debug) Task 2793af67-ee53-460a-b38e-9d75ed2a1c77: ref 1 aborting False >Thread-15913479::WARNING::2013-11-20 03:06:18,024::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913479::DEBUG::2013-11-20 03:06:18,025::task::495::TaskManager.Task::(_debug) Task 2793af67-ee53-460a-b38e-9d75ed2a1c77: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913479::DEBUG::2013-11-20 03:06:18,026::task::495::TaskManager.Task::(_debug) Task 2793af67-ee53-460a-b38e-9d75ed2a1c77: moving from state preparing -> state finished >Thread-15913479::DEBUG::2013-11-20 03:06:18,027::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913479::DEBUG::2013-11-20 03:06:18,027::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913479::DEBUG::2013-11-20 03:06:18,028::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913479::DEBUG::2013-11-20 03:06:18,028::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913479::DEBUG::2013-11-20 03:06:18,029::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913479::DEBUG::2013-11-20 03:06:18,029::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913479::DEBUG::2013-11-20 03:06:18,030::task::495::TaskManager.Task::(_debug) Task 2793af67-ee53-460a-b38e-9d75ed2a1c77: ref 0 aborting False >Thread-15913479::INFO::2013-11-20 03:06:18,031::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:06:19,757::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:20,304::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0760459 s, 13.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:06:22,319::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:22,833::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0575861 s, 17.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:06:24,849::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 03:06:25,068::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:06:25,074::task::495::TaskManager.Task::(_debug) Task 583eefa9-089e-4113-b37f-3c2ada74e4e8: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:06:25,079::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`763b29cc-e72e-40bf-b96e-7c8c7c84e112`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:06:25,080::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:06:25,081::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:06:25,082::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`763b29cc-e72e-40bf-b96e-7c8c7c84e112`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:06:25,084::task::495::TaskManager.Task::(_debug) Task 583eefa9-089e-4113-b37f-3c2ada74e4e8: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:06:25,085::task::495::TaskManager.Task::(_debug) Task 583eefa9-089e-4113-b37f-3c2ada74e4e8: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:06:25,086::task::495::TaskManager.Task::(_debug) Task 583eefa9-089e-4113-b37f-3c2ada74e4e8: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:06:25,087::task::495::TaskManager.Task::(_debug) Task 583eefa9-089e-4113-b37f-3c2ada74e4e8: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:06:25,088::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:06:25,089::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:06:25,090::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:06:25,091::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:06:25,092::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:06:25,093::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:06:25,094::task::495::TaskManager.Task::(_debug) Task 583eefa9-089e-4113-b37f-3c2ada74e4e8: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:06:25,095::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:06:25,371::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0740639 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15913482::INFO::2013-11-20 03:06:26,675::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913482::DEBUG::2013-11-20 03:06:26,676::task::495::TaskManager.Task::(_debug) Task e584abd1-924a-463b-a05f-75a388b5e657: moving from state init -> state preparing >Thread-15913482::DEBUG::2013-11-20 03:06:26,677::task::495::TaskManager.Task::(_debug) Task e584abd1-924a-463b-a05f-75a388b5e657: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000367879867554', 'lastCheck': 1384916785.5345809, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0115990638733', 'lastCheck': 1384916783.0018809, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00162100791931', 'lastCheck': 1384916783.992455, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000425100326538', 'lastCheck': 1384916783.5292339, 'valid': True, 'code': 0}} >Thread-15913482::DEBUG::2013-11-20 03:06:26,678::task::495::TaskManager.Task::(_debug) Task e584abd1-924a-463b-a05f-75a388b5e657: moving from state preparing -> state finished >Thread-15913482::DEBUG::2013-11-20 03:06:26,678::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913482::DEBUG::2013-11-20 03:06:26,679::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913482::DEBUG::2013-11-20 03:06:26,679::task::495::TaskManager.Task::(_debug) Task e584abd1-924a-463b-a05f-75a388b5e657: ref 0 aborting False >Thread-15913482::INFO::2013-11-20 03:06:26,680::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000367879867554', 'lastCheck': 1384916785.5345809, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0115990638733', 'lastCheck': 1384916783.0018809, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00162100791931', 'lastCheck': 1384916783.992455, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000425100326538', 'lastCheck': 1384916783.5292339, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:06:27,386::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:27,948::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0788499 s, 13.0 MB/s\n'; <rc> = 0 >Thread-15913484::DEBUG::2013-11-20 03:06:28,404::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913484::INFO::2013-11-20 03:06:28,405::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913484::DEBUG::2013-11-20 03:06:28,406::task::495::TaskManager.Task::(_debug) Task c1d6c3f8-9b6b-4305-af90-75284367f21b: moving from state init -> state preparing >Thread-15913484::DEBUG::2013-11-20 03:06:28,407::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913484::DEBUG::2013-11-20 03:06:28,407::task::495::TaskManager.Task::(_debug) Task c1d6c3f8-9b6b-4305-af90-75284367f21b: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913484::DEBUG::2013-11-20 03:06:28,408::task::495::TaskManager.Task::(_debug) Task c1d6c3f8-9b6b-4305-af90-75284367f21b: moving from state preparing -> state finished >Thread-15913484::DEBUG::2013-11-20 03:06:28,409::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913484::DEBUG::2013-11-20 03:06:28,409::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913484::DEBUG::2013-11-20 03:06:28,410::task::495::TaskManager.Task::(_debug) Task c1d6c3f8-9b6b-4305-af90-75284367f21b: ref 0 aborting False >Thread-15913484::INFO::2013-11-20 03:06:28,410::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913485::DEBUG::2013-11-20 03:06:28,415::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913485::INFO::2013-11-20 03:06:28,416::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913485::DEBUG::2013-11-20 03:06:28,416::task::495::TaskManager.Task::(_debug) Task f521ebd0-230d-43c2-9f58-d590e2806b21: moving from state init -> state preparing >Thread-15913485::DEBUG::2013-11-20 03:06:28,418::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`80f19f4c-aecc-43db-a8c4-3da07b43743c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913485::DEBUG::2013-11-20 03:06:28,418::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913485::DEBUG::2013-11-20 03:06:28,419::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913485::DEBUG::2013-11-20 03:06:28,420::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`80f19f4c-aecc-43db-a8c4-3da07b43743c`::Granted request >Thread-15913485::DEBUG::2013-11-20 03:06:28,421::task::495::TaskManager.Task::(_debug) Task f521ebd0-230d-43c2-9f58-d590e2806b21: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913485::DEBUG::2013-11-20 03:06:28,421::task::495::TaskManager.Task::(_debug) Task f521ebd0-230d-43c2-9f58-d590e2806b21: ref 1 aborting False >Thread-15913485::WARNING::2013-11-20 03:06:28,423::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913485::DEBUG::2013-11-20 03:06:28,424::task::495::TaskManager.Task::(_debug) Task f521ebd0-230d-43c2-9f58-d590e2806b21: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913485::DEBUG::2013-11-20 03:06:28,425::task::495::TaskManager.Task::(_debug) Task f521ebd0-230d-43c2-9f58-d590e2806b21: moving from state preparing -> state finished >Thread-15913485::DEBUG::2013-11-20 03:06:28,425::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913485::DEBUG::2013-11-20 03:06:28,426::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913485::DEBUG::2013-11-20 03:06:28,427::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913485::DEBUG::2013-11-20 03:06:28,427::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913485::DEBUG::2013-11-20 03:06:28,428::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913485::DEBUG::2013-11-20 03:06:28,428::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913485::DEBUG::2013-11-20 03:06:28,429::task::495::TaskManager.Task::(_debug) Task f521ebd0-230d-43c2-9f58-d590e2806b21: ref 0 aborting False >Thread-15913485::INFO::2013-11-20 03:06:28,430::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:06:29,964::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:30,516::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0727677 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:06:32,532::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:33,016::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0501847 s, 20.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:06:35,031::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:35,607::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0729221 s, 14.0 MB/s\n'; <rc> = 0 >Thread-42838::DEBUG::2013-11-20 03:06:35,753::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 03:06:36,488::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:06:37,622::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:38,233::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0749718 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15913489::DEBUG::2013-11-20 03:06:38,658::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913489::INFO::2013-11-20 03:06:38,659::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913489::DEBUG::2013-11-20 03:06:38,660::task::495::TaskManager.Task::(_debug) Task 82117e64-9f65-4b79-a3a1-b3aaba459cd4: moving from state init -> state preparing >Thread-15913489::DEBUG::2013-11-20 03:06:38,661::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913489::DEBUG::2013-11-20 03:06:38,661::task::495::TaskManager.Task::(_debug) Task 82117e64-9f65-4b79-a3a1-b3aaba459cd4: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913489::DEBUG::2013-11-20 03:06:38,662::task::495::TaskManager.Task::(_debug) Task 82117e64-9f65-4b79-a3a1-b3aaba459cd4: moving from state preparing -> state finished >Thread-15913489::DEBUG::2013-11-20 03:06:38,662::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913489::DEBUG::2013-11-20 03:06:38,663::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913489::DEBUG::2013-11-20 03:06:38,664::task::495::TaskManager.Task::(_debug) Task 82117e64-9f65-4b79-a3a1-b3aaba459cd4: ref 0 aborting False >Thread-15913489::INFO::2013-11-20 03:06:38,664::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913490::DEBUG::2013-11-20 03:06:38,669::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913490::INFO::2013-11-20 03:06:38,670::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913490::DEBUG::2013-11-20 03:06:38,670::task::495::TaskManager.Task::(_debug) Task 6ca435c9-3a52-4eb9-b204-7cd44cfc8919: moving from state init -> state preparing >Thread-15913490::DEBUG::2013-11-20 03:06:38,671::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d3ddac3c-9145-4b84-b727-c5f3279a108b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913490::DEBUG::2013-11-20 03:06:38,672::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913490::DEBUG::2013-11-20 03:06:38,673::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913490::DEBUG::2013-11-20 03:06:38,673::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d3ddac3c-9145-4b84-b727-c5f3279a108b`::Granted request >Thread-15913490::DEBUG::2013-11-20 03:06:38,674::task::495::TaskManager.Task::(_debug) Task 6ca435c9-3a52-4eb9-b204-7cd44cfc8919: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913490::DEBUG::2013-11-20 03:06:38,675::task::495::TaskManager.Task::(_debug) Task 6ca435c9-3a52-4eb9-b204-7cd44cfc8919: ref 1 aborting False >Thread-15913490::WARNING::2013-11-20 03:06:38,677::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913490::DEBUG::2013-11-20 03:06:38,678::task::495::TaskManager.Task::(_debug) Task 6ca435c9-3a52-4eb9-b204-7cd44cfc8919: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913490::DEBUG::2013-11-20 03:06:38,679::task::495::TaskManager.Task::(_debug) Task 6ca435c9-3a52-4eb9-b204-7cd44cfc8919: moving from state preparing -> state finished >Thread-15913490::DEBUG::2013-11-20 03:06:38,679::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913490::DEBUG::2013-11-20 03:06:38,680::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913490::DEBUG::2013-11-20 03:06:38,680::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913490::DEBUG::2013-11-20 03:06:38,681::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913490::DEBUG::2013-11-20 03:06:38,682::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913490::DEBUG::2013-11-20 03:06:38,682::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913490::DEBUG::2013-11-20 03:06:38,683::task::495::TaskManager.Task::(_debug) Task 6ca435c9-3a52-4eb9-b204-7cd44cfc8919: ref 0 aborting False >Thread-15913490::INFO::2013-11-20 03:06:38,683::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:06:40,249::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:40,836::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0747142 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15913492::INFO::2013-11-20 03:06:42,325::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913492::DEBUG::2013-11-20 03:06:42,326::task::495::TaskManager.Task::(_debug) Task 3e6b0eb2-f5a9-4c17-bc33-b94206b73538: moving from state init -> state preparing >Thread-15913492::DEBUG::2013-11-20 03:06:42,327::task::495::TaskManager.Task::(_debug) Task 3e6b0eb2-f5a9-4c17-bc33-b94206b73538: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000316143035889', 'lastCheck': 1384916796.493331, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00192093849182', 'lastCheck': 1384916793.219619, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00172996520996', 'lastCheck': 1384916794.232214, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000494003295898', 'lastCheck': 1384916793.7564139, 'valid': True, 'code': 0}} >Thread-15913492::DEBUG::2013-11-20 03:06:42,328::task::495::TaskManager.Task::(_debug) Task 3e6b0eb2-f5a9-4c17-bc33-b94206b73538: moving from state preparing -> state finished >Thread-15913492::DEBUG::2013-11-20 03:06:42,328::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913492::DEBUG::2013-11-20 03:06:42,329::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913492::DEBUG::2013-11-20 03:06:42,330::task::495::TaskManager.Task::(_debug) Task 3e6b0eb2-f5a9-4c17-bc33-b94206b73538: ref 0 aborting False >Thread-15913492::INFO::2013-11-20 03:06:42,330::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000316143035889', 'lastCheck': 1384916796.493331, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00192093849182', 'lastCheck': 1384916793.219619, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00172996520996', 'lastCheck': 1384916794.232214, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000494003295898', 'lastCheck': 1384916793.7564139, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:06:42,851::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:43,379::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0723255 s, 14.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:06:45,394::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:45,926::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.070443 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:06:47,941::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:48,478::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.075858 s, 13.5 MB/s\n'; <rc> = 0 >Thread-15913496::DEBUG::2013-11-20 03:06:49,326::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913496::INFO::2013-11-20 03:06:49,328::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913496::DEBUG::2013-11-20 03:06:49,328::task::495::TaskManager.Task::(_debug) Task 8e5f84c5-76ac-46e2-ab79-21c6fe96b547: moving from state init -> state preparing >Thread-15913496::DEBUG::2013-11-20 03:06:49,329::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913496::DEBUG::2013-11-20 03:06:49,330::task::495::TaskManager.Task::(_debug) Task 8e5f84c5-76ac-46e2-ab79-21c6fe96b547: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913496::DEBUG::2013-11-20 03:06:49,330::task::495::TaskManager.Task::(_debug) Task 8e5f84c5-76ac-46e2-ab79-21c6fe96b547: moving from state preparing -> state finished >Thread-15913496::DEBUG::2013-11-20 03:06:49,331::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913496::DEBUG::2013-11-20 03:06:49,331::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913496::DEBUG::2013-11-20 03:06:49,332::task::495::TaskManager.Task::(_debug) Task 8e5f84c5-76ac-46e2-ab79-21c6fe96b547: ref 0 aborting False >Thread-15913496::INFO::2013-11-20 03:06:49,333::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913497::DEBUG::2013-11-20 03:06:49,337::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913497::INFO::2013-11-20 03:06:49,338::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913497::DEBUG::2013-11-20 03:06:49,339::task::495::TaskManager.Task::(_debug) Task be423a1a-68d8-4323-b00b-5ed4ddc8adcb: moving from state init -> state preparing >Thread-15913497::DEBUG::2013-11-20 03:06:49,340::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6d30360c-3f6a-44bb-809e-ac266a8bc432`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913497::DEBUG::2013-11-20 03:06:49,340::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913497::DEBUG::2013-11-20 03:06:49,341::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913497::DEBUG::2013-11-20 03:06:49,342::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6d30360c-3f6a-44bb-809e-ac266a8bc432`::Granted request >Thread-15913497::DEBUG::2013-11-20 03:06:49,343::task::495::TaskManager.Task::(_debug) Task be423a1a-68d8-4323-b00b-5ed4ddc8adcb: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913497::DEBUG::2013-11-20 03:06:49,343::task::495::TaskManager.Task::(_debug) Task be423a1a-68d8-4323-b00b-5ed4ddc8adcb: ref 1 aborting False >Thread-15913497::WARNING::2013-11-20 03:06:49,346::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913497::DEBUG::2013-11-20 03:06:49,347::task::495::TaskManager.Task::(_debug) Task be423a1a-68d8-4323-b00b-5ed4ddc8adcb: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913497::DEBUG::2013-11-20 03:06:49,348::task::495::TaskManager.Task::(_debug) Task be423a1a-68d8-4323-b00b-5ed4ddc8adcb: moving from state preparing -> state finished >Thread-15913497::DEBUG::2013-11-20 03:06:49,349::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913497::DEBUG::2013-11-20 03:06:49,350::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913497::DEBUG::2013-11-20 03:06:49,351::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913497::DEBUG::2013-11-20 03:06:49,352::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913497::DEBUG::2013-11-20 03:06:49,352::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913497::DEBUG::2013-11-20 03:06:49,353::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913497::DEBUG::2013-11-20 03:06:49,354::task::495::TaskManager.Task::(_debug) Task be423a1a-68d8-4323-b00b-5ed4ddc8adcb: ref 0 aborting False >Thread-15913497::INFO::2013-11-20 03:06:49,355::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:06:50,492::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:51,030::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0739068 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:06:53,043::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:53,608::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0733065 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:06:55,623::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:56,201::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0803471 s, 12.7 MB/s\n'; <rc> = 0 >Thread-15913500::DEBUG::2013-11-20 03:06:57,024::clientIF::54::vds::(wrapper) [192.168.100.135]::call create with ({'bridge': 'rhevm,rhevm', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.2.0', 'vmId': '4b8ed9c8-dfb1-4de4-935d-6ba4009567b9', 'transparentHugePages': 'true', 'spiceSslCipherSuite': 'DEFAULT', 'cpuType': 'Westmere', 'smp': '2', 'macAddr': '00:1a:4a:a8:64:05,00:1a:4a:a8:64:0b', 'boot': 'c', 'custom': {}, 'vmType': 'kvm', 'memSize': 2048, 'smpCoresPerSocket': '1', 'vmName': 'KLEE-Fil01', 'spiceMonitors': '1', 'nice': '0', 'drives': [{'index': '0', 'domainID': '6b892067-ac7e-4039-a7be-5b7387525b16', 'format': 'cow', 'boot': 'true', 'volumeID': '0ecd1db7-48b0-4663-a038-e40e891b9424', 'imageID': 'b0980185-7de1-4e3b-bd76-4495169ab08b', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}, {'index': '1', 'domainID': '6b892067-ac7e-4039-a7be-5b7387525b16', 'format': 'cow', 'boot': 'false', 'volumeID': '6d82661d-9502-4cb3-a0a0-91cd82dd2a80', 'imageID': '0b89347c-4654-42bb-a8d7-1a627130b7bd', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}], 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'smartcardEnable': 'false', 'nicModel': 'pv,pv', 'keyboardLayout': 'en-us', 'kvmEnable': 'true', 'timeOffset': '3601', 'spiceSecureChannels': 'smain,sinputs', 'display': 'qxl'},) {} >Thread-15913500::INFO::2013-11-20 03:06:57,415::clientIF::523::vds::(_getNetworkIp) network None: using 0 >Thread-15913500::INFO::2013-11-20 03:06:57,417::clientIF::656::vds::(create) vmContainerLock acquired by vm 4b8ed9c8-dfb1-4de4-935d-6ba4009567b9 >Thread-15913501::DEBUG::2013-11-20 03:06:57,422::vm::342::vm.Vm::(_startUnderlyingVm) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Start >Thread-15913500::DEBUG::2013-11-20 03:06:57,423::clientIF::669::vds::(create) Total desktops after creation of 4b8ed9c8-dfb1-4de4-935d-6ba4009567b9 is 2 >Thread-15913501::DEBUG::2013-11-20 03:06:57,424::vm::346::vm.Vm::(_startUnderlyingVm) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::_ongoingCreations acquired >Thread-15913500::DEBUG::2013-11-20 03:06:57,424::clientIF::59::vds::(wrapper) return create with {'status': {'message': 'Done', 'code': 0}, 'vmList': {'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'bridge': 'rhevm,rhevm', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.2.0', 'vmId': '4b8ed9c8-dfb1-4de4-935d-6ba4009567b9', 'pid': '0', 'drives': [{'index': '0', 'domainID': '6b892067-ac7e-4039-a7be-5b7387525b16', 'format': 'cow', 'boot': 'true', 'volumeID': '0ecd1db7-48b0-4663-a038-e40e891b9424', 'imageID': 'b0980185-7de1-4e3b-bd76-4495169ab08b', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}, {'index': '1', 'domainID': '6b892067-ac7e-4039-a7be-5b7387525b16', 'format': 'cow', 'boot': 'false', 'volumeID': '6d82661d-9502-4cb3-a0a0-91cd82dd2a80', 'imageID': '0b89347c-4654-42bb-a8d7-1a627130b7bd', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}], 'transparentHugePages': 'true', 'keyboardLayout': 'en-us', 'displaySecurePort': '-1', 'displayPort': '-1', 'cpuType': 'Westmere', 'custom': {}, 'clientIp': '', 'nicModel': 'pv,pv', 'macAddr': '00:1a:4a:a8:64:05,00:1a:4a:a8:64:0b', 'kvmEnable': 'true', 'smartcardEnable': 'false', 'boot': 'c', 'smp': '2', 'vmType': 'kvm', 'spiceSslCipherSuite': 'DEFAULT', 'status': 'WaitForLaunch', 'timeOffset': '3601', 'memSize': 2048, 'displayIp': '0', 'spiceSecureChannels': 'smain,sinputs', 'smpCoresPerSocket': '1', 'vmName': 'KLEE-Fil01', 'display': 'qxl', 'spiceMonitors': '1', 'nice': '0'}} >Thread-15913501::INFO::2013-11-20 03:06:57,425::libvirtvm::1061::vm.Vm::(_run) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::VM wrapper has started >Thread-15913501::INFO::2013-11-20 03:06:57,429::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913501::DEBUG::2013-11-20 03:06:57,430::task::495::TaskManager.Task::(_debug) Task ce5b44eb-1521-42df-a14a-c21ed0c9f96f: moving from state init -> state preparing >Thread-15913501::DEBUG::2013-11-20 03:06:57,432::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`930891b2-62fa-45ab-8d70-9807178ae8a9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1690' at 'public_prepareVolume' >Thread-15913501::DEBUG::2013-11-20 03:06:57,432::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913501::DEBUG::2013-11-20 03:06:57,433::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913501::DEBUG::2013-11-20 03:06:57,434::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`930891b2-62fa-45ab-8d70-9807178ae8a9`::Granted request >Thread-15913501::DEBUG::2013-11-20 03:06:57,435::task::495::TaskManager.Task::(_debug) Task ce5b44eb-1521-42df-a14a-c21ed0c9f96f: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913501::DEBUG::2013-11-20 03:06:57,435::task::495::TaskManager.Task::(_debug) Task ce5b44eb-1521-42df-a14a-c21ed0c9f96f: ref 1 aborting False >Thread-15913501::DEBUG::2013-11-20 03:06:57,436::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.b0980185-7de1-4e3b-bd76-4495169ab08b`ReqID=`eebdfdf4-0a99-443a-a1a9-d91fa3368f94`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1695' at 'public_prepareVolume' >Thread-15913501::DEBUG::2013-11-20 03:06:57,437::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.b0980185-7de1-4e3b-bd76-4495169ab08b' for lock type 'exclusive' >Thread-15913501::DEBUG::2013-11-20 03:06:57,438::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:06:57,439::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15913503::INFO::2013-11-20 03:06:58,018::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913503::DEBUG::2013-11-20 03:06:58,029::task::495::TaskManager.Task::(_debug) Task 8d429878-aa7a-404b-bc02-99ebfe0564ef: moving from state init -> state preparing >Thread-15913503::DEBUG::2013-11-20 03:06:58,030::task::495::TaskManager.Task::(_debug) Task 8d429878-aa7a-404b-bc02-99ebfe0564ef: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000319004058838', 'lastCheck': 1384916816.8568971, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0014979839325', 'lastCheck': 1384916813.745677, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0211429595947', 'lastCheck': 1384916814.766695, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00043797492981', 'lastCheck': 1384916814.286289, 'valid': True, 'code': 0}} >Thread-15913503::DEBUG::2013-11-20 03:06:58,031::task::495::TaskManager.Task::(_debug) Task 8d429878-aa7a-404b-bc02-99ebfe0564ef: moving from state preparing -> state finished >Thread-15913503::DEBUG::2013-11-20 03:06:58,031::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913503::DEBUG::2013-11-20 03:06:58,032::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913503::DEBUG::2013-11-20 03:06:58,032::task::495::TaskManager.Task::(_debug) Task 8d429878-aa7a-404b-bc02-99ebfe0564ef: ref 0 aborting False >Thread-15913503::INFO::2013-11-20 03:06:58,033::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000319004058838', 'lastCheck': 1384916816.8568971, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0014979839325', 'lastCheck': 1384916813.745677, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0211429595947', 'lastCheck': 1384916814.766695, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00043797492981', 'lastCheck': 1384916814.286289, 'valid': True, 'code': 0}} >Thread-15913501::DEBUG::2013-11-20 03:06:58,140::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:06:58,263::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913501::DEBUG::2013-11-20 03:06:58,618::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:06:58,628::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=21 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:06:59,096::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.111779 s, 9.2 MB/s\n'; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:06:59,130::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.00379108 s, 135 kB/s\n'; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:06:59,131::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.00379108 s, 135 kB/s'], size: 512 >Thread-15913501::DEBUG::2013-11-20 03:06:59,132::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=21 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15913501::DEBUG::2013-11-20 03:06:59,560::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000199929 s, 2.6 MB/s\n'; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:06:59,561::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000199929 s, 2.6 MB/s'], size: 512 >Thread-15913501::INFO::2013-11-20 03:06:59,562::image::340::Storage.Image::(getChain) sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b chain=[<storage.blockVolume.BlockVolume instance at 0x7f48cc193ab8>] >Thread-15913501::DEBUG::2013-11-20 03:06:59,564::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --autobackup n --available y 6b892067-ac7e-4039-a7be-5b7387525b16/0ecd1db7-48b0-4663-a038-e40e891b9424' (cwd None) >Thread-15913505::DEBUG::2013-11-20 03:06:59,786::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913505::INFO::2013-11-20 03:06:59,788::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913505::DEBUG::2013-11-20 03:06:59,789::task::495::TaskManager.Task::(_debug) Task 5b2d598e-fad3-4fef-b958-b1fa69421123: moving from state init -> state preparing >Thread-15913505::DEBUG::2013-11-20 03:06:59,791::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913505::DEBUG::2013-11-20 03:06:59,791::task::495::TaskManager.Task::(_debug) Task 5b2d598e-fad3-4fef-b958-b1fa69421123: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913505::DEBUG::2013-11-20 03:06:59,792::task::495::TaskManager.Task::(_debug) Task 5b2d598e-fad3-4fef-b958-b1fa69421123: moving from state preparing -> state finished >Thread-15913505::DEBUG::2013-11-20 03:06:59,792::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913505::DEBUG::2013-11-20 03:06:59,793::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913505::DEBUG::2013-11-20 03:06:59,794::task::495::TaskManager.Task::(_debug) Task 5b2d598e-fad3-4fef-b958-b1fa69421123: ref 0 aborting False >Thread-15913505::INFO::2013-11-20 03:06:59,794::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913506::DEBUG::2013-11-20 03:06:59,799::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913506::INFO::2013-11-20 03:06:59,800::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913506::DEBUG::2013-11-20 03:06:59,801::task::495::TaskManager.Task::(_debug) Task 70ea544c-9075-4b73-a958-2ab0dd5e8360: moving from state init -> state preparing >Thread-15913506::DEBUG::2013-11-20 03:06:59,802::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`4d5c20cb-fbc6-45e7-b8f3-ca4a1310e6f7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913506::DEBUG::2013-11-20 03:06:59,803::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913506::DEBUG::2013-11-20 03:06:59,804::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913506::DEBUG::2013-11-20 03:06:59,804::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`4d5c20cb-fbc6-45e7-b8f3-ca4a1310e6f7`::Granted request >Thread-15913506::DEBUG::2013-11-20 03:06:59,805::task::495::TaskManager.Task::(_debug) Task 70ea544c-9075-4b73-a958-2ab0dd5e8360: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913506::DEBUG::2013-11-20 03:06:59,806::task::495::TaskManager.Task::(_debug) Task 70ea544c-9075-4b73-a958-2ab0dd5e8360: ref 1 aborting False >Thread-15913506::WARNING::2013-11-20 03:06:59,808::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913506::DEBUG::2013-11-20 03:06:59,809::task::495::TaskManager.Task::(_debug) Task 70ea544c-9075-4b73-a958-2ab0dd5e8360: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913506::DEBUG::2013-11-20 03:06:59,810::task::495::TaskManager.Task::(_debug) Task 70ea544c-9075-4b73-a958-2ab0dd5e8360: moving from state preparing -> state finished >Thread-15913506::DEBUG::2013-11-20 03:06:59,811::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913506::DEBUG::2013-11-20 03:06:59,811::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913506::DEBUG::2013-11-20 03:06:59,812::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913506::DEBUG::2013-11-20 03:06:59,813::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913506::DEBUG::2013-11-20 03:06:59,813::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913506::DEBUG::2013-11-20 03:06:59,814::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913506::DEBUG::2013-11-20 03:06:59,814::task::495::TaskManager.Task::(_debug) Task 70ea544c-9075-4b73-a958-2ab0dd5e8360: ref 0 aborting False >Thread-15913506::INFO::2013-11-20 03:06:59,815::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913501::DEBUG::2013-11-20 03:07:00,455::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:00,456::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:00,457::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:00,458::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.0ecd1db7-48b0-4663-a038-e40e891b9424`ReqID=`aeb39440-87db-4845-97b7-5160e955b048`::Request was made in '/usr/share/vdsm/storage/resourceFactories.py' line '149' at '__getResourceCandidatesList' >Thread-15913501::DEBUG::2013-11-20 03:07:00,459::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.0ecd1db7-48b0-4663-a038-e40e891b9424' for lock type 'exclusive' >Thread-15913501::DEBUG::2013-11-20 03:07:00,460::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.0ecd1db7-48b0-4663-a038-e40e891b9424' is free. Now locking as 'exclusive' (1 active user) >Thread-15913501::DEBUG::2013-11-20 03:07:00,460::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.0ecd1db7-48b0-4663-a038-e40e891b9424`ReqID=`aeb39440-87db-4845-97b7-5160e955b048`::Granted request >Thread-15913501::DEBUG::2013-11-20 03:07:00,461::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.b0980185-7de1-4e3b-bd76-4495169ab08b' is free. Now locking as 'exclusive' (1 active user) >Thread-15913501::DEBUG::2013-11-20 03:07:00,462::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.b0980185-7de1-4e3b-bd76-4495169ab08b`ReqID=`eebdfdf4-0a99-443a-a1a9-d91fa3368f94`::Granted request >Thread-15913501::DEBUG::2013-11-20 03:07:00,463::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:00,464::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:01,112::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913501::DEBUG::2013-11-20 03:07:01,383::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:01,668::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:01,673::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --refresh 6b892067-ac7e-4039-a7be-5b7387525b16/0ecd1db7-48b0-4663-a038-e40e891b9424' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:02,011::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.367596 s, 2.8 MB/s\n'; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:02,317::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:02,319::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:02,320::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913501::INFO::2013-11-20 03:07:02,320::volume::517::Storage.Volume::(prepare) Volume: preparing volume 6b892067-ac7e-4039-a7be-5b7387525b16/0ecd1db7-48b0-4663-a038-e40e891b9424 >Thread-15913501::DEBUG::2013-11-20 03:07:02,321::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:02,322::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15913501::DEBUG::2013-11-20 03:07:03,012::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:03,253::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:03,253::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=21 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15913501::DEBUG::2013-11-20 03:07:03,628::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000289655 s, 1.8 MB/s\n'; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:03,629::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000289655 s, 1.8 MB/s'], size: 512 >Thread-15913501::DEBUG::2013-11-20 03:07:03,630::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=21 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:04,027::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913501::DEBUG::2013-11-20 03:07:04,276::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000333103 s, 1.5 MB/s\n'; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:04,278::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000333103 s, 1.5 MB/s'], size: 512 >Thread-15913501::DEBUG::2013-11-20 03:07:04,280::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.0ecd1db7-48b0-4663-a038-e40e891b9424`ReqID=`f1512aae-3e29-496a-94f6-7da7f963de20`::Request was made in '/usr/share/vdsm/storage/volume.py' line '533' at 'prepare' >Thread-15913501::DEBUG::2013-11-20 03:07:04,281::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.0ecd1db7-48b0-4663-a038-e40e891b9424' for lock type 'exclusive' >Thread-15913501::DEBUG::2013-11-20 03:07:04,282::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.0ecd1db7-48b0-4663-a038-e40e891b9424' is free. Now locking as 'exclusive' (1 active user) >Thread-15913501::DEBUG::2013-11-20 03:07:04,282::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.0ecd1db7-48b0-4663-a038-e40e891b9424`ReqID=`f1512aae-3e29-496a-94f6-7da7f963de20`::Granted request >Thread-15913501::DEBUG::2013-11-20 03:07:04,284::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=21 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:04,764::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.146654 s, 7.0 MB/s\n'; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:04,821::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000270587 s, 1.9 MB/s\n'; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:04,822::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000270587 s, 1.9 MB/s'], size: 512 >Thread-15913501::DEBUG::2013-11-20 03:07:04,824::task::495::TaskManager.Task::(_debug) Task ce5b44eb-1521-42df-a14a-c21ed0c9f96f: finished: None >Thread-15913501::DEBUG::2013-11-20 03:07:04,824::task::495::TaskManager.Task::(_debug) Task ce5b44eb-1521-42df-a14a-c21ed0c9f96f: moving from state preparing -> state finished >Thread-15913501::DEBUG::2013-11-20 03:07:04,825::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913501::DEBUG::2013-11-20 03:07:04,826::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913501::DEBUG::2013-11-20 03:07:04,826::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913501::DEBUG::2013-11-20 03:07:04,827::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913501::DEBUG::2013-11-20 03:07:04,828::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913501::DEBUG::2013-11-20 03:07:04,828::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913501::DEBUG::2013-11-20 03:07:04,829::task::495::TaskManager.Task::(_debug) Task ce5b44eb-1521-42df-a14a-c21ed0c9f96f: ref 0 aborting False >Thread-15913501::INFO::2013-11-20 03:07:04,829::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15913501::INFO::2013-11-20 03:07:04,831::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913501::DEBUG::2013-11-20 03:07:04,831::task::495::TaskManager.Task::(_debug) Task b1af75d5-11a7-42dd-9e91-347fddeb2d7b: moving from state init -> state preparing >Thread-15913501::DEBUG::2013-11-20 03:07:04,832::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`9bdfdb9d-2642-4548-a522-2bacdc647654`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1666' at 'public_getVolumePath' >Thread-15913501::DEBUG::2013-11-20 03:07:04,833::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913501::DEBUG::2013-11-20 03:07:04,834::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913501::DEBUG::2013-11-20 03:07:04,834::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`9bdfdb9d-2642-4548-a522-2bacdc647654`::Granted request >Thread-15913501::DEBUG::2013-11-20 03:07:04,835::task::495::TaskManager.Task::(_debug) Task b1af75d5-11a7-42dd-9e91-347fddeb2d7b: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913501::DEBUG::2013-11-20 03:07:04,836::task::495::TaskManager.Task::(_debug) Task b1af75d5-11a7-42dd-9e91-347fddeb2d7b: ref 1 aborting False >Thread-15913501::DEBUG::2013-11-20 03:07:04,837::task::495::TaskManager.Task::(_debug) Task b1af75d5-11a7-42dd-9e91-347fddeb2d7b: finished: {'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/b0980185-7de1-4e3b-bd76-4495169ab08b/0ecd1db7-48b0-4663-a038-e40e891b9424'} >Thread-15913501::DEBUG::2013-11-20 03:07:04,837::task::495::TaskManager.Task::(_debug) Task b1af75d5-11a7-42dd-9e91-347fddeb2d7b: moving from state preparing -> state finished >Thread-15913501::DEBUG::2013-11-20 03:07:04,838::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913501::DEBUG::2013-11-20 03:07:04,838::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913501::DEBUG::2013-11-20 03:07:04,839::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913501::DEBUG::2013-11-20 03:07:04,839::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913501::DEBUG::2013-11-20 03:07:04,840::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913501::DEBUG::2013-11-20 03:07:04,840::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913501::DEBUG::2013-11-20 03:07:04,841::task::495::TaskManager.Task::(_debug) Task b1af75d5-11a7-42dd-9e91-347fddeb2d7b: ref 0 aborting False >Thread-15913501::INFO::2013-11-20 03:07:04,842::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, Return response: {'status': {'message': 'OK', 'code': 0}, 'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/b0980185-7de1-4e3b-bd76-4495169ab08b/0ecd1db7-48b0-4663-a038-e40e891b9424'} >Thread-15913501::INFO::2013-11-20 03:07:04,842::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913501::DEBUG::2013-11-20 03:07:04,843::task::495::TaskManager.Task::(_debug) Task 92047c46-90e8-469c-b10b-19c106dfd47c: moving from state init -> state preparing >Thread-15913501::DEBUG::2013-11-20 03:07:04,844::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b0c07c72-55b3-4781-b87e-b1e5f390f4f4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1690' at 'public_prepareVolume' >Thread-15913501::DEBUG::2013-11-20 03:07:04,844::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913501::DEBUG::2013-11-20 03:07:04,845::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913501::DEBUG::2013-11-20 03:07:04,846::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b0c07c72-55b3-4781-b87e-b1e5f390f4f4`::Granted request >Thread-15913501::DEBUG::2013-11-20 03:07:04,846::task::495::TaskManager.Task::(_debug) Task 92047c46-90e8-469c-b10b-19c106dfd47c: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913501::DEBUG::2013-11-20 03:07:04,847::task::495::TaskManager.Task::(_debug) Task 92047c46-90e8-469c-b10b-19c106dfd47c: ref 1 aborting False >Thread-15913501::DEBUG::2013-11-20 03:07:04,848::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.0b89347c-4654-42bb-a8d7-1a627130b7bd`ReqID=`7dd2c752-5acf-4b7c-b2df-bb8711c7f4f9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1695' at 'public_prepareVolume' >Thread-15913501::DEBUG::2013-11-20 03:07:04,849::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.0b89347c-4654-42bb-a8d7-1a627130b7bd' for lock type 'exclusive' >Thread-15913501::DEBUG::2013-11-20 03:07:04,850::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:04,851::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15913501::DEBUG::2013-11-20 03:07:05,602::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:05,830::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:05,832::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=22 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15913501::DEBUG::2013-11-20 03:07:06,228::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.00533553 s, 96.0 kB/s\n'; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:06,229::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.00533553 s, 96.0 kB/s'], size: 512 >Thread-15913501::DEBUG::2013-11-20 03:07:06,230::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=22 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15913501::DEBUG::2013-11-20 03:07:06,637::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000215463 s, 2.4 MB/s\n'; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:06,638::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000215463 s, 2.4 MB/s'], size: 512 >Thread-15913501::INFO::2013-11-20 03:07:06,639::image::340::Storage.Image::(getChain) sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd chain=[<storage.blockVolume.BlockVolume instance at 0x7f487c135bd8>] >Thread-15913501::DEBUG::2013-11-20 03:07:06,641::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --autobackup n --available y 6b892067-ac7e-4039-a7be-5b7387525b16/6d82661d-9502-4cb3-a0a0-91cd82dd2a80' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:06,865::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913509::DEBUG::2013-11-20 03:07:07,369::clientIF::54::vds::(wrapper) [192.168.100.135]::call create with ({'bridge': 'rhevm', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.2.0', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'transparentHugePages': 'true', 'spiceSslCipherSuite': 'DEFAULT', 'cpuType': 'Westmere', 'smp': '8', 'macAddr': '00:1a:4a:a8:64:08', 'boot': 'cd', 'custom': {}, 'vmType': 'kvm', 'memSize': 20480, 'smpCoresPerSocket': '4', 'vmName': 'KLEE-AXDB01', 'spiceMonitors': '1', 'nice': '0', 'drives': [{'index': '0', 'domainID': '772472cb-2429-466d-96cd-00b3301ef78a', 'format': 'raw', 'boot': 'true', 'volumeID': 'e6984064-e09e-4c0c-8b51-a8a1711994cf', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}, {'index': '1', 'domainID': '772472cb-2429-466d-96cd-00b3301ef78a', 'format': 'raw', 'boot': 'false', 'volumeID': '2e903e43-1016-4339-a7f3-c5723a995725', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}, {'index': '2', 'domainID': '772472cb-2429-466d-96cd-00b3301ef78a', 'format': 'raw', 'boot': 'false', 'volumeID': '088640f7-316c-4e2b-bfdc-0f64cf2913c8', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}, {'index': '3', 'domainID': '772472cb-2429-466d-96cd-00b3301ef78a', 'format': 'raw', 'boot': 'false', 'volumeID': 'fdc91117-476f-4b82-8fc4-d682fa283ecc', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}], 'smartcardEnable': 'false', 'nicModel': 'pv', 'keyboardLayout': 'en-us', 'kvmEnable': 'true', 'timeOffset': '-1', 'spiceSecureChannels': 'smain,sinputs', 'display': 'qxl'},) {} >Thread-15913501::DEBUG::2013-11-20 03:07:07,604::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:07,607::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:07,608::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:07,611::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.6d82661d-9502-4cb3-a0a0-91cd82dd2a80`ReqID=`ea58bda5-a1e2-42dc-b7ab-d2e7e13825ae`::Request was made in '/usr/share/vdsm/storage/resourceFactories.py' line '149' at '__getResourceCandidatesList' >Thread-15913501::DEBUG::2013-11-20 03:07:07,612::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.6d82661d-9502-4cb3-a0a0-91cd82dd2a80' for lock type 'exclusive' >Thread-15913501::DEBUG::2013-11-20 03:07:07,613::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.6d82661d-9502-4cb3-a0a0-91cd82dd2a80' is free. Now locking as 'exclusive' (1 active user) >Thread-15913501::DEBUG::2013-11-20 03:07:07,614::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.6d82661d-9502-4cb3-a0a0-91cd82dd2a80`ReqID=`ea58bda5-a1e2-42dc-b7ab-d2e7e13825ae`::Granted request >Thread-15913501::DEBUG::2013-11-20 03:07:07,616::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.0b89347c-4654-42bb-a8d7-1a627130b7bd' is free. Now locking as 'exclusive' (1 active user) >Thread-15913501::DEBUG::2013-11-20 03:07:07,616::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.0b89347c-4654-42bb-a8d7-1a627130b7bd`ReqID=`7dd2c752-5acf-4b7c-b2df-bb8711c7f4f9`::Granted request >Thread-15913501::DEBUG::2013-11-20 03:07:07,620::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:07,631::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15913509::INFO::2013-11-20 03:07:07,972::clientIF::523::vds::(_getNetworkIp) network None: using 0 >Thread-15913509::INFO::2013-11-20 03:07:07,975::clientIF::656::vds::(create) vmContainerLock acquired by vm 6574d88c-00ac-46f4-99ce-a2726932bfda >Thread-15913509::DEBUG::2013-11-20 03:07:08,005::clientIF::669::vds::(create) Total desktops after creation of 6574d88c-00ac-46f4-99ce-a2726932bfda is 3 >Thread-15913510::DEBUG::2013-11-20 03:07:08,006::vm::342::vm.Vm::(_startUnderlyingVm) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Start >Thread-15913509::DEBUG::2013-11-20 03:07:08,007::clientIF::59::vds::(wrapper) return create with {'status': {'message': 'Done', 'code': 0}, 'vmList': {'status': 'WaitForLaunch', 'bridge': 'rhevm', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.2.0', 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'pid': '0', 'drives': [{'index': '0', 'domainID': '772472cb-2429-466d-96cd-00b3301ef78a', 'format': 'raw', 'boot': 'true', 'volumeID': 'e6984064-e09e-4c0c-8b51-a8a1711994cf', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}, {'index': '1', 'domainID': '772472cb-2429-466d-96cd-00b3301ef78a', 'format': 'raw', 'boot': 'false', 'volumeID': '2e903e43-1016-4339-a7f3-c5723a995725', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}, {'index': '2', 'domainID': '772472cb-2429-466d-96cd-00b3301ef78a', 'format': 'raw', 'boot': 'false', 'volumeID': '088640f7-316c-4e2b-bfdc-0f64cf2913c8', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}, {'index': '3', 'domainID': '772472cb-2429-466d-96cd-00b3301ef78a', 'format': 'raw', 'boot': 'false', 'volumeID': 'fdc91117-476f-4b82-8fc4-d682fa283ecc', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}], 'transparentHugePages': 'true', 'keyboardLayout': 'en-us', 'displayPort': '-1', 'displaySecurePort': '-1', 'spiceSslCipherSuite': 'DEFAULT', 'cpuType': 'Westmere', 'custom': {}, 'clientIp': '', 'nicModel': 'pv', 'macAddr': '00:1a:4a:a8:64:08', 'kvmEnable': 'true', 'boot': 'cd', 'smp': '8', 'vmType': 'kvm', 'smartcardEnable': 'false', 'timeOffset': '-1', 'memSize': 20480, 'displayIp': '0', 'spiceSecureChannels': 'smain,sinputs', 'smpCoresPerSocket': '4', 'vmName': 'KLEE-AXDB01', 'display': 'qxl', 'spiceMonitors': '1', 'nice': '0'}} >Dummy-15912651::DEBUG::2013-11-20 03:07:08,095::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.690654 s, 1.5 MB/s\n'; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:08,510::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:08,789::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:08,791::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --refresh 6b892067-ac7e-4039-a7be-5b7387525b16/6d82661d-9502-4cb3-a0a0-91cd82dd2a80' (cwd None) >Thread-15913501::DEBUG::2013-11-20 03:07:09,411::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:09,412::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:09,413::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913501::INFO::2013-11-20 03:07:09,414::volume::517::Storage.Volume::(prepare) Volume: preparing volume 6b892067-ac7e-4039-a7be-5b7387525b16/6d82661d-9502-4cb3-a0a0-91cd82dd2a80 >Thread-15913501::DEBUG::2013-11-20 03:07:09,414::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:09,416::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:10,109::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913501::DEBUG::2013-11-20 03:07:10,387::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913513::DEBUG::2013-11-20 03:07:10,389::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913513::INFO::2013-11-20 03:07:10,466::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913513::DEBUG::2013-11-20 03:07:10,605::task::495::TaskManager.Task::(_debug) Task a5970242-47d5-42bb-ab77-056a166cbd37: moving from state init -> state preparing >Thread-15913513::DEBUG::2013-11-20 03:07:10,714::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913501::DEBUG::2013-11-20 03:07:10,719::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913513::DEBUG::2013-11-20 03:07:10,720::task::495::TaskManager.Task::(_debug) Task a5970242-47d5-42bb-ab77-056a166cbd37: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913513::DEBUG::2013-11-20 03:07:10,721::task::495::TaskManager.Task::(_debug) Task a5970242-47d5-42bb-ab77-056a166cbd37: moving from state preparing -> state finished >Thread-15913501::DEBUG::2013-11-20 03:07:10,722::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=22 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15913513::DEBUG::2013-11-20 03:07:10,722::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913513::DEBUG::2013-11-20 03:07:10,724::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913513::DEBUG::2013-11-20 03:07:10,924::task::495::TaskManager.Task::(_debug) Task a5970242-47d5-42bb-ab77-056a166cbd37: ref 0 aborting False >Thread-15913513::INFO::2013-11-20 03:07:10,926::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913514::DEBUG::2013-11-20 03:07:10,935::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913514::INFO::2013-11-20 03:07:10,938::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913514::DEBUG::2013-11-20 03:07:10,939::task::495::TaskManager.Task::(_debug) Task 1783eae2-e767-400d-8851-9f1cf510ac98: moving from state init -> state preparing >Thread-15913514::DEBUG::2013-11-20 03:07:10,941::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fe25814a-5d1d-4aeb-b5e1-6f1ae398a8af`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913514::DEBUG::2013-11-20 03:07:10,941::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913514::DEBUG::2013-11-20 03:07:10,943::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913514::DEBUG::2013-11-20 03:07:10,944::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fe25814a-5d1d-4aeb-b5e1-6f1ae398a8af`::Granted request >Thread-15913514::DEBUG::2013-11-20 03:07:10,946::task::495::TaskManager.Task::(_debug) Task 1783eae2-e767-400d-8851-9f1cf510ac98: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913514::DEBUG::2013-11-20 03:07:10,946::task::495::TaskManager.Task::(_debug) Task 1783eae2-e767-400d-8851-9f1cf510ac98: ref 1 aborting False >Thread-15913514::WARNING::2013-11-20 03:07:10,950::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913514::DEBUG::2013-11-20 03:07:10,951::task::495::TaskManager.Task::(_debug) Task 1783eae2-e767-400d-8851-9f1cf510ac98: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913514::DEBUG::2013-11-20 03:07:10,952::task::495::TaskManager.Task::(_debug) Task 1783eae2-e767-400d-8851-9f1cf510ac98: moving from state preparing -> state finished >Thread-15913514::DEBUG::2013-11-20 03:07:10,953::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913514::DEBUG::2013-11-20 03:07:10,954::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913514::DEBUG::2013-11-20 03:07:10,955::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913514::DEBUG::2013-11-20 03:07:10,956::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913514::DEBUG::2013-11-20 03:07:10,957::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913514::DEBUG::2013-11-20 03:07:10,958::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913514::DEBUG::2013-11-20 03:07:10,959::task::495::TaskManager.Task::(_debug) Task 1783eae2-e767-400d-8851-9f1cf510ac98: ref 0 aborting False >Thread-15913514::INFO::2013-11-20 03:07:10,959::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:07:11,112::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.488368 s, 2.1 MB/s\n'; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:11,131::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000188178 s, 2.7 MB/s\n'; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:11,132::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000188178 s, 2.7 MB/s'], size: 512 >Thread-15913501::DEBUG::2013-11-20 03:07:11,133::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=22 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15913501::DEBUG::2013-11-20 03:07:11,571::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000292993 s, 1.7 MB/s\n'; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:11,574::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000292993 s, 1.7 MB/s'], size: 512 >Thread-15913501::DEBUG::2013-11-20 03:07:11,575::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.6d82661d-9502-4cb3-a0a0-91cd82dd2a80`ReqID=`4c7e9522-ee21-4c23-89df-9a87e04d5509`::Request was made in '/usr/share/vdsm/storage/volume.py' line '533' at 'prepare' >Thread-15913501::DEBUG::2013-11-20 03:07:11,576::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.6d82661d-9502-4cb3-a0a0-91cd82dd2a80' for lock type 'exclusive' >Thread-15913501::DEBUG::2013-11-20 03:07:11,577::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.6d82661d-9502-4cb3-a0a0-91cd82dd2a80' is free. Now locking as 'exclusive' (1 active user) >Thread-15913501::DEBUG::2013-11-20 03:07:11,578::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.6d82661d-9502-4cb3-a0a0-91cd82dd2a80`ReqID=`4c7e9522-ee21-4c23-89df-9a87e04d5509`::Granted request >Thread-15913501::DEBUG::2013-11-20 03:07:11,581::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=22 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15913501::DEBUG::2013-11-20 03:07:11,988::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000344947 s, 1.5 MB/s\n'; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:11,989::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000344947 s, 1.5 MB/s'], size: 512 >Thread-15913501::DEBUG::2013-11-20 03:07:11,991::task::495::TaskManager.Task::(_debug) Task 92047c46-90e8-469c-b10b-19c106dfd47c: finished: None >Thread-15913501::DEBUG::2013-11-20 03:07:11,992::task::495::TaskManager.Task::(_debug) Task 92047c46-90e8-469c-b10b-19c106dfd47c: moving from state preparing -> state finished >Thread-15913501::DEBUG::2013-11-20 03:07:11,992::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913501::DEBUG::2013-11-20 03:07:11,993::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913501::DEBUG::2013-11-20 03:07:11,994::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913501::DEBUG::2013-11-20 03:07:11,995::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913501::DEBUG::2013-11-20 03:07:11,995::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913501::DEBUG::2013-11-20 03:07:11,996::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913501::DEBUG::2013-11-20 03:07:11,996::task::495::TaskManager.Task::(_debug) Task 92047c46-90e8-469c-b10b-19c106dfd47c: ref 0 aborting False >Thread-15913501::INFO::2013-11-20 03:07:11,997::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15913501::INFO::2013-11-20 03:07:11,998::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913501::DEBUG::2013-11-20 03:07:11,999::task::495::TaskManager.Task::(_debug) Task 22aceda3-c582-4f79-8e84-c202526533a1: moving from state init -> state preparing >Thread-15913501::DEBUG::2013-11-20 03:07:12,000::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`dfc2d478-11df-4cc7-a5b6-d4a3b5c51cbb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1666' at 'public_getVolumePath' >Thread-15913501::DEBUG::2013-11-20 03:07:12,001::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913501::DEBUG::2013-11-20 03:07:12,001::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913501::DEBUG::2013-11-20 03:07:12,002::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`dfc2d478-11df-4cc7-a5b6-d4a3b5c51cbb`::Granted request >Thread-15913501::DEBUG::2013-11-20 03:07:12,003::task::495::TaskManager.Task::(_debug) Task 22aceda3-c582-4f79-8e84-c202526533a1: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913501::DEBUG::2013-11-20 03:07:12,003::task::495::TaskManager.Task::(_debug) Task 22aceda3-c582-4f79-8e84-c202526533a1: ref 1 aborting False >Thread-15913501::DEBUG::2013-11-20 03:07:12,004::task::495::TaskManager.Task::(_debug) Task 22aceda3-c582-4f79-8e84-c202526533a1: finished: {'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/0b89347c-4654-42bb-a8d7-1a627130b7bd/6d82661d-9502-4cb3-a0a0-91cd82dd2a80'} >Thread-15913501::DEBUG::2013-11-20 03:07:12,005::task::495::TaskManager.Task::(_debug) Task 22aceda3-c582-4f79-8e84-c202526533a1: moving from state preparing -> state finished >Thread-15913501::DEBUG::2013-11-20 03:07:12,005::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913501::DEBUG::2013-11-20 03:07:12,006::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913501::DEBUG::2013-11-20 03:07:12,007::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913501::DEBUG::2013-11-20 03:07:12,007::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913501::DEBUG::2013-11-20 03:07:12,008::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913501::DEBUG::2013-11-20 03:07:12,008::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913501::DEBUG::2013-11-20 03:07:12,009::task::495::TaskManager.Task::(_debug) Task 22aceda3-c582-4f79-8e84-c202526533a1: ref 0 aborting False >Thread-15913501::INFO::2013-11-20 03:07:12,009::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, Return response: {'status': {'message': 'OK', 'code': 0}, 'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/0b89347c-4654-42bb-a8d7-1a627130b7bd/6d82661d-9502-4cb3-a0a0-91cd82dd2a80'} >Thread-15913501::INFO::2013-11-20 03:07:12,011::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913501::DEBUG::2013-11-20 03:07:12,012::task::495::TaskManager.Task::(_debug) Task a922f760-a39f-4807-98ba-08aa31f59c65: moving from state init -> state preparing >Thread-15913501::DEBUG::2013-11-20 03:07:12,013::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`68b3efb9-86fa-4428-898c-9886dcb5b933`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913501::DEBUG::2013-11-20 03:07:12,014::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913501::DEBUG::2013-11-20 03:07:12,014::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913501::DEBUG::2013-11-20 03:07:12,015::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`68b3efb9-86fa-4428-898c-9886dcb5b933`::Granted request >Thread-15913501::DEBUG::2013-11-20 03:07:12,016::task::495::TaskManager.Task::(_debug) Task a922f760-a39f-4807-98ba-08aa31f59c65: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913501::DEBUG::2013-11-20 03:07:12,016::task::495::TaskManager.Task::(_debug) Task a922f760-a39f-4807-98ba-08aa31f59c65: ref 1 aborting False >Thread-15913501::DEBUG::2013-11-20 03:07:12,017::task::495::TaskManager.Task::(_debug) Task a922f760-a39f-4807-98ba-08aa31f59c65: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913501::DEBUG::2013-11-20 03:07:12,018::task::495::TaskManager.Task::(_debug) Task a922f760-a39f-4807-98ba-08aa31f59c65: moving from state preparing -> state finished >Thread-15913501::DEBUG::2013-11-20 03:07:12,018::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913501::DEBUG::2013-11-20 03:07:12,019::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913501::DEBUG::2013-11-20 03:07:12,019::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913501::DEBUG::2013-11-20 03:07:12,020::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913501::DEBUG::2013-11-20 03:07:12,020::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913501::DEBUG::2013-11-20 03:07:12,021::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913501::DEBUG::2013-11-20 03:07:12,021::task::495::TaskManager.Task::(_debug) Task a922f760-a39f-4807-98ba-08aa31f59c65: ref 0 aborting False >Thread-15913501::INFO::2013-11-20 03:07:12,022::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913501::INFO::2013-11-20 03:07:12,023::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16) >Thread-15913501::DEBUG::2013-11-20 03:07:12,023::task::495::TaskManager.Task::(_debug) Task f9338935-cfdf-4b83-9010-d9ef6a3869d7: moving from state init -> state preparing >Thread-15913501::INFO::2013-11-20 03:07:12,024::blockSD::674::Storage.StorageDomain::(validate) sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 >Thread-15913501::DEBUG::2013-11-20 03:07:12,024::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:12,025::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:12,026::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:12,027::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:12,028::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:12,029::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15913501::DEBUG::2013-11-20 03:07:12,696::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:12,700::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:12,701::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'_SHA_CKSUM=716ea7492d9134a4b13de27ac61a336ad474727c', u'POOL_SPM_ID=2', u'POOL_SPM_LVER=287'] >Thread-15913501::DEBUG::2013-11-20 03:07:12,703::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c43e7ea6-d79f-408a-b684-937fdc62587f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1297' at 'public_getStorageDomainInfo' >Thread-15913501::DEBUG::2013-11-20 03:07:12,704::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913501::DEBUG::2013-11-20 03:07:12,705::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913501::DEBUG::2013-11-20 03:07:12,705::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c43e7ea6-d79f-408a-b684-937fdc62587f`::Granted request >Thread-15913501::DEBUG::2013-11-20 03:07:12,706::task::495::TaskManager.Task::(_debug) Task f9338935-cfdf-4b83-9010-d9ef6a3869d7: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913501::DEBUG::2013-11-20 03:07:12,707::task::495::TaskManager.Task::(_debug) Task f9338935-cfdf-4b83-9010-d9ef6a3869d7: ref 1 aborting False >Thread-15913501::WARNING::2013-11-20 03:07:12,709::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913501::DEBUG::2013-11-20 03:07:12,710::task::495::TaskManager.Task::(_debug) Task f9338935-cfdf-4b83-9010-d9ef6a3869d7: finished: {'info': {'uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'vguuid': 'aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', 'lver': 287, 'state': 'OK', 'version': '2', 'role': 'Master', 'master_ver': 1, 'spm_id': 2, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'OS'}} >Thread-15913501::DEBUG::2013-11-20 03:07:12,711::task::495::TaskManager.Task::(_debug) Task f9338935-cfdf-4b83-9010-d9ef6a3869d7: moving from state preparing -> state finished >Thread-15913501::DEBUG::2013-11-20 03:07:12,711::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913501::DEBUG::2013-11-20 03:07:12,712::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913501::DEBUG::2013-11-20 03:07:12,713::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913501::DEBUG::2013-11-20 03:07:12,713::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913501::DEBUG::2013-11-20 03:07:12,714::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913501::DEBUG::2013-11-20 03:07:12,714::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913501::DEBUG::2013-11-20 03:07:12,715::task::495::TaskManager.Task::(_debug) Task f9338935-cfdf-4b83-9010-d9ef6a3869d7: ref 0 aborting False >Thread-15913501::INFO::2013-11-20 03:07:12,715::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'vguuid': 'aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', 'lver': 287, 'state': 'OK', 'version': '2', 'role': 'Master', 'master_ver': 1, 'spm_id': 2, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'OS'}} >Thread-15913501::INFO::2013-11-20 03:07:12,716::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913501::DEBUG::2013-11-20 03:07:12,717::task::495::TaskManager.Task::(_debug) Task d79850ca-fd0b-4a3b-bb60-cb4c1e74d8c9: moving from state init -> state preparing >Thread-15913501::DEBUG::2013-11-20 03:07:12,718::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`301f15d2-38f4-451f-83db-785c8e8d89da`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913501::DEBUG::2013-11-20 03:07:12,718::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913501::DEBUG::2013-11-20 03:07:12,719::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913501::DEBUG::2013-11-20 03:07:12,720::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`301f15d2-38f4-451f-83db-785c8e8d89da`::Granted request >Thread-15913501::DEBUG::2013-11-20 03:07:12,720::task::495::TaskManager.Task::(_debug) Task d79850ca-fd0b-4a3b-bb60-cb4c1e74d8c9: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913501::DEBUG::2013-11-20 03:07:12,721::task::495::TaskManager.Task::(_debug) Task d79850ca-fd0b-4a3b-bb60-cb4c1e74d8c9: ref 1 aborting False >Thread-15913501::DEBUG::2013-11-20 03:07:12,722::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:12,723::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:13,131::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913501::DEBUG::2013-11-20 03:07:13,399::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:13,704::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:13,706::task::495::TaskManager.Task::(_debug) Task d79850ca-fd0b-4a3b-bb60-cb4c1e74d8c9: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913501::DEBUG::2013-11-20 03:07:13,707::task::495::TaskManager.Task::(_debug) Task d79850ca-fd0b-4a3b-bb60-cb4c1e74d8c9: moving from state preparing -> state finished >Thread-15913501::DEBUG::2013-11-20 03:07:13,708::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913501::DEBUG::2013-11-20 03:07:13,708::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913501::DEBUG::2013-11-20 03:07:13,710::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913501::DEBUG::2013-11-20 03:07:13,711::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913501::DEBUG::2013-11-20 03:07:13,712::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913501::DEBUG::2013-11-20 03:07:13,713::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913501::DEBUG::2013-11-20 03:07:13,714::task::495::TaskManager.Task::(_debug) Task d79850ca-fd0b-4a3b-bb60-cb4c1e74d8c9: ref 0 aborting False >Thread-15913501::INFO::2013-11-20 03:07:13,715::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913501::INFO::2013-11-20 03:07:13,716::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16) >Thread-15913501::DEBUG::2013-11-20 03:07:13,717::task::495::TaskManager.Task::(_debug) Task a5f67c37-ca6d-452f-a3fe-56de12b9eec7: moving from state init -> state preparing >Thread-15913501::INFO::2013-11-20 03:07:13,718::blockSD::674::Storage.StorageDomain::(validate) sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 >Thread-15913501::DEBUG::2013-11-20 03:07:13,719::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:13,720::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:13,721::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:13,722::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:13,724::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:13,725::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:14,068::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.447456 s, 2.3 MB/s\n'; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:14,392::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:14,396::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15913501::DEBUG::2013-11-20 03:07:14,397::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'_SHA_CKSUM=716ea7492d9134a4b13de27ac61a336ad474727c', u'POOL_SPM_ID=2', u'POOL_SPM_LVER=287'] >Thread-15913501::DEBUG::2013-11-20 03:07:14,399::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`02bc3e30-fefa-4b7d-985f-b5263a4be490`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1297' at 'public_getStorageDomainInfo' >Thread-15913501::DEBUG::2013-11-20 03:07:14,400::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913501::DEBUG::2013-11-20 03:07:14,401::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913501::DEBUG::2013-11-20 03:07:14,401::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`02bc3e30-fefa-4b7d-985f-b5263a4be490`::Granted request >Thread-15913501::DEBUG::2013-11-20 03:07:14,403::task::495::TaskManager.Task::(_debug) Task a5f67c37-ca6d-452f-a3fe-56de12b9eec7: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913501::DEBUG::2013-11-20 03:07:14,403::task::495::TaskManager.Task::(_debug) Task a5f67c37-ca6d-452f-a3fe-56de12b9eec7: ref 1 aborting False >Thread-15913501::WARNING::2013-11-20 03:07:14,407::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913501::DEBUG::2013-11-20 03:07:14,408::task::495::TaskManager.Task::(_debug) Task a5f67c37-ca6d-452f-a3fe-56de12b9eec7: finished: {'info': {'uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'vguuid': 'aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', 'lver': 287, 'state': 'OK', 'version': '2', 'role': 'Master', 'master_ver': 1, 'spm_id': 2, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'OS'}} >Thread-15913501::DEBUG::2013-11-20 03:07:14,409::task::495::TaskManager.Task::(_debug) Task a5f67c37-ca6d-452f-a3fe-56de12b9eec7: moving from state preparing -> state finished >Thread-15913501::DEBUG::2013-11-20 03:07:14,410::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913516::INFO::2013-11-20 03:07:14,412::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913501::DEBUG::2013-11-20 03:07:14,413::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913516::DEBUG::2013-11-20 03:07:14,414::task::495::TaskManager.Task::(_debug) Task 29f582fd-374c-4835-b39b-3a904d9601f7: moving from state init -> state preparing >Thread-15913501::DEBUG::2013-11-20 03:07:14,415::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913516::DEBUG::2013-11-20 03:07:14,416::task::495::TaskManager.Task::(_debug) Task 29f582fd-374c-4835-b39b-3a904d9601f7: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000458002090454', 'lastCheck': 1384916827.1933119, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0236279964447', 'lastCheck': 1384916834.4015639, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00136518478394', 'lastCheck': 1384916825.078372, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00197911262512', 'lastCheck': 1384916824.616087, 'valid': True, 'code': 0}} >Thread-15913501::DEBUG::2013-11-20 03:07:14,417::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913516::DEBUG::2013-11-20 03:07:14,418::task::495::TaskManager.Task::(_debug) Task 29f582fd-374c-4835-b39b-3a904d9601f7: moving from state preparing -> state finished >Thread-15913501::DEBUG::2013-11-20 03:07:14,419::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913516::DEBUG::2013-11-20 03:07:14,419::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913501::DEBUG::2013-11-20 03:07:14,420::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913516::DEBUG::2013-11-20 03:07:14,421::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913501::DEBUG::2013-11-20 03:07:14,421::task::495::TaskManager.Task::(_debug) Task a5f67c37-ca6d-452f-a3fe-56de12b9eec7: ref 0 aborting False >Thread-15913516::DEBUG::2013-11-20 03:07:14,422::task::495::TaskManager.Task::(_debug) Task 29f582fd-374c-4835-b39b-3a904d9601f7: ref 0 aborting False >Thread-15913501::INFO::2013-11-20 03:07:14,423::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'vguuid': 'aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', 'lver': 287, 'state': 'OK', 'version': '2', 'role': 'Master', 'master_ver': 1, 'spm_id': 2, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'OS'}} >Thread-15913516::INFO::2013-11-20 03:07:14,424::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000458002090454', 'lastCheck': 1384916827.1933119, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0236279964447', 'lastCheck': 1384916834.4015639, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00136518478394', 'lastCheck': 1384916825.078372, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00197911262512', 'lastCheck': 1384916824.616087, 'valid': True, 'code': 0}} >Thread-15913501::DEBUG::2013-11-20 03:07:14,442::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/libexec/vdsm/hooks/before_vm_start/50_vhostmd' (cwd None) >Thread-15913501::DEBUG::2013-11-20 03:07:14,860::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913501::INFO::2013-11-20 03:07:14,861::hooks::51::root::(_runHooksDir) >Thread-15913501::DEBUG::2013-11-20 03:07:14,862::libvirtvm::1077::vm.Vm::(_run) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::<?xml version="1.0" encoding="utf-8"?> ><domain type="kvm"> > <name>KLEE-Fil01</name> > <uuid>4b8ed9c8-dfb1-4de4-935d-6ba4009567b9</uuid> > <memory>2097152</memory> > <currentMemory>2097152</currentMemory> > <vcpu>2</vcpu> > <devices> > <disk device="disk" type="block"> > <source dev="/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/b0980185-7de1-4e3b-bd76-4495169ab08b/0ecd1db7-48b0-4663-a038-e40e891b9424"/> > <target bus="virtio" dev="vda"/> > <serial>3b-bd76-4495169ab08b</serial> > <driver cache="none" error_policy="stop" io="native" name="qemu" type="qcow2"/> > </disk> > <disk device="disk" type="block"> > <source dev="/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/0b89347c-4654-42bb-a8d7-1a627130b7bd/6d82661d-9502-4cb3-a0a0-91cd82dd2a80"/> > <target bus="virtio" dev="vdb"/> > <serial>bb-a8d7-1a627130b7bd</serial> > <driver cache="none" error_policy="stop" io="native" name="qemu" type="qcow2"/> > </disk> > <disk device="cdrom" type="file"> > <source file="/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO"/> > <target bus="ide" dev="hdc"/> > </disk> > <controller index="0" ports="16" type="virtio-serial"/> > <channel type="unix"> > <target name="com.redhat.rhevm.vdsm" type="virtio"/> > <source mode="bind" path="/var/lib/libvirt/qemu/channels/KLEE-Fil01.com.redhat.rhevm.vdsm"/> > </channel> > <memballoon model="none"/> > <interface type="bridge"> > <mac address="00:1a:4a:a8:64:05"/> > <model type="virtio"/> > <source bridge="rhevm"/> > </interface> > <interface type="bridge"> > <mac address="00:1a:4a:a8:64:0b"/> > <model type="virtio"/> > <source bridge="rhevm"/> > </interface> > <input bus="ps2" type="mouse"/> > <video> > <model heads="1" type="qxl" vram="65536"/> > </video> > <channel type="spicevmc"> > <target name="com.redhat.spice.0" type="virtio"/> > </channel> > <graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice"> > <channel mode="secure" name="main"/> > <channel mode="secure" name="inputs"/> > </graphics> > </devices> > <os> > <type arch="x86_64" machine="rhel6.2.0">hvm</type> > <boot dev="hd"/> > <smbios mode="sysinfo"/> > </os> > <sysinfo type="smbios"> > <system> > <entry name="manufacturer">Red Hat</entry> > <entry name="product">RHEV Hypervisor</entry> > <entry name="version">6.3-20121012.0.el6_3</entry> > <entry name="serial">30343536-3138-5A43-4A32-333630373442_AC:16:2D:76:EB:F0</entry> > <entry name="uuid">4b8ed9c8-dfb1-4de4-935d-6ba4009567b9</entry> > </system> > </sysinfo> > <clock adjustment="3601" offset="variable"> > <timer name="rtc" tickpolicy="catchup"/> > </clock> > <features> > <acpi/> > </features> > <cpu match="exact"> > <model>Westmere</model> > <topology cores="1" sockets="2" threads="1"/> > </cpu> ></domain> > >libvirtEventLoop::DEBUG::2013-11-20 03:07:15,825::libvirtvm::1401::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::event Started detail 0 opaque None >Thread-15913501::DEBUG::2013-11-20 03:07:15,893::utils::307::vm.Vm::(start) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Start statistics collection >Thread-15913518::DEBUG::2013-11-20 03:07:15,894::utils::336::vm.Vm::(run) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Stats thread started >Thread-15913519::DEBUG::2013-11-20 03:07:15,904::guestIF::47::root::(_connect) Attempting connection to /var/lib/libvirt/qemu/channels/KLEE-Fil01.com.redhat.rhevm.vdsm >Thread-15913501::WARNING::2013-11-20 03:07:15,907::libvirtvm::1124::vm.Vm::(_readPauseCode) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::_readPauseCode unsupported by libvirt vm >Thread-15913519::DEBUG::2013-11-20 03:07:15,908::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /usr/share/vdsm/prepare-vmchannel /var/lib/libvirt/qemu/channels/KLEE-Fil01.com.redhat.rhevm.vdsm' (cwd None) >Thread-15913501::DEBUG::2013-11-20 03:07:15,911::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /usr/share/vdsm/get-vm-pid KLEE-Fil01' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:16,391::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913519::DEBUG::2013-11-20 03:07:16,619::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:16,621::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913519::DEBUG::2013-11-20 03:07:16,622::guestIF::53::root::(_connect) connected to /var/lib/libvirt/qemu/channels/KLEE-Fil01.com.redhat.rhevm.vdsm >Thread-15913501::DEBUG::2013-11-20 03:07:16,633::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /sbin/service ksmtuned retune' (cwd None) >Thread-15913518::INFO::2013-11-20 03:07:17,092::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:07:17,108::task::495::TaskManager.Task::(_debug) Task 14d61759-be24-4e42-be69-39ef5444c500: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:07:17,120::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`5709af13-0d8a-4b7a-ae59-43acd2daa53b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:07:17,121::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:07:17,135::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:07:17,148::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`5709af13-0d8a-4b7a-ae59-43acd2daa53b`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:07:17,164::task::495::TaskManager.Task::(_debug) Task 14d61759-be24-4e42-be69-39ef5444c500: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:07:17,176::task::495::TaskManager.Task::(_debug) Task 14d61759-be24-4e42-be69-39ef5444c500: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:07:17,182::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913518::DEBUG::2013-11-20 03:07:17,204::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15913501::DEBUG::2013-11-20 03:07:17,215::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913501::DEBUG::2013-11-20 03:07:17,427::vm::358::vm.Vm::(_startUnderlyingVm) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::_ongoingCreations released >Thread-15913510::DEBUG::2013-11-20 03:07:17,429::vm::346::vm.Vm::(_startUnderlyingVm) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::_ongoingCreations acquired >Thread-15913510::INFO::2013-11-20 03:07:17,436::libvirtvm::1061::vm.Vm::(_run) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::VM wrapper has started >Thread-15913510::INFO::2013-11-20 03:07:17,439::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913510::DEBUG::2013-11-20 03:07:17,441::task::495::TaskManager.Task::(_debug) Task a11f83b3-356c-4c77-9f46-1b6ba987df6b: moving from state init -> state preparing >Thread-15913510::DEBUG::2013-11-20 03:07:17,446::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`5520ddc5-423c-43ba-8021-db33274695dc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1690' at 'public_prepareVolume' >Thread-15913510::DEBUG::2013-11-20 03:07:17,447::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913510::DEBUG::2013-11-20 03:07:17,450::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:17,453::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`5520ddc5-423c-43ba-8021-db33274695dc`::Granted request >Dummy-15912651::DEBUG::2013-11-20 03:07:17,454::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.105188 s, 9.7 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:17,457::task::495::TaskManager.Task::(_debug) Task a11f83b3-356c-4c77-9f46-1b6ba987df6b: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913510::DEBUG::2013-11-20 03:07:17,458::task::495::TaskManager.Task::(_debug) Task a11f83b3-356c-4c77-9f46-1b6ba987df6b: ref 1 aborting False >Thread-15913510::DEBUG::2013-11-20 03:07:17,476::resourceManager::155::ResourceManager.Request::(__init__) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_imageNS.c4460480-32d3-469f-b0f4-c03bb6d74e41`ReqID=`ce99774b-8c3e-476a-84bf-0c9b90d6c626`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1695' at 'public_prepareVolume' >Thread-15913510::DEBUG::2013-11-20 03:07:17,477::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.c4460480-32d3-469f-b0f4-c03bb6d74e41' for lock type 'exclusive' >Thread-15913510::DEBUG::2013-11-20 03:07:17,480::lvm::480::OperationMutex::(_reloadlvs) Got the operational mutex >Thread-15913510::DEBUG::2013-11-20 03:07:17,483::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15913518::DEBUG::2013-11-20 03:07:18,209::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913521::DEBUG::2013-11-20 03:07:18,873::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4b8ed9c8-dfb1-4de4-935d-6ba4009567b9',) {} >Thread-15913510::DEBUG::2013-11-20 03:07:18,912::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913521::DEBUG::2013-11-20 03:07:18,917::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::CPU stats not available >Thread-15913521::DEBUG::2013-11-20 03:07:19,002::libvirtvm::202::vm.Vm::(_getNetworkStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Network stats not available >Thread-15913521::DEBUG::2013-11-20 03:07:19,028::libvirtvm::202::vm.Vm::(_getNetworkStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Network stats not available >Thread-15913510::DEBUG::2013-11-20 03:07:19,070::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=7 bs=512 if=/dev/772472cb-2429-466d-96cd-00b3301ef78a/metadata count=1' (cwd None) >Thread-15913521::DEBUG::2013-11-20 03:07:19,073::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913521::DEBUG::2013-11-20 03:07:19,128::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913518::DEBUG::2013-11-20 03:07:19,463::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913521::DEBUG::2013-11-20 03:07:19,469::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913518::DEBUG::2013-11-20 03:07:19,471::task::495::TaskManager.Task::(_debug) Task 14d61759-be24-4e42-be69-39ef5444c500: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913521::DEBUG::2013-11-20 03:07:19,474::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913518::DEBUG::2013-11-20 03:07:19,475::task::495::TaskManager.Task::(_debug) Task 14d61759-be24-4e42-be69-39ef5444c500: moving from state preparing -> state finished >Thread-15913521::DEBUG::2013-11-20 03:07:19,476::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '46387', 'displayIp': '0', 'displayPort': u'5902', 'session': 'Unknown', 'displaySecurePort': u'5903', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet1': {'state': 'unknown', 'macAddr': u'00:1a:4a:a8:64:05', 'speed': '1000', 'name': u'vnet1'}, u'vnet2': {'state': 'unknown', 'macAddr': u'00:1a:4a:a8:64:0b', 'speed': '1000', 'name': u'vnet2'}}, 'vmId': '4b8ed9c8-dfb1-4de4-935d-6ba4009567b9', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '30064771072', 'writeLatency': '0', 'imageID': 'b0980185-7de1-4e3b-bd76-4495169ab08b', 'flushLatency': '0', 'truesize': '30064771072'}, u'vdb': {'readLatency': '0', 'apparentsize': '826781204480', 'writeLatency': '0', 'imageID': '0b89347c-4654-42bb-a8d7-1a627130b7bd', 'flushLatency': '0', 'truesize': '826781204480'}}, 'boot': 'c', 'statsAge': '3.01', 'cpuIdle': '100.00', 'elapsedTime': '21', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '3601', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913518::DEBUG::2013-11-20 03:07:19,478::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Dummy-15912651::DEBUG::2013-11-20 03:07:19,479::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913518::DEBUG::2013-11-20 03:07:19,484::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:07:19,737::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:07:19,742::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:07:19,744::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913510::DEBUG::2013-11-20 03:07:19,745::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000353597 s, 1.4 MB/s\n'; <rc> = 0 >Thread-15913518::DEBUG::2013-11-20 03:07:19,746::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913510::DEBUG::2013-11-20 03:07:19,747::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000353597 s, 1.4 MB/s'], size: 512 >Thread-15913510::DEBUG::2013-11-20 03:07:19,749::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=7 bs=512 if=/dev/772472cb-2429-466d-96cd-00b3301ef78a/metadata count=1' (cwd None) >Thread-15913518::DEBUG::2013-11-20 03:07:19,750::task::495::TaskManager.Task::(_debug) Task 14d61759-be24-4e42-be69-39ef5444c500: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:07:19,752::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:07:20,129::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:07:20,131::task::495::TaskManager.Task::(_debug) Task 8a79cdf3-6016-44fb-90e9-69c998dbbd64: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:07:20,134::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`355c8b57-248c-43d6-a127-5c697996e646`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:07:20,136::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:07:20,138::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:07:20,139::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`355c8b57-248c-43d6-a127-5c697996e646`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:07:20,145::task::495::TaskManager.Task::(_debug) Task 8a79cdf3-6016-44fb-90e9-69c998dbbd64: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:07:20,146::task::495::TaskManager.Task::(_debug) Task 8a79cdf3-6016-44fb-90e9-69c998dbbd64: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:07:20,152::task::495::TaskManager.Task::(_debug) Task 8a79cdf3-6016-44fb-90e9-69c998dbbd64: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:07:20,153::task::495::TaskManager.Task::(_debug) Task 8a79cdf3-6016-44fb-90e9-69c998dbbd64: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:07:20,154::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:07:20,155::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:07:20,156::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:07:20,157::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:07:20,158::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:07:20,158::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:07:20,160::task::495::TaskManager.Task::(_debug) Task 8a79cdf3-6016-44fb-90e9-69c998dbbd64: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:07:20,161::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:07:20,286::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.187592 s, 5.5 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:20,311::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.00035512 s, 1.4 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:20,312::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.00035512 s, 1.4 MB/s'], size: 512 >Thread-15913510::INFO::2013-11-20 03:07:20,313::image::340::Storage.Image::(getChain) sdUUID=772472cb-2429-466d-96cd-00b3301ef78a imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 chain=[<storage.blockVolume.BlockVolume instance at 0x7f48841837a0>] >Thread-15913510::DEBUG::2013-11-20 03:07:20,314::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --autobackup n --available y 772472cb-2429-466d-96cd-00b3301ef78a/e6984064-e09e-4c0c-8b51-a8a1711994cf' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:21,183::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:21,184::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:21,185::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:21,187::resourceManager::155::ResourceManager.Request::(__init__) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.e6984064-e09e-4c0c-8b51-a8a1711994cf`ReqID=`bc967614-1770-4a2a-9cb1-1d44f4e518f0`::Request was made in '/usr/share/vdsm/storage/resourceFactories.py' line '149' at '__getResourceCandidatesList' >Thread-15913510::DEBUG::2013-11-20 03:07:21,187::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.e6984064-e09e-4c0c-8b51-a8a1711994cf' for lock type 'exclusive' >Thread-15913510::DEBUG::2013-11-20 03:07:21,188::resourceManager::505::ResourceManager::(registerResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.e6984064-e09e-4c0c-8b51-a8a1711994cf' is free. Now locking as 'exclusive' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:21,189::resourceManager::192::ResourceManager.Request::(grant) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.e6984064-e09e-4c0c-8b51-a8a1711994cf`ReqID=`bc967614-1770-4a2a-9cb1-1d44f4e518f0`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:21,190::resourceManager::505::ResourceManager::(registerResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.c4460480-32d3-469f-b0f4-c03bb6d74e41' is free. Now locking as 'exclusive' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:21,192::resourceManager::192::ResourceManager.Request::(grant) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_imageNS.c4460480-32d3-469f-b0f4-c03bb6d74e41`ReqID=`ce99774b-8c3e-476a-84bf-0c9b90d6c626`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:21,194::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:21,195::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15913522::DEBUG::2013-11-20 03:07:21,412::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913522::INFO::2013-11-20 03:07:21,414::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913522::DEBUG::2013-11-20 03:07:21,432::task::495::TaskManager.Task::(_debug) Task 577f2649-c035-4fca-bd1d-02fe70513ab6: moving from state init -> state preparing >Thread-15913522::DEBUG::2013-11-20 03:07:21,433::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913522::DEBUG::2013-11-20 03:07:21,434::task::495::TaskManager.Task::(_debug) Task 577f2649-c035-4fca-bd1d-02fe70513ab6: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913522::DEBUG::2013-11-20 03:07:21,434::task::495::TaskManager.Task::(_debug) Task 577f2649-c035-4fca-bd1d-02fe70513ab6: moving from state preparing -> state finished >Thread-15913522::DEBUG::2013-11-20 03:07:21,435::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913522::DEBUG::2013-11-20 03:07:21,436::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913522::DEBUG::2013-11-20 03:07:21,436::task::495::TaskManager.Task::(_debug) Task 577f2649-c035-4fca-bd1d-02fe70513ab6: ref 0 aborting False >Thread-15913522::INFO::2013-11-20 03:07:21,437::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913523::DEBUG::2013-11-20 03:07:21,443::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913523::INFO::2013-11-20 03:07:21,444::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913523::DEBUG::2013-11-20 03:07:21,444::task::495::TaskManager.Task::(_debug) Task cc8432f1-915d-4a2c-bf80-ed18ff78c1fa: moving from state init -> state preparing >Thread-15913523::DEBUG::2013-11-20 03:07:21,446::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`60197461-af0a-4e4c-8d1d-fafd7f0112d5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913523::DEBUG::2013-11-20 03:07:21,447::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913523::DEBUG::2013-11-20 03:07:21,448::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913523::DEBUG::2013-11-20 03:07:21,448::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`60197461-af0a-4e4c-8d1d-fafd7f0112d5`::Granted request >Thread-15913523::DEBUG::2013-11-20 03:07:21,449::task::495::TaskManager.Task::(_debug) Task cc8432f1-915d-4a2c-bf80-ed18ff78c1fa: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913523::DEBUG::2013-11-20 03:07:21,450::task::495::TaskManager.Task::(_debug) Task cc8432f1-915d-4a2c-bf80-ed18ff78c1fa: ref 1 aborting False >Thread-15913523::WARNING::2013-11-20 03:07:21,452::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913523::DEBUG::2013-11-20 03:07:21,453::task::495::TaskManager.Task::(_debug) Task cc8432f1-915d-4a2c-bf80-ed18ff78c1fa: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913523::DEBUG::2013-11-20 03:07:21,454::task::495::TaskManager.Task::(_debug) Task cc8432f1-915d-4a2c-bf80-ed18ff78c1fa: moving from state preparing -> state finished >Thread-15913523::DEBUG::2013-11-20 03:07:21,455::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913523::DEBUG::2013-11-20 03:07:21,455::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913523::DEBUG::2013-11-20 03:07:21,456::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913523::DEBUG::2013-11-20 03:07:21,457::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913523::DEBUG::2013-11-20 03:07:21,457::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913523::DEBUG::2013-11-20 03:07:21,458::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913523::DEBUG::2013-11-20 03:07:21,458::task::495::TaskManager.Task::(_debug) Task cc8432f1-915d-4a2c-bf80-ed18ff78c1fa: ref 0 aborting False >Thread-15913523::INFO::2013-11-20 03:07:21,459::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913510::DEBUG::2013-11-20 03:07:21,893::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:21,932::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:21,936::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --refresh 772472cb-2429-466d-96cd-00b3301ef78a/e6984064-e09e-4c0c-8b51-a8a1711994cf' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:22,300::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:22,584::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:22,586::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:22,587::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913510::INFO::2013-11-20 03:07:22,587::volume::517::Storage.Volume::(prepare) Volume: preparing volume 772472cb-2429-466d-96cd-00b3301ef78a/e6984064-e09e-4c0c-8b51-a8a1711994cf >Thread-15913510::DEBUG::2013-11-20 03:07:22,588::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:22,589::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15913525::DEBUG::2013-11-20 03:07:22,820::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4b8ed9c8-dfb1-4de4-935d-6ba4009567b9',) {} >Thread-15913525::DEBUG::2013-11-20 03:07:22,822::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::CPU stats not available >Thread-15913525::DEBUG::2013-11-20 03:07:22,823::libvirtvm::202::vm.Vm::(_getNetworkStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Network stats not available >Thread-15913525::DEBUG::2013-11-20 03:07:22,824::libvirtvm::202::vm.Vm::(_getNetworkStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Network stats not available >Thread-15913525::DEBUG::2013-11-20 03:07:22,824::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913525::DEBUG::2013-11-20 03:07:22,825::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913525::DEBUG::2013-11-20 03:07:22,826::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913525::DEBUG::2013-11-20 03:07:22,827::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913525::DEBUG::2013-11-20 03:07:22,828::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '46387', 'displayIp': '0', 'displayPort': u'5902', 'session': 'Unknown', 'displaySecurePort': u'5903', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet1': {'state': 'unknown', 'macAddr': u'00:1a:4a:a8:64:05', 'speed': '1000', 'name': u'vnet1'}, u'vnet2': {'state': 'unknown', 'macAddr': u'00:1a:4a:a8:64:0b', 'speed': '1000', 'name': u'vnet2'}}, 'vmId': '4b8ed9c8-dfb1-4de4-935d-6ba4009567b9', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '30064771072', 'writeLatency': '0', 'imageID': 'b0980185-7de1-4e3b-bd76-4495169ab08b', 'flushLatency': '0', 'truesize': '30064771072'}, u'vdb': {'readLatency': '0', 'apparentsize': '826781204480', 'writeLatency': '0', 'imageID': '0b89347c-4654-42bb-a8d7-1a627130b7bd', 'flushLatency': '0', 'truesize': '826781204480'}}, 'boot': 'c', 'statsAge': '0.62', 'cpuIdle': '100.00', 'elapsedTime': '25', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '3601', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:07:22,950::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.246242 s, 4.2 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:23,265::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:23,289::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:23,290::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=7 bs=512 if=/dev/772472cb-2429-466d-96cd-00b3301ef78a/metadata count=1' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:23,688::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.0003183 s, 1.6 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:23,689::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.0003183 s, 1.6 MB/s'], size: 512 >Thread-15913510::DEBUG::2013-11-20 03:07:23,690::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=7 bs=512 if=/dev/772472cb-2429-466d-96cd-00b3301ef78a/metadata count=1' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:24,082::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.00032937 s, 1.6 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:24,083::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.00032937 s, 1.6 MB/s'], size: 512 >Thread-15913510::DEBUG::2013-11-20 03:07:24,084::resourceManager::155::ResourceManager.Request::(__init__) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.e6984064-e09e-4c0c-8b51-a8a1711994cf`ReqID=`b9f74782-7b06-441d-a176-903d0ae7d74f`::Request was made in '/usr/share/vdsm/storage/volume.py' line '533' at 'prepare' >Thread-15913510::DEBUG::2013-11-20 03:07:24,085::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.e6984064-e09e-4c0c-8b51-a8a1711994cf' for lock type 'exclusive' >Thread-15913510::DEBUG::2013-11-20 03:07:24,086::resourceManager::505::ResourceManager::(registerResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.e6984064-e09e-4c0c-8b51-a8a1711994cf' is free. Now locking as 'exclusive' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:24,087::resourceManager::192::ResourceManager.Request::(grant) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.e6984064-e09e-4c0c-8b51-a8a1711994cf`ReqID=`b9f74782-7b06-441d-a176-903d0ae7d74f`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:24,089::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=7 bs=512 if=/dev/772472cb-2429-466d-96cd-00b3301ef78a/metadata count=1' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:24,486::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000221009 s, 2.3 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:24,488::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000221009 s, 2.3 MB/s'], size: 512 >Thread-15913510::DEBUG::2013-11-20 03:07:24,490::task::495::TaskManager.Task::(_debug) Task a11f83b3-356c-4c77-9f46-1b6ba987df6b: finished: None >Thread-15913510::DEBUG::2013-11-20 03:07:24,490::task::495::TaskManager.Task::(_debug) Task a11f83b3-356c-4c77-9f46-1b6ba987df6b: moving from state preparing -> state finished >Thread-15913510::DEBUG::2013-11-20 03:07:24,494::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913510::DEBUG::2013-11-20 03:07:24,495::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913510::DEBUG::2013-11-20 03:07:24,496::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913510::DEBUG::2013-11-20 03:07:24,497::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913510::DEBUG::2013-11-20 03:07:24,497::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913510::DEBUG::2013-11-20 03:07:24,498::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913510::DEBUG::2013-11-20 03:07:24,499::task::495::TaskManager.Task::(_debug) Task a11f83b3-356c-4c77-9f46-1b6ba987df6b: ref 0 aborting False >Thread-15913510::INFO::2013-11-20 03:07:24,500::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15913510::INFO::2013-11-20 03:07:24,501::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913510::DEBUG::2013-11-20 03:07:24,501::task::495::TaskManager.Task::(_debug) Task ca93d5ff-b466-472b-86db-24ec0ad4708e: moving from state init -> state preparing >Thread-15913510::DEBUG::2013-11-20 03:07:24,503::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`db7ae64f-ec43-4d1b-8a04-8ce7ac8ccdb1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1666' at 'public_getVolumePath' >Thread-15913510::DEBUG::2013-11-20 03:07:24,504::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913510::DEBUG::2013-11-20 03:07:24,505::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:24,505::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`db7ae64f-ec43-4d1b-8a04-8ce7ac8ccdb1`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:24,506::task::495::TaskManager.Task::(_debug) Task ca93d5ff-b466-472b-86db-24ec0ad4708e: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913510::DEBUG::2013-11-20 03:07:24,507::task::495::TaskManager.Task::(_debug) Task ca93d5ff-b466-472b-86db-24ec0ad4708e: ref 1 aborting False >Thread-15913510::DEBUG::2013-11-20 03:07:24,508::task::495::TaskManager.Task::(_debug) Task ca93d5ff-b466-472b-86db-24ec0ad4708e: finished: {'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/772472cb-2429-466d-96cd-00b3301ef78a/images/c4460480-32d3-469f-b0f4-c03bb6d74e41/e6984064-e09e-4c0c-8b51-a8a1711994cf'} >Thread-15913510::DEBUG::2013-11-20 03:07:24,509::task::495::TaskManager.Task::(_debug) Task ca93d5ff-b466-472b-86db-24ec0ad4708e: moving from state preparing -> state finished >Thread-15913510::DEBUG::2013-11-20 03:07:24,510::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913510::DEBUG::2013-11-20 03:07:24,510::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913510::DEBUG::2013-11-20 03:07:24,511::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913510::DEBUG::2013-11-20 03:07:24,512::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913510::DEBUG::2013-11-20 03:07:24,513::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913510::DEBUG::2013-11-20 03:07:24,513::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913510::DEBUG::2013-11-20 03:07:24,514::task::495::TaskManager.Task::(_debug) Task ca93d5ff-b466-472b-86db-24ec0ad4708e: ref 0 aborting False >Thread-15913510::INFO::2013-11-20 03:07:24,515::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, Return response: {'status': {'message': 'OK', 'code': 0}, 'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/772472cb-2429-466d-96cd-00b3301ef78a/images/c4460480-32d3-469f-b0f4-c03bb6d74e41/e6984064-e09e-4c0c-8b51-a8a1711994cf'} >Thread-15913510::INFO::2013-11-20 03:07:24,515::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913510::DEBUG::2013-11-20 03:07:24,516::task::495::TaskManager.Task::(_debug) Task 24b989cb-d7f3-43d9-8f2f-e17729224734: moving from state init -> state preparing >Thread-15913510::DEBUG::2013-11-20 03:07:24,517::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`627d5507-97d4-4c38-a267-7eccd932a56a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1690' at 'public_prepareVolume' >Thread-15913510::DEBUG::2013-11-20 03:07:24,518::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913510::DEBUG::2013-11-20 03:07:24,519::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:24,520::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`627d5507-97d4-4c38-a267-7eccd932a56a`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:24,530::task::495::TaskManager.Task::(_debug) Task 24b989cb-d7f3-43d9-8f2f-e17729224734: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913510::DEBUG::2013-11-20 03:07:24,531::task::495::TaskManager.Task::(_debug) Task 24b989cb-d7f3-43d9-8f2f-e17729224734: ref 1 aborting False >Thread-15913510::DEBUG::2013-11-20 03:07:24,533::resourceManager::155::ResourceManager.Request::(__init__) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_imageNS.df8b790c-c762-4feb-9378-d64a321ce43c`ReqID=`bc64df71-203d-46b4-bf3d-73c43284919a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1695' at 'public_prepareVolume' >Thread-15913510::DEBUG::2013-11-20 03:07:24,535::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.df8b790c-c762-4feb-9378-d64a321ce43c' for lock type 'exclusive' >Thread-15913510::DEBUG::2013-11-20 03:07:24,536::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:24,538::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:24,965::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:25,139::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:25,158::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:25,166::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=6 bs=512 if=/dev/772472cb-2429-466d-96cd-00b3301ef78a/metadata count=1' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:25,563::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.140498 s, 7.3 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:25,577::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.0001589 s, 3.2 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:25,577::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.0001589 s, 3.2 MB/s'], size: 512 >Thread-15913510::DEBUG::2013-11-20 03:07:25,578::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=6 bs=512 if=/dev/772472cb-2429-466d-96cd-00b3301ef78a/metadata count=1' (cwd None) >Thread-15913527::DEBUG::2013-11-20 03:07:25,856::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4b8ed9c8-dfb1-4de4-935d-6ba4009567b9',) {} >Thread-15913527::DEBUG::2013-11-20 03:07:25,857::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::CPU stats not available >Thread-15913527::DEBUG::2013-11-20 03:07:25,858::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913527::DEBUG::2013-11-20 03:07:25,859::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913527::DEBUG::2013-11-20 03:07:25,859::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913527::DEBUG::2013-11-20 03:07:25,860::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913527::DEBUG::2013-11-20 03:07:25,861::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '46387', 'displayIp': '0', 'displayPort': u'5902', 'session': 'Unknown', 'displaySecurePort': u'5903', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet1': {'macAddr': u'00:1a:4a:a8:64:05', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet1'}, u'vnet2': {'macAddr': u'00:1a:4a:a8:64:0b', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet2'}}, 'vmId': '4b8ed9c8-dfb1-4de4-935d-6ba4009567b9', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '30064771072', 'writeLatency': '0', 'imageID': 'b0980185-7de1-4e3b-bd76-4495169ab08b', 'flushLatency': '0', 'truesize': '30064771072'}, u'vdb': {'readLatency': '0', 'apparentsize': '826781204480', 'writeLatency': '0', 'imageID': '0b89347c-4654-42bb-a8d7-1a627130b7bd', 'flushLatency': '0', 'truesize': '826781204480'}}, 'boot': 'c', 'statsAge': '0.42', 'cpuIdle': '100.00', 'elapsedTime': '28', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '3601', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913510::DEBUG::2013-11-20 03:07:25,917::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000271766 s, 1.9 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:25,917::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000271766 s, 1.9 MB/s'], size: 512 >Thread-15913510::INFO::2013-11-20 03:07:25,918::image::340::Storage.Image::(getChain) sdUUID=772472cb-2429-466d-96cd-00b3301ef78a imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c chain=[<storage.blockVolume.BlockVolume instance at 0x7f4884183998>] >Thread-15913510::DEBUG::2013-11-20 03:07:25,919::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --autobackup n --available y 772472cb-2429-466d-96cd-00b3301ef78a/2e903e43-1016-4339-a7f3-c5723a995725' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:26,592::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:26,594::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:26,595::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:26,596::resourceManager::155::ResourceManager.Request::(__init__) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.2e903e43-1016-4339-a7f3-c5723a995725`ReqID=`8c78613c-cf3a-4f65-83a7-5d7ea4cc7c16`::Request was made in '/usr/share/vdsm/storage/resourceFactories.py' line '149' at '__getResourceCandidatesList' >Thread-15913510::DEBUG::2013-11-20 03:07:26,597::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.2e903e43-1016-4339-a7f3-c5723a995725' for lock type 'exclusive' >Thread-15913510::DEBUG::2013-11-20 03:07:26,597::resourceManager::505::ResourceManager::(registerResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.2e903e43-1016-4339-a7f3-c5723a995725' is free. Now locking as 'exclusive' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:26,598::resourceManager::192::ResourceManager.Request::(grant) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.2e903e43-1016-4339-a7f3-c5723a995725`ReqID=`8c78613c-cf3a-4f65-83a7-5d7ea4cc7c16`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:26,600::resourceManager::505::ResourceManager::(registerResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.df8b790c-c762-4feb-9378-d64a321ce43c' is free. Now locking as 'exclusive' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:26,600::resourceManager::192::ResourceManager.Request::(grant) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_imageNS.df8b790c-c762-4feb-9378-d64a321ce43c`ReqID=`bc64df71-203d-46b4-bf3d-73c43284919a`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:26,602::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:26,603::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15912392::INFO::2013-11-20 03:07:26,802::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:07:26,803::task::495::TaskManager.Task::(_debug) Task 73feca8b-63dc-4d30-b69f-f4ecb94d1d58: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:07:26,805::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c260c3f2-b3a2-4f9f-a10a-2438286ac70f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:07:26,805::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:07:26,806::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:07:26,806::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c260c3f2-b3a2-4f9f-a10a-2438286ac70f`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:07:26,807::task::495::TaskManager.Task::(_debug) Task 73feca8b-63dc-4d30-b69f-f4ecb94d1d58: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:07:26,808::task::495::TaskManager.Task::(_debug) Task 73feca8b-63dc-4d30-b69f-f4ecb94d1d58: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:07:26,808::task::495::TaskManager.Task::(_debug) Task 73feca8b-63dc-4d30-b69f-f4ecb94d1d58: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:07:26,808::task::495::TaskManager.Task::(_debug) Task 73feca8b-63dc-4d30-b69f-f4ecb94d1d58: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:07:26,809::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:07:26,809::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:07:26,810::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:07:26,810::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:07:26,810::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:07:26,811::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:07:26,811::task::495::TaskManager.Task::(_debug) Task 73feca8b-63dc-4d30-b69f-f4ecb94d1d58: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:07:26,811::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913510::DEBUG::2013-11-20 03:07:27,279::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:27,307::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:27,312::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --refresh 772472cb-2429-466d-96cd-00b3301ef78a/2e903e43-1016-4339-a7f3-c5723a995725' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:27,578::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:28,090::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.048421 s, 21.1 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:28,117::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:28,118::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:28,119::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913510::INFO::2013-11-20 03:07:28,120::volume::517::Storage.Volume::(prepare) Volume: preparing volume 772472cb-2429-466d-96cd-00b3301ef78a/2e903e43-1016-4339-a7f3-c5723a995725 >Thread-15913510::DEBUG::2013-11-20 03:07:28,120::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:28,126::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:28,773::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:28,787::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:28,788::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=6 bs=512 if=/dev/772472cb-2429-466d-96cd-00b3301ef78a/metadata count=1' (cwd None) >Thread-15913529::DEBUG::2013-11-20 03:07:28,982::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4b8ed9c8-dfb1-4de4-935d-6ba4009567b9',) {} >Thread-15913529::DEBUG::2013-11-20 03:07:28,983::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::CPU stats not available >Thread-15913529::DEBUG::2013-11-20 03:07:28,984::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913529::DEBUG::2013-11-20 03:07:28,985::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913529::DEBUG::2013-11-20 03:07:28,985::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913529::DEBUG::2013-11-20 03:07:28,986::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913529::DEBUG::2013-11-20 03:07:28,987::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '46387', 'displayIp': '0', 'displayPort': u'5902', 'session': 'Unknown', 'displaySecurePort': u'5903', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet1': {'macAddr': u'00:1a:4a:a8:64:05', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet1'}, u'vnet2': {'macAddr': u'00:1a:4a:a8:64:0b', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet2'}}, 'vmId': '4b8ed9c8-dfb1-4de4-935d-6ba4009567b9', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '30064771072', 'writeLatency': '0', 'imageID': 'b0980185-7de1-4e3b-bd76-4495169ab08b', 'flushLatency': '0', 'truesize': '30064771072'}, u'vdb': {'readLatency': '0', 'apparentsize': '826781204480', 'writeLatency': '0', 'imageID': '0b89347c-4654-42bb-a8d7-1a627130b7bd', 'flushLatency': '0', 'truesize': '826781204480'}}, 'boot': 'c', 'statsAge': '0.53', 'cpuIdle': '100.00', 'elapsedTime': '31', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '3601', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913510::DEBUG::2013-11-20 03:07:29,116::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000246451 s, 2.1 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:29,117::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000246451 s, 2.1 MB/s'], size: 512 >Thread-15913510::DEBUG::2013-11-20 03:07:29,117::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=6 bs=512 if=/dev/772472cb-2429-466d-96cd-00b3301ef78a/metadata count=1' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:29,499::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000279414 s, 1.8 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:29,501::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000279414 s, 1.8 MB/s'], size: 512 >Thread-15913510::DEBUG::2013-11-20 03:07:29,503::resourceManager::155::ResourceManager.Request::(__init__) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.2e903e43-1016-4339-a7f3-c5723a995725`ReqID=`37ad25ea-25d0-47de-ba4a-7668f32ea2ce`::Request was made in '/usr/share/vdsm/storage/volume.py' line '533' at 'prepare' >Thread-15913510::DEBUG::2013-11-20 03:07:29,504::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.2e903e43-1016-4339-a7f3-c5723a995725' for lock type 'exclusive' >Thread-15913510::DEBUG::2013-11-20 03:07:29,506::resourceManager::505::ResourceManager::(registerResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.2e903e43-1016-4339-a7f3-c5723a995725' is free. Now locking as 'exclusive' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:29,507::resourceManager::192::ResourceManager.Request::(grant) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.2e903e43-1016-4339-a7f3-c5723a995725`ReqID=`37ad25ea-25d0-47de-ba4a-7668f32ea2ce`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:29,509::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=6 bs=512 if=/dev/772472cb-2429-466d-96cd-00b3301ef78a/metadata count=1' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:29,932::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000196792 s, 2.6 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:29,933::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000196792 s, 2.6 MB/s'], size: 512 >Thread-15913510::DEBUG::2013-11-20 03:07:29,935::task::495::TaskManager.Task::(_debug) Task 24b989cb-d7f3-43d9-8f2f-e17729224734: finished: None >Thread-15913510::DEBUG::2013-11-20 03:07:29,936::task::495::TaskManager.Task::(_debug) Task 24b989cb-d7f3-43d9-8f2f-e17729224734: moving from state preparing -> state finished >Thread-15913510::DEBUG::2013-11-20 03:07:29,936::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913510::DEBUG::2013-11-20 03:07:29,937::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913510::DEBUG::2013-11-20 03:07:29,938::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913510::DEBUG::2013-11-20 03:07:29,939::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913510::DEBUG::2013-11-20 03:07:29,939::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913510::DEBUG::2013-11-20 03:07:29,940::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913510::DEBUG::2013-11-20 03:07:29,940::task::495::TaskManager.Task::(_debug) Task 24b989cb-d7f3-43d9-8f2f-e17729224734: ref 0 aborting False >Thread-15913510::INFO::2013-11-20 03:07:29,941::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15913510::INFO::2013-11-20 03:07:29,942::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913510::DEBUG::2013-11-20 03:07:29,942::task::495::TaskManager.Task::(_debug) Task 2418428e-77e7-46ab-a63d-934a29257383: moving from state init -> state preparing >Thread-15913510::DEBUG::2013-11-20 03:07:29,944::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`734a3000-5d6e-4f8c-9f8d-b2b71e544af2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1666' at 'public_getVolumePath' >Thread-15913510::DEBUG::2013-11-20 03:07:29,944::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913510::DEBUG::2013-11-20 03:07:29,945::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:29,945::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`734a3000-5d6e-4f8c-9f8d-b2b71e544af2`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:29,946::task::495::TaskManager.Task::(_debug) Task 2418428e-77e7-46ab-a63d-934a29257383: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913510::DEBUG::2013-11-20 03:07:29,947::task::495::TaskManager.Task::(_debug) Task 2418428e-77e7-46ab-a63d-934a29257383: ref 1 aborting False >Thread-15913510::DEBUG::2013-11-20 03:07:29,948::task::495::TaskManager.Task::(_debug) Task 2418428e-77e7-46ab-a63d-934a29257383: finished: {'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/772472cb-2429-466d-96cd-00b3301ef78a/images/df8b790c-c762-4feb-9378-d64a321ce43c/2e903e43-1016-4339-a7f3-c5723a995725'} >Thread-15913510::DEBUG::2013-11-20 03:07:29,949::task::495::TaskManager.Task::(_debug) Task 2418428e-77e7-46ab-a63d-934a29257383: moving from state preparing -> state finished >Thread-15913510::DEBUG::2013-11-20 03:07:29,949::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913510::DEBUG::2013-11-20 03:07:29,950::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913510::DEBUG::2013-11-20 03:07:29,950::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913510::DEBUG::2013-11-20 03:07:29,951::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913510::DEBUG::2013-11-20 03:07:29,951::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913510::DEBUG::2013-11-20 03:07:29,952::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913510::DEBUG::2013-11-20 03:07:29,952::task::495::TaskManager.Task::(_debug) Task 2418428e-77e7-46ab-a63d-934a29257383: ref 0 aborting False >Thread-15913510::INFO::2013-11-20 03:07:29,953::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, Return response: {'status': {'message': 'OK', 'code': 0}, 'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/772472cb-2429-466d-96cd-00b3301ef78a/images/df8b790c-c762-4feb-9378-d64a321ce43c/2e903e43-1016-4339-a7f3-c5723a995725'} >Thread-15913510::INFO::2013-11-20 03:07:29,954::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913510::DEBUG::2013-11-20 03:07:29,954::task::495::TaskManager.Task::(_debug) Task 5b9741d7-a3ab-4621-ae82-1f3cdad89906: moving from state init -> state preparing >Thread-15913510::DEBUG::2013-11-20 03:07:29,955::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`6b146df4-bcc3-49c1-9576-d4df50c45155`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1690' at 'public_prepareVolume' >Thread-15913510::DEBUG::2013-11-20 03:07:29,956::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913510::DEBUG::2013-11-20 03:07:29,956::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:29,957::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`6b146df4-bcc3-49c1-9576-d4df50c45155`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:29,958::task::495::TaskManager.Task::(_debug) Task 5b9741d7-a3ab-4621-ae82-1f3cdad89906: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913510::DEBUG::2013-11-20 03:07:29,958::task::495::TaskManager.Task::(_debug) Task 5b9741d7-a3ab-4621-ae82-1f3cdad89906: ref 1 aborting False >Thread-15913510::DEBUG::2013-11-20 03:07:29,959::resourceManager::155::ResourceManager.Request::(__init__) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_imageNS.13efb45b-f4d2-4c94-bbdb-91ce9b9b7321`ReqID=`271150b3-a987-44f3-9b21-c01ebef1b045`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1695' at 'public_prepareVolume' >Thread-15913510::DEBUG::2013-11-20 03:07:29,960::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.13efb45b-f4d2-4c94-bbdb-91ce9b9b7321' for lock type 'exclusive' >Thread-15913510::DEBUG::2013-11-20 03:07:29,961::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:29,972::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:30,156::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:30,710::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:30,741::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:30,749::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=5 bs=512 if=/dev/772472cb-2429-466d-96cd-00b3301ef78a/metadata count=1' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:31,123::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.384285 s, 2.7 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:31,142::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000296711 s, 1.7 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:31,143::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000296711 s, 1.7 MB/s'], size: 512 >Thread-15913510::DEBUG::2013-11-20 03:07:31,151::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=5 bs=512 if=/dev/772472cb-2429-466d-96cd-00b3301ef78a/metadata count=1' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:31,596::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000702549 s, 729 kB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:31,598::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000702549 s, 729 kB/s'], size: 512 >Thread-15913510::INFO::2013-11-20 03:07:31,599::image::340::Storage.Image::(getChain) sdUUID=772472cb-2429-466d-96cd-00b3301ef78a imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 chain=[<storage.blockVolume.BlockVolume instance at 0x7f4884183680>] >Thread-15913510::DEBUG::2013-11-20 03:07:31,601::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --autobackup n --available y 772472cb-2429-466d-96cd-00b3301ef78a/088640f7-316c-4e2b-bfdc-0f64cf2913c8' (cwd None) >Thread-15913530::INFO::2013-11-20 03:07:32,011::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913530::DEBUG::2013-11-20 03:07:32,012::task::495::TaskManager.Task::(_debug) Task cfeb5266-6674-4310-9f4c-fd37f859485b: moving from state init -> state preparing >Thread-15913530::DEBUG::2013-11-20 03:07:32,013::task::495::TaskManager.Task::(_debug) Task cfeb5266-6674-4310-9f4c-fd37f859485b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000334978103638', 'lastCheck': 1384916847.912523, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.225172042847', 'lastCheck': 1384916844.7642519, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.240998983383', 'lastCheck': 1384916846.1448989, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00772905349731', 'lastCheck': 1384916845.4402161, 'valid': True, 'code': 0}} >Thread-15913530::DEBUG::2013-11-20 03:07:32,014::task::495::TaskManager.Task::(_debug) Task cfeb5266-6674-4310-9f4c-fd37f859485b: moving from state preparing -> state finished >Thread-15913530::DEBUG::2013-11-20 03:07:32,014::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913530::DEBUG::2013-11-20 03:07:32,015::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913530::DEBUG::2013-11-20 03:07:32,015::task::495::TaskManager.Task::(_debug) Task cfeb5266-6674-4310-9f4c-fd37f859485b: ref 0 aborting False >Thread-15913530::INFO::2013-11-20 03:07:32,016::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000334978103638', 'lastCheck': 1384916847.912523, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.225172042847', 'lastCheck': 1384916844.7642519, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.240998983383', 'lastCheck': 1384916846.1448989, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00772905349731', 'lastCheck': 1384916845.4402161, 'valid': True, 'code': 0}} >Thread-15913531::DEBUG::2013-11-20 03:07:32,030::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::CPU stats not available >Thread-15913531::DEBUG::2013-11-20 03:07:32,030::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913531::DEBUG::2013-11-20 03:07:32,031::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913531::DEBUG::2013-11-20 03:07:32,031::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913531::DEBUG::2013-11-20 03:07:32,033::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913532::DEBUG::2013-11-20 03:07:32,053::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913532::INFO::2013-11-20 03:07:32,054::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913532::DEBUG::2013-11-20 03:07:32,055::task::495::TaskManager.Task::(_debug) Task 87a54070-bf4a-423c-8f26-d902002437d9: moving from state init -> state preparing >Thread-15913532::DEBUG::2013-11-20 03:07:32,055::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913532::DEBUG::2013-11-20 03:07:32,056::task::495::TaskManager.Task::(_debug) Task 87a54070-bf4a-423c-8f26-d902002437d9: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913532::DEBUG::2013-11-20 03:07:32,056::task::495::TaskManager.Task::(_debug) Task 87a54070-bf4a-423c-8f26-d902002437d9: moving from state preparing -> state finished >Thread-15913532::DEBUG::2013-11-20 03:07:32,057::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913532::DEBUG::2013-11-20 03:07:32,057::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913532::DEBUG::2013-11-20 03:07:32,058::task::495::TaskManager.Task::(_debug) Task 87a54070-bf4a-423c-8f26-d902002437d9: ref 0 aborting False >Thread-15913532::INFO::2013-11-20 03:07:32,059::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913533::DEBUG::2013-11-20 03:07:32,063::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913533::INFO::2013-11-20 03:07:32,064::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913533::DEBUG::2013-11-20 03:07:32,064::task::495::TaskManager.Task::(_debug) Task aff89841-7c4a-4bc0-a0ca-6c32d4205b70: moving from state init -> state preparing >Thread-15913533::DEBUG::2013-11-20 03:07:32,065::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9928261b-6e46-4b47-812f-31187b818477`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913533::DEBUG::2013-11-20 03:07:32,066::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913533::DEBUG::2013-11-20 03:07:32,067::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913533::DEBUG::2013-11-20 03:07:32,067::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9928261b-6e46-4b47-812f-31187b818477`::Granted request >Thread-15913533::DEBUG::2013-11-20 03:07:32,068::task::495::TaskManager.Task::(_debug) Task aff89841-7c4a-4bc0-a0ca-6c32d4205b70: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913533::DEBUG::2013-11-20 03:07:32,069::task::495::TaskManager.Task::(_debug) Task aff89841-7c4a-4bc0-a0ca-6c32d4205b70: ref 1 aborting False >Thread-15913533::WARNING::2013-11-20 03:07:32,071::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913533::DEBUG::2013-11-20 03:07:32,072::task::495::TaskManager.Task::(_debug) Task aff89841-7c4a-4bc0-a0ca-6c32d4205b70: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913533::DEBUG::2013-11-20 03:07:32,072::task::495::TaskManager.Task::(_debug) Task aff89841-7c4a-4bc0-a0ca-6c32d4205b70: moving from state preparing -> state finished >Thread-15913533::DEBUG::2013-11-20 03:07:32,073::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913533::DEBUG::2013-11-20 03:07:32,073::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913533::DEBUG::2013-11-20 03:07:32,074::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913533::DEBUG::2013-11-20 03:07:32,075::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913533::DEBUG::2013-11-20 03:07:32,075::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913533::DEBUG::2013-11-20 03:07:32,076::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913533::DEBUG::2013-11-20 03:07:32,076::task::495::TaskManager.Task::(_debug) Task aff89841-7c4a-4bc0-a0ca-6c32d4205b70: ref 0 aborting False >Thread-15913533::INFO::2013-11-20 03:07:32,077::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913510::DEBUG::2013-11-20 03:07:32,383::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:32,384::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:32,385::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:32,386::resourceManager::155::ResourceManager.Request::(__init__) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.088640f7-316c-4e2b-bfdc-0f64cf2913c8`ReqID=`2e2ab8cd-2af8-45cb-9048-a361977d94d2`::Request was made in '/usr/share/vdsm/storage/resourceFactories.py' line '149' at '__getResourceCandidatesList' >Thread-15913510::DEBUG::2013-11-20 03:07:32,387::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.088640f7-316c-4e2b-bfdc-0f64cf2913c8' for lock type 'exclusive' >Thread-15913510::DEBUG::2013-11-20 03:07:32,388::resourceManager::505::ResourceManager::(registerResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.088640f7-316c-4e2b-bfdc-0f64cf2913c8' is free. Now locking as 'exclusive' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:32,389::resourceManager::192::ResourceManager.Request::(grant) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.088640f7-316c-4e2b-bfdc-0f64cf2913c8`ReqID=`2e2ab8cd-2af8-45cb-9048-a361977d94d2`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:32,390::resourceManager::505::ResourceManager::(registerResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.13efb45b-f4d2-4c94-bbdb-91ce9b9b7321' is free. Now locking as 'exclusive' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:32,391::resourceManager::192::ResourceManager.Request::(grant) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_imageNS.13efb45b-f4d2-4c94-bbdb-91ce9b9b7321`ReqID=`271150b3-a987-44f3-9b21-c01ebef1b045`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:32,392::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:32,393::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:33,067::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:33,091::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:33,092::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --refresh 772472cb-2429-466d-96cd-00b3301ef78a/088640f7-316c-4e2b-bfdc-0f64cf2913c8' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:33,289::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:33,829::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:33,831::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:33,833::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913510::INFO::2013-11-20 03:07:33,833::volume::517::Storage.Volume::(prepare) Volume: preparing volume 772472cb-2429-466d-96cd-00b3301ef78a/088640f7-316c-4e2b-bfdc-0f64cf2913c8 >Thread-15913510::DEBUG::2013-11-20 03:07:33,835::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:33,837::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:34,188::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.337543 s, 3.0 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:34,534::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:34,559::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:34,561::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=5 bs=512 if=/dev/772472cb-2429-466d-96cd-00b3301ef78a/metadata count=1' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:34,962::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000196947 s, 2.6 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:34,963::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000196947 s, 2.6 MB/s'], size: 512 >Thread-15913510::DEBUG::2013-11-20 03:07:34,964::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=5 bs=512 if=/dev/772472cb-2429-466d-96cd-00b3301ef78a/metadata count=1' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:35,352::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000221347 s, 2.3 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:35,353::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000221347 s, 2.3 MB/s'], size: 512 >Thread-15913510::DEBUG::2013-11-20 03:07:35,354::resourceManager::155::ResourceManager.Request::(__init__) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.088640f7-316c-4e2b-bfdc-0f64cf2913c8`ReqID=`1b8c91d9-a270-4fdc-9cd5-b2e9cd5892ee`::Request was made in '/usr/share/vdsm/storage/volume.py' line '533' at 'prepare' >Thread-15913510::DEBUG::2013-11-20 03:07:35,355::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.088640f7-316c-4e2b-bfdc-0f64cf2913c8' for lock type 'exclusive' >Thread-15913510::DEBUG::2013-11-20 03:07:35,355::resourceManager::505::ResourceManager::(registerResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.088640f7-316c-4e2b-bfdc-0f64cf2913c8' is free. Now locking as 'exclusive' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:35,356::resourceManager::192::ResourceManager.Request::(grant) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.088640f7-316c-4e2b-bfdc-0f64cf2913c8`ReqID=`1b8c91d9-a270-4fdc-9cd5-b2e9cd5892ee`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:35,357::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=5 bs=512 if=/dev/772472cb-2429-466d-96cd-00b3301ef78a/metadata count=1' (cwd None) >Thread-15913535::DEBUG::2013-11-20 03:07:35,727::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4b8ed9c8-dfb1-4de4-935d-6ba4009567b9',) {} >Thread-15913535::DEBUG::2013-11-20 03:07:35,728::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::CPU stats not available >Thread-15913535::DEBUG::2013-11-20 03:07:35,729::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913535::DEBUG::2013-11-20 03:07:35,729::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913535::DEBUG::2013-11-20 03:07:35,730::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913535::DEBUG::2013-11-20 03:07:35,730::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913535::DEBUG::2013-11-20 03:07:35,731::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '46387', 'displayIp': '0', 'displayPort': u'5902', 'session': 'Unknown', 'displaySecurePort': u'5903', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet1': {'macAddr': u'00:1a:4a:a8:64:05', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet1'}, u'vnet2': {'macAddr': u'00:1a:4a:a8:64:0b', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet2'}}, 'vmId': '4b8ed9c8-dfb1-4de4-935d-6ba4009567b9', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '30064771072', 'writeLatency': '0', 'imageID': 'b0980185-7de1-4e3b-bd76-4495169ab08b', 'flushLatency': '0', 'truesize': '30064771072'}, u'vdb': {'readLatency': '0', 'apparentsize': '826781204480', 'writeLatency': '0', 'imageID': '0b89347c-4654-42bb-a8d7-1a627130b7bd', 'flushLatency': '0', 'truesize': '826781204480'}}, 'boot': 'c', 'statsAge': '0.93', 'cpuIdle': '100.00', 'elapsedTime': '38', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '3601', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913510::DEBUG::2013-11-20 03:07:35,736::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000146744 s, 3.5 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:35,737::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000146744 s, 3.5 MB/s'], size: 512 >Thread-15913510::DEBUG::2013-11-20 03:07:35,741::task::495::TaskManager.Task::(_debug) Task 5b9741d7-a3ab-4621-ae82-1f3cdad89906: finished: None >Thread-15913510::DEBUG::2013-11-20 03:07:35,742::task::495::TaskManager.Task::(_debug) Task 5b9741d7-a3ab-4621-ae82-1f3cdad89906: moving from state preparing -> state finished >Thread-15913510::DEBUG::2013-11-20 03:07:35,742::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913510::DEBUG::2013-11-20 03:07:35,742::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913510::DEBUG::2013-11-20 03:07:35,743::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913510::DEBUG::2013-11-20 03:07:35,744::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913510::DEBUG::2013-11-20 03:07:35,744::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913510::DEBUG::2013-11-20 03:07:35,744::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913510::DEBUG::2013-11-20 03:07:35,745::task::495::TaskManager.Task::(_debug) Task 5b9741d7-a3ab-4621-ae82-1f3cdad89906: ref 0 aborting False >Thread-15913510::INFO::2013-11-20 03:07:35,745::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15913510::INFO::2013-11-20 03:07:35,746::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913510::DEBUG::2013-11-20 03:07:35,746::task::495::TaskManager.Task::(_debug) Task 2eca9d61-eaf3-4aef-b986-76c0fc3ab582: moving from state init -> state preparing >Thread-15913510::DEBUG::2013-11-20 03:07:35,747::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`b794f281-91b9-474c-991f-1c80d3808153`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1666' at 'public_getVolumePath' >Thread-15913510::DEBUG::2013-11-20 03:07:35,748::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913510::DEBUG::2013-11-20 03:07:35,748::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:35,748::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`b794f281-91b9-474c-991f-1c80d3808153`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:35,749::task::495::TaskManager.Task::(_debug) Task 2eca9d61-eaf3-4aef-b986-76c0fc3ab582: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913510::DEBUG::2013-11-20 03:07:35,749::task::495::TaskManager.Task::(_debug) Task 2eca9d61-eaf3-4aef-b986-76c0fc3ab582: ref 1 aborting False >Thread-15913510::DEBUG::2013-11-20 03:07:35,750::task::495::TaskManager.Task::(_debug) Task 2eca9d61-eaf3-4aef-b986-76c0fc3ab582: finished: {'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/772472cb-2429-466d-96cd-00b3301ef78a/images/13efb45b-f4d2-4c94-bbdb-91ce9b9b7321/088640f7-316c-4e2b-bfdc-0f64cf2913c8'} >Thread-15913510::DEBUG::2013-11-20 03:07:35,750::task::495::TaskManager.Task::(_debug) Task 2eca9d61-eaf3-4aef-b986-76c0fc3ab582: moving from state preparing -> state finished >Thread-15913510::DEBUG::2013-11-20 03:07:35,751::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913510::DEBUG::2013-11-20 03:07:35,751::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913510::DEBUG::2013-11-20 03:07:35,752::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913510::DEBUG::2013-11-20 03:07:35,752::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913510::DEBUG::2013-11-20 03:07:35,752::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913510::DEBUG::2013-11-20 03:07:35,753::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913510::DEBUG::2013-11-20 03:07:35,753::task::495::TaskManager.Task::(_debug) Task 2eca9d61-eaf3-4aef-b986-76c0fc3ab582: ref 0 aborting False >Thread-15913510::INFO::2013-11-20 03:07:35,753::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, Return response: {'status': {'message': 'OK', 'code': 0}, 'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/772472cb-2429-466d-96cd-00b3301ef78a/images/13efb45b-f4d2-4c94-bbdb-91ce9b9b7321/088640f7-316c-4e2b-bfdc-0f64cf2913c8'} >Thread-15913510::INFO::2013-11-20 03:07:35,754::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913510::DEBUG::2013-11-20 03:07:35,754::task::495::TaskManager.Task::(_debug) Task d069d465-769c-4c8d-a31f-064353f94623: moving from state init -> state preparing >Thread-15913510::DEBUG::2013-11-20 03:07:35,755::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`c44e0094-d1ea-4a5d-a886-8cdd338df392`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1690' at 'public_prepareVolume' >Thread-15913510::DEBUG::2013-11-20 03:07:35,755::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913510::DEBUG::2013-11-20 03:07:35,756::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:35,756::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`c44e0094-d1ea-4a5d-a886-8cdd338df392`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:35,762::task::495::TaskManager.Task::(_debug) Task d069d465-769c-4c8d-a31f-064353f94623: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913510::DEBUG::2013-11-20 03:07:35,762::task::495::TaskManager.Task::(_debug) Task d069d465-769c-4c8d-a31f-064353f94623: ref 1 aborting False >Thread-15913510::DEBUG::2013-11-20 03:07:35,763::resourceManager::155::ResourceManager.Request::(__init__) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_imageNS.d368307e-425e-423a-85b7-266c1e4a99ec`ReqID=`ac5a0032-f66a-4183-830c-50efd74affcc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1695' at 'public_prepareVolume' >Thread-15913510::DEBUG::2013-11-20 03:07:35,763::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.d368307e-425e-423a-85b7-266c1e4a99ec' for lock type 'exclusive' >Thread-15913510::DEBUG::2013-11-20 03:07:35,764::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:35,771::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:36,203::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:36,442::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:36,460::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:36,462::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=4 bs=512 if=/dev/772472cb-2429-466d-96cd-00b3301ef78a/metadata count=1' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:36,912::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.170107 s, 6.0 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:36,988::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000253222 s, 2.0 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:36,990::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000253222 s, 2.0 MB/s'], size: 512 >Thread-15913510::DEBUG::2013-11-20 03:07:37,001::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=4 bs=512 if=/dev/772472cb-2429-466d-96cd-00b3301ef78a/metadata count=1' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:37,444::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000338903 s, 1.5 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:37,445::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000338903 s, 1.5 MB/s'], size: 512 >Thread-15913510::INFO::2013-11-20 03:07:37,446::image::340::Storage.Image::(getChain) sdUUID=772472cb-2429-466d-96cd-00b3301ef78a imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec chain=[<storage.blockVolume.BlockVolume instance at 0x7f487c03bd40>] >Thread-15913510::DEBUG::2013-11-20 03:07:37,448::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --autobackup n --available y 772472cb-2429-466d-96cd-00b3301ef78a/fdc91117-476f-4b82-8fc4-d682fa283ecc' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:38,110::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:38,112::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:38,113::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:38,115::resourceManager::155::ResourceManager.Request::(__init__) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.fdc91117-476f-4b82-8fc4-d682fa283ecc`ReqID=`d0ad7fbf-de53-4daa-9081-0220ccc55a9c`::Request was made in '/usr/share/vdsm/storage/resourceFactories.py' line '149' at '__getResourceCandidatesList' >Thread-15913510::DEBUG::2013-11-20 03:07:38,116::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.fdc91117-476f-4b82-8fc4-d682fa283ecc' for lock type 'exclusive' >Thread-15913510::DEBUG::2013-11-20 03:07:38,117::resourceManager::505::ResourceManager::(registerResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.fdc91117-476f-4b82-8fc4-d682fa283ecc' is free. Now locking as 'exclusive' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:38,118::resourceManager::192::ResourceManager.Request::(grant) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_volumeNS.fdc91117-476f-4b82-8fc4-d682fa283ecc`ReqID=`d0ad7fbf-de53-4daa-9081-0220ccc55a9c`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:38,119::resourceManager::505::ResourceManager::(registerResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_imageNS.d368307e-425e-423a-85b7-266c1e4a99ec' is free. Now locking as 'exclusive' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:38,120::resourceManager::192::ResourceManager.Request::(grant) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_imageNS.d368307e-425e-423a-85b7-266c1e4a99ec`ReqID=`ac5a0032-f66a-4183-830c-50efd74affcc`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:38,123::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:38,125::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15913537::DEBUG::2013-11-20 03:07:38,757::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4b8ed9c8-dfb1-4de4-935d-6ba4009567b9',) {} >Thread-15913537::DEBUG::2013-11-20 03:07:38,759::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913537::DEBUG::2013-11-20 03:07:38,759::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913537::DEBUG::2013-11-20 03:07:38,760::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913537::DEBUG::2013-11-20 03:07:38,760::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913537::DEBUG::2013-11-20 03:07:38,761::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '46387', 'displayIp': '0', 'displayPort': u'5902', 'session': 'Unknown', 'displaySecurePort': u'5903', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet1': {'macAddr': u'00:1a:4a:a8:64:05', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet1'}, u'vnet2': {'macAddr': u'00:1a:4a:a8:64:0b', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet2'}}, 'vmId': '4b8ed9c8-dfb1-4de4-935d-6ba4009567b9', 'monitorResponse': '0', 'cpuUser': '75.79', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '30064771072', 'writeLatency': '0', 'imageID': 'b0980185-7de1-4e3b-bd76-4495169ab08b', 'flushLatency': '0', 'truesize': '30064771072'}, u'vdb': {'readLatency': '0', 'apparentsize': '826781204480', 'writeLatency': '0', 'imageID': '0b89347c-4654-42bb-a8d7-1a627130b7bd', 'flushLatency': '0', 'truesize': '826781204480'}}, 'boot': 'c', 'statsAge': '1.76', 'cpuIdle': '24.21', 'elapsedTime': '41', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '3601', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913510::DEBUG::2013-11-20 03:07:38,822::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:38,847::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:38,855::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --refresh 772472cb-2429-466d-96cd-00b3301ef78a/fdc91117-476f-4b82-8fc4-d682fa283ecc' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:39,110::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:39,696::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:39,697::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:39,698::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913510::INFO::2013-11-20 03:07:39,699::volume::517::Storage.Volume::(prepare) Volume: preparing volume 772472cb-2429-466d-96cd-00b3301ef78a/fdc91117-476f-4b82-8fc4-d682fa283ecc >Thread-15913510::DEBUG::2013-11-20 03:07:39,699::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:39,706::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:39,970::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.278717 s, 3.7 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:40,346::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:40,368::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:40,369::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=4 bs=512 if=/dev/772472cb-2429-466d-96cd-00b3301ef78a/metadata count=1' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:40,756::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000255151 s, 2.0 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:40,757::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000255151 s, 2.0 MB/s'], size: 512 >Thread-15913510::DEBUG::2013-11-20 03:07:40,759::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=4 bs=512 if=/dev/772472cb-2429-466d-96cd-00b3301ef78a/metadata count=1' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:41,229::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000292759 s, 1.7 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:41,231::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000292759 s, 1.7 MB/s'], size: 512 >Thread-15913510::DEBUG::2013-11-20 03:07:41,233::resourceManager::155::ResourceManager.Request::(__init__) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.fdc91117-476f-4b82-8fc4-d682fa283ecc`ReqID=`6264dc1b-7b1f-4852-a5da-e468ec66ada5`::Request was made in '/usr/share/vdsm/storage/volume.py' line '533' at 'prepare' >Thread-15913510::DEBUG::2013-11-20 03:07:41,234::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.fdc91117-476f-4b82-8fc4-d682fa283ecc' for lock type 'exclusive' >Thread-15913510::DEBUG::2013-11-20 03:07:41,236::resourceManager::505::ResourceManager::(registerResource) Resource '772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.fdc91117-476f-4b82-8fc4-d682fa283ecc' is free. Now locking as 'exclusive' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:41,236::resourceManager::192::ResourceManager.Request::(grant) ResName=`772472cb-2429-466d-96cd-00b3301ef78a_lvmActivationNS.fdc91117-476f-4b82-8fc4-d682fa283ecc`ReqID=`6264dc1b-7b1f-4852-a5da-e468ec66ada5`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:41,239::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=4 bs=512 if=/dev/772472cb-2429-466d-96cd-00b3301ef78a/metadata count=1' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:41,694::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000200104 s, 2.6 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:41,696::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000200104 s, 2.6 MB/s'], size: 512 >Thread-15913510::DEBUG::2013-11-20 03:07:41,698::task::495::TaskManager.Task::(_debug) Task d069d465-769c-4c8d-a31f-064353f94623: finished: None >Thread-15913510::DEBUG::2013-11-20 03:07:41,699::task::495::TaskManager.Task::(_debug) Task d069d465-769c-4c8d-a31f-064353f94623: moving from state preparing -> state finished >Thread-15913510::DEBUG::2013-11-20 03:07:41,700::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913510::DEBUG::2013-11-20 03:07:41,701::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913510::DEBUG::2013-11-20 03:07:41,702::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913510::DEBUG::2013-11-20 03:07:41,703::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913510::DEBUG::2013-11-20 03:07:41,704::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913510::DEBUG::2013-11-20 03:07:41,704::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913510::DEBUG::2013-11-20 03:07:41,706::task::495::TaskManager.Task::(_debug) Task d069d465-769c-4c8d-a31f-064353f94623: ref 0 aborting False >Thread-15913510::INFO::2013-11-20 03:07:41,707::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15913510::INFO::2013-11-20 03:07:41,708::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913510::DEBUG::2013-11-20 03:07:41,709::task::495::TaskManager.Task::(_debug) Task 94b7ec3e-f38a-4f67-969d-311086419f64: moving from state init -> state preparing >Thread-15913510::DEBUG::2013-11-20 03:07:41,711::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`77ff3a75-11fe-47e0-96a1-7e2d3aec52de`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1666' at 'public_getVolumePath' >Thread-15913510::DEBUG::2013-11-20 03:07:41,711::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913510::DEBUG::2013-11-20 03:07:41,712::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:41,713::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`77ff3a75-11fe-47e0-96a1-7e2d3aec52de`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:41,715::task::495::TaskManager.Task::(_debug) Task 94b7ec3e-f38a-4f67-969d-311086419f64: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913510::DEBUG::2013-11-20 03:07:41,716::task::495::TaskManager.Task::(_debug) Task 94b7ec3e-f38a-4f67-969d-311086419f64: ref 1 aborting False >Thread-15913510::DEBUG::2013-11-20 03:07:41,717::task::495::TaskManager.Task::(_debug) Task 94b7ec3e-f38a-4f67-969d-311086419f64: finished: {'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/772472cb-2429-466d-96cd-00b3301ef78a/images/d368307e-425e-423a-85b7-266c1e4a99ec/fdc91117-476f-4b82-8fc4-d682fa283ecc'} >Thread-15913510::DEBUG::2013-11-20 03:07:41,718::task::495::TaskManager.Task::(_debug) Task 94b7ec3e-f38a-4f67-969d-311086419f64: moving from state preparing -> state finished >Thread-15913510::DEBUG::2013-11-20 03:07:41,719::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913510::DEBUG::2013-11-20 03:07:41,720::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913510::DEBUG::2013-11-20 03:07:41,721::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913510::DEBUG::2013-11-20 03:07:41,722::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913510::DEBUG::2013-11-20 03:07:41,723::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913510::DEBUG::2013-11-20 03:07:41,724::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913510::DEBUG::2013-11-20 03:07:41,725::task::495::TaskManager.Task::(_debug) Task 94b7ec3e-f38a-4f67-969d-311086419f64: ref 0 aborting False >Thread-15913510::INFO::2013-11-20 03:07:41,726::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, Return response: {'status': {'message': 'OK', 'code': 0}, 'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/772472cb-2429-466d-96cd-00b3301ef78a/images/d368307e-425e-423a-85b7-266c1e4a99ec/fdc91117-476f-4b82-8fc4-d682fa283ecc'} >Thread-15913510::INFO::2013-11-20 03:07:41,727::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913510::DEBUG::2013-11-20 03:07:41,728::task::495::TaskManager.Task::(_debug) Task 840ddc4a-4251-493c-a994-e80f101887ad: moving from state init -> state preparing >Thread-15913510::DEBUG::2013-11-20 03:07:41,729::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`69aa8778-9d4b-4788-9fba-c7555ee2e8e2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913510::DEBUG::2013-11-20 03:07:41,730::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913510::DEBUG::2013-11-20 03:07:41,731::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:41,732::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`69aa8778-9d4b-4788-9fba-c7555ee2e8e2`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:41,733::task::495::TaskManager.Task::(_debug) Task 840ddc4a-4251-493c-a994-e80f101887ad: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913510::DEBUG::2013-11-20 03:07:41,734::task::495::TaskManager.Task::(_debug) Task 840ddc4a-4251-493c-a994-e80f101887ad: ref 1 aborting False >Thread-15913510::DEBUG::2013-11-20 03:07:41,735::task::495::TaskManager.Task::(_debug) Task 840ddc4a-4251-493c-a994-e80f101887ad: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913510::DEBUG::2013-11-20 03:07:41,736::task::495::TaskManager.Task::(_debug) Task 840ddc4a-4251-493c-a994-e80f101887ad: moving from state preparing -> state finished >Thread-15913510::DEBUG::2013-11-20 03:07:41,736::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913510::DEBUG::2013-11-20 03:07:41,737::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913510::DEBUG::2013-11-20 03:07:41,738::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913510::DEBUG::2013-11-20 03:07:41,738::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913510::DEBUG::2013-11-20 03:07:41,739::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913510::DEBUG::2013-11-20 03:07:41,740::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913510::DEBUG::2013-11-20 03:07:41,741::task::495::TaskManager.Task::(_debug) Task 840ddc4a-4251-493c-a994-e80f101887ad: ref 0 aborting False >Thread-15913510::INFO::2013-11-20 03:07:41,741::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913510::INFO::2013-11-20 03:07:41,743::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a) >Thread-15913510::DEBUG::2013-11-20 03:07:41,743::task::495::TaskManager.Task::(_debug) Task 1882563a-1e45-446c-92a3-20552b325bea: moving from state init -> state preparing >Thread-15913510::INFO::2013-11-20 03:07:41,744::blockSD::674::Storage.StorageDomain::(validate) sdUUID=772472cb-2429-466d-96cd-00b3301ef78a >Thread-15913510::DEBUG::2013-11-20 03:07:41,746::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:41,747::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:41,748::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:41,749::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:41,750::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:41,752::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15913539::DEBUG::2013-11-20 03:07:41,974::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4b8ed9c8-dfb1-4de4-935d-6ba4009567b9',) {} >Thread-15913539::DEBUG::2013-11-20 03:07:41,976::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913539::DEBUG::2013-11-20 03:07:41,977::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913539::DEBUG::2013-11-20 03:07:41,977::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913539::DEBUG::2013-11-20 03:07:41,978::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913539::DEBUG::2013-11-20 03:07:41,979::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '46387', 'displayIp': '0', 'displayPort': u'5902', 'session': 'Unknown', 'displaySecurePort': u'5903', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet1': {'macAddr': u'00:1a:4a:a8:64:05', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet1'}, u'vnet2': {'macAddr': u'00:1a:4a:a8:64:0b', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet2'}}, 'vmId': '4b8ed9c8-dfb1-4de4-935d-6ba4009567b9', 'monitorResponse': '0', 'cpuUser': '75.79', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '30064771072', 'writeLatency': '0', 'imageID': 'b0980185-7de1-4e3b-bd76-4495169ab08b', 'flushLatency': '0', 'truesize': '30064771072'}, u'vdb': {'readLatency': '0', 'apparentsize': '826781204480', 'writeLatency': '0', 'imageID': '0b89347c-4654-42bb-a8d7-1a627130b7bd', 'flushLatency': '0', 'truesize': '826781204480'}}, 'boot': 'c', 'statsAge': '0.46', 'cpuIdle': '24.21', 'elapsedTime': '44', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '3601', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:07:41,987::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:42,441::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:42,446::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:42,452::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'SDUUID=772472cb-2429-466d-96cd-00b3301ef78a', u'VERSION=2', u'LOCKPOLICY=', u'VGUUID=XTb294-NYv6-dVpO-bB1n-4f9P-d0GL-dhd2DI', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'LEASETIMESEC=5', u'LOCKRENEWALINTERVALSEC=5', u'ROLE=Regular', u'PHYBLKSIZE=512', u'DESCRIPTION=Database', u'IOOPTIMEOUTSEC=1', u'CLASS=Data', u'TYPE=FCP', u'PV0=pv:3600c0ff0001520c7414f605001000000,uuid:NujM33-vNiD-sBHY-mrPo-EQ2f-1qsz-HXfn0q,pestart:0,pecount:8929,mapoffset:0', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'MASTER_VERSION=0', u'_SHA_CKSUM=ef8840d9fdef24516a56fd970bed1960d2b1ae92'] >Thread-15913510::DEBUG::2013-11-20 03:07:42,457::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`6c2d98ef-fe6f-4855-abc4-f64f0a8c44d2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1297' at 'public_getStorageDomainInfo' >Thread-15913510::DEBUG::2013-11-20 03:07:42,458::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913510::DEBUG::2013-11-20 03:07:42,459::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:42,459::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`6c2d98ef-fe6f-4855-abc4-f64f0a8c44d2`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:42,460::task::495::TaskManager.Task::(_debug) Task 1882563a-1e45-446c-92a3-20552b325bea: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913510::DEBUG::2013-11-20 03:07:42,461::task::495::TaskManager.Task::(_debug) Task 1882563a-1e45-446c-92a3-20552b325bea: ref 1 aborting False >Thread-15913510::DEBUG::2013-11-20 03:07:42,461::task::495::TaskManager.Task::(_debug) Task 1882563a-1e45-446c-92a3-20552b325bea: finished: {'info': {'uuid': '772472cb-2429-466d-96cd-00b3301ef78a', 'vguuid': 'XTb294-NYv6-dVpO-bB1n-4f9P-d0GL-dhd2DI', 'lver': -1, 'state': 'OK', 'version': '2', 'role': 'Regular', 'master_ver': 0, 'spm_id': -1, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'Database'}} >Thread-15913510::DEBUG::2013-11-20 03:07:42,466::task::495::TaskManager.Task::(_debug) Task 1882563a-1e45-446c-92a3-20552b325bea: moving from state preparing -> state finished >Thread-15913510::DEBUG::2013-11-20 03:07:42,467::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913510::DEBUG::2013-11-20 03:07:42,467::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913510::DEBUG::2013-11-20 03:07:42,467::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913510::DEBUG::2013-11-20 03:07:42,468::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913510::DEBUG::2013-11-20 03:07:42,468::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913510::DEBUG::2013-11-20 03:07:42,469::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913510::DEBUG::2013-11-20 03:07:42,469::task::495::TaskManager.Task::(_debug) Task 1882563a-1e45-446c-92a3-20552b325bea: ref 0 aborting False >Thread-15913510::INFO::2013-11-20 03:07:42,470::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'uuid': '772472cb-2429-466d-96cd-00b3301ef78a', 'vguuid': 'XTb294-NYv6-dVpO-bB1n-4f9P-d0GL-dhd2DI', 'lver': -1, 'state': 'OK', 'version': '2', 'role': 'Regular', 'master_ver': 0, 'spm_id': -1, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'Database'}} >Thread-15913510::INFO::2013-11-20 03:07:42,470::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913510::DEBUG::2013-11-20 03:07:42,471::task::495::TaskManager.Task::(_debug) Task cce91dc9-d6ba-446e-9205-c68d79494c1e: moving from state init -> state preparing >Thread-15913510::DEBUG::2013-11-20 03:07:42,472::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`efc26c6b-dcda-4d2a-a328-8057420c099e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913510::DEBUG::2013-11-20 03:07:42,472::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913510::DEBUG::2013-11-20 03:07:42,473::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:42,473::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`efc26c6b-dcda-4d2a-a328-8057420c099e`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:42,474::task::495::TaskManager.Task::(_debug) Task cce91dc9-d6ba-446e-9205-c68d79494c1e: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913510::DEBUG::2013-11-20 03:07:42,475::task::495::TaskManager.Task::(_debug) Task cce91dc9-d6ba-446e-9205-c68d79494c1e: ref 1 aborting False >Thread-15913510::DEBUG::2013-11-20 03:07:42,476::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:42,481::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15913540::DEBUG::2013-11-20 03:07:42,734::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913540::INFO::2013-11-20 03:07:42,737::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913540::DEBUG::2013-11-20 03:07:42,738::task::495::TaskManager.Task::(_debug) Task 83d5cf8e-9cc1-4491-9266-2d7d62bfc847: moving from state init -> state preparing >Thread-15913540::DEBUG::2013-11-20 03:07:42,739::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913540::DEBUG::2013-11-20 03:07:42,740::task::495::TaskManager.Task::(_debug) Task 83d5cf8e-9cc1-4491-9266-2d7d62bfc847: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913540::DEBUG::2013-11-20 03:07:42,742::task::495::TaskManager.Task::(_debug) Task 83d5cf8e-9cc1-4491-9266-2d7d62bfc847: moving from state preparing -> state finished >Thread-15913540::DEBUG::2013-11-20 03:07:42,743::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913540::DEBUG::2013-11-20 03:07:42,743::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913540::DEBUG::2013-11-20 03:07:42,745::task::495::TaskManager.Task::(_debug) Task 83d5cf8e-9cc1-4491-9266-2d7d62bfc847: ref 0 aborting False >Thread-15913540::INFO::2013-11-20 03:07:42,746::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913541::DEBUG::2013-11-20 03:07:42,752::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913541::INFO::2013-11-20 03:07:42,753::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913541::DEBUG::2013-11-20 03:07:42,754::task::495::TaskManager.Task::(_debug) Task 68b6c3f2-3f98-4eb1-8702-d021d0a3f69b: moving from state init -> state preparing >Thread-15913541::DEBUG::2013-11-20 03:07:42,755::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`dd75f309-cefc-4cc1-a1b9-69662c021eb8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913541::DEBUG::2013-11-20 03:07:42,756::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913541::DEBUG::2013-11-20 03:07:42,758::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913541::DEBUG::2013-11-20 03:07:42,759::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`dd75f309-cefc-4cc1-a1b9-69662c021eb8`::Granted request >Thread-15913541::DEBUG::2013-11-20 03:07:42,760::task::495::TaskManager.Task::(_debug) Task 68b6c3f2-3f98-4eb1-8702-d021d0a3f69b: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913541::DEBUG::2013-11-20 03:07:42,761::task::495::TaskManager.Task::(_debug) Task 68b6c3f2-3f98-4eb1-8702-d021d0a3f69b: ref 1 aborting False >Thread-15913541::WARNING::2013-11-20 03:07:42,764::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913541::DEBUG::2013-11-20 03:07:42,766::task::495::TaskManager.Task::(_debug) Task 68b6c3f2-3f98-4eb1-8702-d021d0a3f69b: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913541::DEBUG::2013-11-20 03:07:42,767::task::495::TaskManager.Task::(_debug) Task 68b6c3f2-3f98-4eb1-8702-d021d0a3f69b: moving from state preparing -> state finished >Thread-15913541::DEBUG::2013-11-20 03:07:42,768::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913541::DEBUG::2013-11-20 03:07:42,768::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913541::DEBUG::2013-11-20 03:07:42,769::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913541::DEBUG::2013-11-20 03:07:42,770::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913541::DEBUG::2013-11-20 03:07:42,771::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913541::DEBUG::2013-11-20 03:07:42,772::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913541::DEBUG::2013-11-20 03:07:42,773::task::495::TaskManager.Task::(_debug) Task 68b6c3f2-3f98-4eb1-8702-d021d0a3f69b: ref 0 aborting False >Thread-15913541::INFO::2013-11-20 03:07:42,774::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:07:42,894::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.350143 s, 2.9 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:43,195::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:43,233::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:43,234::task::495::TaskManager.Task::(_debug) Task cce91dc9-d6ba-446e-9205-c68d79494c1e: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913510::DEBUG::2013-11-20 03:07:43,235::task::495::TaskManager.Task::(_debug) Task cce91dc9-d6ba-446e-9205-c68d79494c1e: moving from state preparing -> state finished >Thread-15913510::DEBUG::2013-11-20 03:07:43,236::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913510::DEBUG::2013-11-20 03:07:43,236::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913510::DEBUG::2013-11-20 03:07:43,237::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913510::DEBUG::2013-11-20 03:07:43,238::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913510::DEBUG::2013-11-20 03:07:43,238::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913510::DEBUG::2013-11-20 03:07:43,239::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913510::DEBUG::2013-11-20 03:07:43,239::task::495::TaskManager.Task::(_debug) Task cce91dc9-d6ba-446e-9205-c68d79494c1e: ref 0 aborting False >Thread-15913510::INFO::2013-11-20 03:07:43,240::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913510::INFO::2013-11-20 03:07:43,241::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a) >Thread-15913510::DEBUG::2013-11-20 03:07:43,242::task::495::TaskManager.Task::(_debug) Task d98f2249-6323-4563-8fad-95a0411e138c: moving from state init -> state preparing >Thread-15913510::INFO::2013-11-20 03:07:43,242::blockSD::674::Storage.StorageDomain::(validate) sdUUID=772472cb-2429-466d-96cd-00b3301ef78a >Thread-15913510::DEBUG::2013-11-20 03:07:43,243::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:43,244::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:43,245::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:43,246::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:43,247::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:43,248::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:43,964::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:43,973::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:43,974::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'SDUUID=772472cb-2429-466d-96cd-00b3301ef78a', u'VERSION=2', u'LOCKPOLICY=', u'VGUUID=XTb294-NYv6-dVpO-bB1n-4f9P-d0GL-dhd2DI', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'LEASETIMESEC=5', u'LOCKRENEWALINTERVALSEC=5', u'ROLE=Regular', u'PHYBLKSIZE=512', u'DESCRIPTION=Database', u'IOOPTIMEOUTSEC=1', u'CLASS=Data', u'TYPE=FCP', u'PV0=pv:3600c0ff0001520c7414f605001000000,uuid:NujM33-vNiD-sBHY-mrPo-EQ2f-1qsz-HXfn0q,pestart:0,pecount:8929,mapoffset:0', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'MASTER_VERSION=0', u'_SHA_CKSUM=ef8840d9fdef24516a56fd970bed1960d2b1ae92'] >Thread-15913510::DEBUG::2013-11-20 03:07:43,977::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`3375f726-e955-46e9-bf67-ddb11989f83e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1297' at 'public_getStorageDomainInfo' >Thread-15913510::DEBUG::2013-11-20 03:07:43,978::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913510::DEBUG::2013-11-20 03:07:43,978::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:43,979::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`3375f726-e955-46e9-bf67-ddb11989f83e`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:43,980::task::495::TaskManager.Task::(_debug) Task d98f2249-6323-4563-8fad-95a0411e138c: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913510::DEBUG::2013-11-20 03:07:43,981::task::495::TaskManager.Task::(_debug) Task d98f2249-6323-4563-8fad-95a0411e138c: ref 1 aborting False >Thread-15913510::DEBUG::2013-11-20 03:07:43,982::task::495::TaskManager.Task::(_debug) Task d98f2249-6323-4563-8fad-95a0411e138c: finished: {'info': {'uuid': '772472cb-2429-466d-96cd-00b3301ef78a', 'vguuid': 'XTb294-NYv6-dVpO-bB1n-4f9P-d0GL-dhd2DI', 'lver': -1, 'state': 'OK', 'version': '2', 'role': 'Regular', 'master_ver': 0, 'spm_id': -1, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'Database'}} >Thread-15913510::DEBUG::2013-11-20 03:07:43,987::task::495::TaskManager.Task::(_debug) Task d98f2249-6323-4563-8fad-95a0411e138c: moving from state preparing -> state finished >Thread-15913510::DEBUG::2013-11-20 03:07:43,987::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913510::DEBUG::2013-11-20 03:07:43,988::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913510::DEBUG::2013-11-20 03:07:43,989::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913510::DEBUG::2013-11-20 03:07:43,990::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913510::DEBUG::2013-11-20 03:07:43,990::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913510::DEBUG::2013-11-20 03:07:43,991::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913510::DEBUG::2013-11-20 03:07:43,991::task::495::TaskManager.Task::(_debug) Task d98f2249-6323-4563-8fad-95a0411e138c: ref 0 aborting False >Thread-15913510::INFO::2013-11-20 03:07:43,992::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'uuid': '772472cb-2429-466d-96cd-00b3301ef78a', 'vguuid': 'XTb294-NYv6-dVpO-bB1n-4f9P-d0GL-dhd2DI', 'lver': -1, 'state': 'OK', 'version': '2', 'role': 'Regular', 'master_ver': 0, 'spm_id': -1, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'Database'}} >Thread-15913510::INFO::2013-11-20 03:07:43,997::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913510::DEBUG::2013-11-20 03:07:43,997::task::495::TaskManager.Task::(_debug) Task 8f965bcb-bfa1-46a2-89e5-e66a493cb4cf: moving from state init -> state preparing >Thread-15913510::DEBUG::2013-11-20 03:07:43,999::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`5f1fec54-a948-4881-a89f-d7fa65d254bd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913510::DEBUG::2013-11-20 03:07:43,999::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913510::DEBUG::2013-11-20 03:07:44,000::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:44,001::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`5f1fec54-a948-4881-a89f-d7fa65d254bd`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:44,002::task::495::TaskManager.Task::(_debug) Task 8f965bcb-bfa1-46a2-89e5-e66a493cb4cf: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913510::DEBUG::2013-11-20 03:07:44,002::task::495::TaskManager.Task::(_debug) Task 8f965bcb-bfa1-46a2-89e5-e66a493cb4cf: ref 1 aborting False >Thread-15913510::DEBUG::2013-11-20 03:07:44,003::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:44,009::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:44,653::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:44,675::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:44,676::task::495::TaskManager.Task::(_debug) Task 8f965bcb-bfa1-46a2-89e5-e66a493cb4cf: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913510::DEBUG::2013-11-20 03:07:44,677::task::495::TaskManager.Task::(_debug) Task 8f965bcb-bfa1-46a2-89e5-e66a493cb4cf: moving from state preparing -> state finished >Thread-15913510::DEBUG::2013-11-20 03:07:44,677::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913510::DEBUG::2013-11-20 03:07:44,678::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913510::DEBUG::2013-11-20 03:07:44,679::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913510::DEBUG::2013-11-20 03:07:44,680::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913510::DEBUG::2013-11-20 03:07:44,680::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913510::DEBUG::2013-11-20 03:07:44,681::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913510::DEBUG::2013-11-20 03:07:44,682::task::495::TaskManager.Task::(_debug) Task 8f965bcb-bfa1-46a2-89e5-e66a493cb4cf: ref 0 aborting False >Thread-15913510::INFO::2013-11-20 03:07:44,682::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913510::INFO::2013-11-20 03:07:44,683::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a) >Thread-15913510::DEBUG::2013-11-20 03:07:44,684::task::495::TaskManager.Task::(_debug) Task 759f54f4-d673-4bf8-8f1e-6b766bcd843a: moving from state init -> state preparing >Thread-15913510::INFO::2013-11-20 03:07:44,685::blockSD::674::Storage.StorageDomain::(validate) sdUUID=772472cb-2429-466d-96cd-00b3301ef78a >Thread-15913510::DEBUG::2013-11-20 03:07:44,685::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:44,686::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:44,687::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:44,688::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:44,689::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:44,690::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:44,911::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913543::DEBUG::2013-11-20 03:07:45,278::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4b8ed9c8-dfb1-4de4-935d-6ba4009567b9',) {} >Thread-15913543::DEBUG::2013-11-20 03:07:45,280::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913543::DEBUG::2013-11-20 03:07:45,280::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913543::DEBUG::2013-11-20 03:07:45,281::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913543::DEBUG::2013-11-20 03:07:45,282::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913543::DEBUG::2013-11-20 03:07:45,282::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '46387', 'displayIp': '0', 'displayPort': u'5902', 'session': 'Unknown', 'displaySecurePort': u'5903', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet1': {'macAddr': u'00:1a:4a:a8:64:05', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet1'}, u'vnet2': {'macAddr': u'00:1a:4a:a8:64:0b', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet2'}}, 'vmId': '4b8ed9c8-dfb1-4de4-935d-6ba4009567b9', 'monitorResponse': '0', 'cpuUser': '75.79', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '30064771072', 'writeLatency': '0', 'imageID': 'b0980185-7de1-4e3b-bd76-4495169ab08b', 'flushLatency': '0', 'truesize': '30064771072'}, u'vdb': {'readLatency': '0', 'apparentsize': '826781204480', 'writeLatency': '0', 'imageID': '0b89347c-4654-42bb-a8d7-1a627130b7bd', 'flushLatency': '0', 'truesize': '826781204480'}}, 'boot': 'c', 'statsAge': '1.74', 'cpuIdle': '24.21', 'elapsedTime': '47', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '3601', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913510::DEBUG::2013-11-20 03:07:45,542::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:45,546::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:45,549::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'SDUUID=772472cb-2429-466d-96cd-00b3301ef78a', u'VERSION=2', u'LOCKPOLICY=', u'VGUUID=XTb294-NYv6-dVpO-bB1n-4f9P-d0GL-dhd2DI', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'LEASETIMESEC=5', u'LOCKRENEWALINTERVALSEC=5', u'ROLE=Regular', u'PHYBLKSIZE=512', u'DESCRIPTION=Database', u'IOOPTIMEOUTSEC=1', u'CLASS=Data', u'TYPE=FCP', u'PV0=pv:3600c0ff0001520c7414f605001000000,uuid:NujM33-vNiD-sBHY-mrPo-EQ2f-1qsz-HXfn0q,pestart:0,pecount:8929,mapoffset:0', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'MASTER_VERSION=0', u'_SHA_CKSUM=ef8840d9fdef24516a56fd970bed1960d2b1ae92'] >Thread-15913510::DEBUG::2013-11-20 03:07:45,558::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`c20851bd-5995-41a0-83bc-25f139573c74`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1297' at 'public_getStorageDomainInfo' >Thread-15913510::DEBUG::2013-11-20 03:07:45,559::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913510::DEBUG::2013-11-20 03:07:45,560::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:45,561::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`c20851bd-5995-41a0-83bc-25f139573c74`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:45,564::task::495::TaskManager.Task::(_debug) Task 759f54f4-d673-4bf8-8f1e-6b766bcd843a: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913510::DEBUG::2013-11-20 03:07:45,565::task::495::TaskManager.Task::(_debug) Task 759f54f4-d673-4bf8-8f1e-6b766bcd843a: ref 1 aborting False >Thread-15913510::DEBUG::2013-11-20 03:07:45,566::task::495::TaskManager.Task::(_debug) Task 759f54f4-d673-4bf8-8f1e-6b766bcd843a: finished: {'info': {'uuid': '772472cb-2429-466d-96cd-00b3301ef78a', 'vguuid': 'XTb294-NYv6-dVpO-bB1n-4f9P-d0GL-dhd2DI', 'lver': -1, 'state': 'OK', 'version': '2', 'role': 'Regular', 'master_ver': 0, 'spm_id': -1, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'Database'}} >Thread-15913510::DEBUG::2013-11-20 03:07:45,568::task::495::TaskManager.Task::(_debug) Task 759f54f4-d673-4bf8-8f1e-6b766bcd843a: moving from state preparing -> state finished >Thread-15913510::DEBUG::2013-11-20 03:07:45,569::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913510::DEBUG::2013-11-20 03:07:45,570::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913510::DEBUG::2013-11-20 03:07:45,571::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913510::DEBUG::2013-11-20 03:07:45,572::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913510::DEBUG::2013-11-20 03:07:45,573::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913510::DEBUG::2013-11-20 03:07:45,574::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913510::DEBUG::2013-11-20 03:07:45,575::task::495::TaskManager.Task::(_debug) Task 759f54f4-d673-4bf8-8f1e-6b766bcd843a: ref 0 aborting False >Thread-15913510::INFO::2013-11-20 03:07:45,576::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'uuid': '772472cb-2429-466d-96cd-00b3301ef78a', 'vguuid': 'XTb294-NYv6-dVpO-bB1n-4f9P-d0GL-dhd2DI', 'lver': -1, 'state': 'OK', 'version': '2', 'role': 'Regular', 'master_ver': 0, 'spm_id': -1, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'Database'}} >Thread-15913510::INFO::2013-11-20 03:07:45,578::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913510::DEBUG::2013-11-20 03:07:45,579::task::495::TaskManager.Task::(_debug) Task aff57eed-5bec-44d5-9a86-1e3591bcc895: moving from state init -> state preparing >Thread-15913510::DEBUG::2013-11-20 03:07:45,581::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`af8c0c41-36cc-4ad8-aad5-c18d6c7baf9d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913510::DEBUG::2013-11-20 03:07:45,582::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913510::DEBUG::2013-11-20 03:07:45,583::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:45,584::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`af8c0c41-36cc-4ad8-aad5-c18d6c7baf9d`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:45,586::task::495::TaskManager.Task::(_debug) Task aff57eed-5bec-44d5-9a86-1e3591bcc895: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913510::DEBUG::2013-11-20 03:07:45,587::task::495::TaskManager.Task::(_debug) Task aff57eed-5bec-44d5-9a86-1e3591bcc895: ref 1 aborting False >Thread-15913510::DEBUG::2013-11-20 03:07:45,588::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:45,589::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:45,920::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.426244 s, 2.4 MB/s\n'; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:46,280::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:46,303::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:46,304::task::495::TaskManager.Task::(_debug) Task aff57eed-5bec-44d5-9a86-1e3591bcc895: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913510::DEBUG::2013-11-20 03:07:46,305::task::495::TaskManager.Task::(_debug) Task aff57eed-5bec-44d5-9a86-1e3591bcc895: moving from state preparing -> state finished >Thread-15913510::DEBUG::2013-11-20 03:07:46,305::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913510::DEBUG::2013-11-20 03:07:46,306::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913510::DEBUG::2013-11-20 03:07:46,307::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913510::DEBUG::2013-11-20 03:07:46,308::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913510::DEBUG::2013-11-20 03:07:46,308::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913510::DEBUG::2013-11-20 03:07:46,309::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913510::DEBUG::2013-11-20 03:07:46,309::task::495::TaskManager.Task::(_debug) Task aff57eed-5bec-44d5-9a86-1e3591bcc895: ref 0 aborting False >Thread-15913510::INFO::2013-11-20 03:07:46,310::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913510::INFO::2013-11-20 03:07:46,311::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a) >Thread-15913510::DEBUG::2013-11-20 03:07:46,312::task::495::TaskManager.Task::(_debug) Task 38ffd94f-9d66-4d2a-b7ff-030d5e2eb5f7: moving from state init -> state preparing >Thread-15913510::INFO::2013-11-20 03:07:46,312::blockSD::674::Storage.StorageDomain::(validate) sdUUID=772472cb-2429-466d-96cd-00b3301ef78a >Thread-15913510::DEBUG::2013-11-20 03:07:46,313::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:46,314::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:46,315::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:46,316::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:46,316::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:46,318::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 03:07:46,911::lvm::439::OperationMutex::(_reloadvgs) Got the operational mutex >Thread-42840::DEBUG::2013-11-20 03:07:46,914::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:47,129::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:47,136::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'SDUUID=772472cb-2429-466d-96cd-00b3301ef78a', u'VERSION=2', u'LOCKPOLICY=', u'VGUUID=XTb294-NYv6-dVpO-bB1n-4f9P-d0GL-dhd2DI', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'LEASETIMESEC=5', u'LOCKRENEWALINTERVALSEC=5', u'ROLE=Regular', u'PHYBLKSIZE=512', u'DESCRIPTION=Database', u'IOOPTIMEOUTSEC=1', u'CLASS=Data', u'TYPE=FCP', u'PV0=pv:3600c0ff0001520c7414f605001000000,uuid:NujM33-vNiD-sBHY-mrPo-EQ2f-1qsz-HXfn0q,pestart:0,pecount:8929,mapoffset:0', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'MASTER_VERSION=0', u'_SHA_CKSUM=ef8840d9fdef24516a56fd970bed1960d2b1ae92'] >Thread-15913510::DEBUG::2013-11-20 03:07:47,139::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`f7377826-33bb-48c1-b52b-0995890ad4e1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1297' at 'public_getStorageDomainInfo' >Thread-15913510::DEBUG::2013-11-20 03:07:47,140::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913510::DEBUG::2013-11-20 03:07:47,141::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913510::DEBUG::2013-11-20 03:07:47,142::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`f7377826-33bb-48c1-b52b-0995890ad4e1`::Granted request >Thread-15913510::DEBUG::2013-11-20 03:07:47,143::task::495::TaskManager.Task::(_debug) Task 38ffd94f-9d66-4d2a-b7ff-030d5e2eb5f7: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913510::DEBUG::2013-11-20 03:07:47,144::task::495::TaskManager.Task::(_debug) Task 38ffd94f-9d66-4d2a-b7ff-030d5e2eb5f7: ref 1 aborting False >Thread-15913510::DEBUG::2013-11-20 03:07:47,145::task::495::TaskManager.Task::(_debug) Task 38ffd94f-9d66-4d2a-b7ff-030d5e2eb5f7: finished: {'info': {'uuid': '772472cb-2429-466d-96cd-00b3301ef78a', 'vguuid': 'XTb294-NYv6-dVpO-bB1n-4f9P-d0GL-dhd2DI', 'lver': -1, 'state': 'OK', 'version': '2', 'role': 'Regular', 'master_ver': 0, 'spm_id': -1, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'Database'}} >Thread-15913510::DEBUG::2013-11-20 03:07:47,146::task::495::TaskManager.Task::(_debug) Task 38ffd94f-9d66-4d2a-b7ff-030d5e2eb5f7: moving from state preparing -> state finished >Thread-15913510::DEBUG::2013-11-20 03:07:47,147::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913510::DEBUG::2013-11-20 03:07:47,147::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913510::DEBUG::2013-11-20 03:07:47,148::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913510::DEBUG::2013-11-20 03:07:47,149::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913510::DEBUG::2013-11-20 03:07:47,149::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913510::DEBUG::2013-11-20 03:07:47,150::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913510::DEBUG::2013-11-20 03:07:47,151::task::495::TaskManager.Task::(_debug) Task 38ffd94f-9d66-4d2a-b7ff-030d5e2eb5f7: ref 0 aborting False >Thread-15913510::INFO::2013-11-20 03:07:47,151::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'uuid': '772472cb-2429-466d-96cd-00b3301ef78a', 'vguuid': 'XTb294-NYv6-dVpO-bB1n-4f9P-d0GL-dhd2DI', 'lver': -1, 'state': 'OK', 'version': '2', 'role': 'Regular', 'master_ver': 0, 'spm_id': -1, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'Database'}} >Thread-15913510::DEBUG::2013-11-20 03:07:47,165::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/libexec/vdsm/hooks/before_vm_start/50_vhostmd' (cwd None) >Thread-42840::DEBUG::2013-11-20 03:07:47,668::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-42840::DEBUG::2013-11-20 03:07:47,672::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15913510::DEBUG::2013-11-20 03:07:47,755::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::INFO::2013-11-20 03:07:47,756::hooks::51::root::(_runHooksDir) >Thread-15913510::DEBUG::2013-11-20 03:07:47,757::libvirtvm::1077::vm.Vm::(_run) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::<?xml version="1.0" encoding="utf-8"?> ><domain type="kvm"> > <name>KLEE-AXDB01</name> > <uuid>6574d88c-00ac-46f4-99ce-a2726932bfda</uuid> > <memory>20971520</memory> > <currentMemory>20971520</currentMemory> > <vcpu>8</vcpu> > <devices> > <disk device="disk" type="block"> > <source dev="/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/772472cb-2429-466d-96cd-00b3301ef78a/images/c4460480-32d3-469f-b0f4-c03bb6d74e41/e6984064-e09e-4c0c-8b51-a8a1711994cf"/> > <target bus="virtio" dev="vda"/> > <serial>9f-b0f4-c03bb6d74e41</serial> > <driver cache="none" error_policy="stop" io="native" name="qemu" type="raw"/> > </disk> > <disk device="disk" type="block"> > <source dev="/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/772472cb-2429-466d-96cd-00b3301ef78a/images/df8b790c-c762-4feb-9378-d64a321ce43c/2e903e43-1016-4339-a7f3-c5723a995725"/> > <target bus="virtio" dev="vdb"/> > <serial>eb-9378-d64a321ce43c</serial> > <driver cache="none" error_policy="stop" io="native" name="qemu" type="raw"/> > </disk> > <disk device="disk" type="block"> > <source dev="/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/772472cb-2429-466d-96cd-00b3301ef78a/images/13efb45b-f4d2-4c94-bbdb-91ce9b9b7321/088640f7-316c-4e2b-bfdc-0f64cf2913c8"/> > <target bus="virtio" dev="vdc"/> > <serial>94-bbdb-91ce9b9b7321</serial> > <driver cache="none" error_policy="stop" io="native" name="qemu" type="raw"/> > </disk> > <disk device="disk" type="block"> > <source dev="/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/772472cb-2429-466d-96cd-00b3301ef78a/images/d368307e-425e-423a-85b7-266c1e4a99ec/fdc91117-476f-4b82-8fc4-d682fa283ecc"/> > <target bus="virtio" dev="vdd"/> > <serial>3a-85b7-266c1e4a99ec</serial> > <driver cache="none" error_policy="stop" io="native" name="qemu" type="raw"/> > </disk> > <disk device="cdrom" type="file"> > <target bus="ide" dev="hdc"/> > </disk> > <controller index="0" ports="16" type="virtio-serial"/> > <channel type="unix"> > <target name="com.redhat.rhevm.vdsm" type="virtio"/> > <source mode="bind" path="/var/lib/libvirt/qemu/channels/KLEE-AXDB01.com.redhat.rhevm.vdsm"/> > </channel> > <memballoon model="none"/> > <interface type="bridge"> > <mac address="00:1a:4a:a8:64:08"/> > <model type="virtio"/> > <source bridge="rhevm"/> > </interface> > <input bus="ps2" type="mouse"/> > <video> > <model heads="1" type="qxl" vram="65536"/> > </video> > <channel type="spicevmc"> > <target name="com.redhat.spice.0" type="virtio"/> > </channel> > <graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice"> > <channel mode="secure" name="main"/> > <channel mode="secure" name="inputs"/> > </graphics> > </devices> > <os> > <type arch="x86_64" machine="rhel6.2.0">hvm</type> > <boot dev="hd"/> > <boot dev="cdrom"/> > <smbios mode="sysinfo"/> > </os> > <sysinfo type="smbios"> > <system> > <entry name="manufacturer">Red Hat</entry> > <entry name="product">RHEV Hypervisor</entry> > <entry name="version">6.3-20121012.0.el6_3</entry> > <entry name="serial">30343536-3138-5A43-4A32-333630373442_AC:16:2D:76:EB:F0</entry> > <entry name="uuid">6574d88c-00ac-46f4-99ce-a2726932bfda</entry> > </system> > </sysinfo> > <clock adjustment="-1" offset="variable"> > <timer name="rtc" tickpolicy="catchup"/> > </clock> > <features> > <acpi/> > </features> > <cpu match="exact"> > <model>Westmere</model> > <topology cores="4" sockets="2" threads="1"/> > </cpu> ></domain> > >Dummy-15912651::DEBUG::2013-11-20 03:07:47,937::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913544::INFO::2013-11-20 03:07:48,309::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913544::DEBUG::2013-11-20 03:07:48,315::task::495::TaskManager.Task::(_debug) Task a6faac54-ebf4-4221-bc12-dd8be7d4381a: moving from state init -> state preparing >Thread-15913544::DEBUG::2013-11-20 03:07:48,318::task::495::TaskManager.Task::(_debug) Task a6faac54-ebf4-4221-bc12-dd8be7d4381a: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000380992889404', 'lastCheck': 1384916858.3912649, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0188131332397', 'lastCheck': 1384916865.935183, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00117206573486', 'lastCheck': 1384916867.5679879, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000330924987793', 'lastCheck': 1384916867.673826, 'valid': True, 'code': 0}} >Thread-15913544::DEBUG::2013-11-20 03:07:48,321::task::495::TaskManager.Task::(_debug) Task a6faac54-ebf4-4221-bc12-dd8be7d4381a: moving from state preparing -> state finished >Thread-15913544::DEBUG::2013-11-20 03:07:48,329::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913544::DEBUG::2013-11-20 03:07:48,330::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913544::DEBUG::2013-11-20 03:07:48,331::task::495::TaskManager.Task::(_debug) Task a6faac54-ebf4-4221-bc12-dd8be7d4381a: ref 0 aborting False >Thread-15913544::INFO::2013-11-20 03:07:48,332::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000380992889404', 'lastCheck': 1384916858.3912649, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0188131332397', 'lastCheck': 1384916865.935183, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00117206573486', 'lastCheck': 1384916867.5679879, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000330924987793', 'lastCheck': 1384916867.673826, 'valid': True, 'code': 0}} >Thread-15913545::DEBUG::2013-11-20 03:07:48,349::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913545::DEBUG::2013-11-20 03:07:48,350::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913545::DEBUG::2013-11-20 03:07:48,350::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913545::DEBUG::2013-11-20 03:07:48,351::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Dummy-15912651::DEBUG::2013-11-20 03:07:48,489::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.078454 s, 13.1 MB/s\n'; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-20 03:07:49,390::libvirtvm::1401::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::event Started detail 0 opaque None >Thread-15913510::DEBUG::2013-11-20 03:07:49,467::utils::307::vm.Vm::(start) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Start statistics collection >Thread-15913546::DEBUG::2013-11-20 03:07:49,468::utils::336::vm.Vm::(run) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Stats thread started >Thread-15913546::INFO::2013-11-20 03:07:49,471::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913547::DEBUG::2013-11-20 03:07:49,471::guestIF::47::root::(_connect) Attempting connection to /var/lib/libvirt/qemu/channels/KLEE-AXDB01.com.redhat.rhevm.vdsm >Thread-15913546::DEBUG::2013-11-20 03:07:49,473::task::495::TaskManager.Task::(_debug) Task f444ca60-691d-4316-aef0-97cc13b088af: moving from state init -> state preparing >Thread-15913547::DEBUG::2013-11-20 03:07:49,474::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /usr/share/vdsm/prepare-vmchannel /var/lib/libvirt/qemu/channels/KLEE-AXDB01.com.redhat.rhevm.vdsm' (cwd None) >Thread-15913546::DEBUG::2013-11-20 03:07:49,476::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`819f1538-83b7-44ff-8be5-83ec25c2b021`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913510::WARNING::2013-11-20 03:07:49,477::libvirtvm::1124::vm.Vm::(_readPauseCode) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::_readPauseCode unsupported by libvirt vm >Thread-15913546::DEBUG::2013-11-20 03:07:49,477::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913510::DEBUG::2013-11-20 03:07:49,774::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /usr/share/vdsm/get-vm-pid KLEE-AXDB01' (cwd None) >Thread-15913546::DEBUG::2013-11-20 03:07:49,785::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:07:50,069::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`819f1538-83b7-44ff-8be5-83ec25c2b021`::Granted request >Thread-15913547::DEBUG::2013-11-20 03:07:50,074::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913546::DEBUG::2013-11-20 03:07:50,076::task::495::TaskManager.Task::(_debug) Task f444ca60-691d-4316-aef0-97cc13b088af: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:07:50,077::task::495::TaskManager.Task::(_debug) Task f444ca60-691d-4316-aef0-97cc13b088af: ref 1 aborting False >Thread-15913547::DEBUG::2013-11-20 03:07:50,078::guestIF::53::root::(_connect) connected to /var/lib/libvirt/qemu/channels/KLEE-AXDB01.com.redhat.rhevm.vdsm >Thread-15913546::DEBUG::2013-11-20 03:07:50,079::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913546::DEBUG::2013-11-20 03:07:50,082::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15913510::DEBUG::2013-11-20 03:07:50,445::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:50,459::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /sbin/service ksmtuned retune' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:50,836::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913546::DEBUG::2013-11-20 03:07:51,118::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913510::DEBUG::2013-11-20 03:07:51,120::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913546::DEBUG::2013-11-20 03:07:51,139::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913546::DEBUG::2013-11-20 03:07:51,141::task::495::TaskManager.Task::(_debug) Task f444ca60-691d-4316-aef0-97cc13b088af: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913510::DEBUG::2013-11-20 03:07:51,141::vm::358::vm.Vm::(_startUnderlyingVm) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::_ongoingCreations released >Thread-15913546::DEBUG::2013-11-20 03:07:51,142::task::495::TaskManager.Task::(_debug) Task f444ca60-691d-4316-aef0-97cc13b088af: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:07:51,145::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:07:51,146::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:07:51,148::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:07:51,151::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:07:51,152::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:07:51,153::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:07:51,155::task::495::TaskManager.Task::(_debug) Task f444ca60-691d-4316-aef0-97cc13b088af: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:07:51,156::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:07:51,157::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:07:51,164::task::495::TaskManager.Task::(_debug) Task c2ca1c41-de9e-4b49-a9cc-f6f6b529040b: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:07:51,166::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`1732c5c8-a7c5-4155-a9d3-ff9292112bd2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:07:51,167::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:07:51,169::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:07:51,170::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`1732c5c8-a7c5-4155-a9d3-ff9292112bd2`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:07:51,172::task::495::TaskManager.Task::(_debug) Task c2ca1c41-de9e-4b49-a9cc-f6f6b529040b: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:07:51,173::task::495::TaskManager.Task::(_debug) Task c2ca1c41-de9e-4b49-a9cc-f6f6b529040b: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:07:51,174::task::495::TaskManager.Task::(_debug) Task c2ca1c41-de9e-4b49-a9cc-f6f6b529040b: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:07:51,175::task::495::TaskManager.Task::(_debug) Task c2ca1c41-de9e-4b49-a9cc-f6f6b529040b: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:07:51,175::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:07:51,176::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:07:51,176::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:07:51,177::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:07:51,178::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:07:51,179::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:07:51,179::task::495::TaskManager.Task::(_debug) Task c2ca1c41-de9e-4b49-a9cc-f6f6b529040b: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:07:51,180::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:07:51,181::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:07:51,181::task::495::TaskManager.Task::(_debug) Task de2f09d7-026f-47ad-86a2-222ced8d3911: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:07:51,182::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`c02de5cf-49ff-4210-aded-ec70819fdbab`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:07:51,183::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:07:51,183::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:07:51,184::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`c02de5cf-49ff-4210-aded-ec70819fdbab`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:07:51,184::task::495::TaskManager.Task::(_debug) Task de2f09d7-026f-47ad-86a2-222ced8d3911: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:07:51,185::task::495::TaskManager.Task::(_debug) Task de2f09d7-026f-47ad-86a2-222ced8d3911: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:07:51,185::task::495::TaskManager.Task::(_debug) Task de2f09d7-026f-47ad-86a2-222ced8d3911: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:07:51,186::task::495::TaskManager.Task::(_debug) Task de2f09d7-026f-47ad-86a2-222ced8d3911: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:07:51,186::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:07:51,187::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:07:51,187::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:07:51,188::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:07:51,188::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:07:51,189::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:07:51,189::task::495::TaskManager.Task::(_debug) Task de2f09d7-026f-47ad-86a2-222ced8d3911: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:07:51,190::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:07:51,190::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:07:51,191::task::495::TaskManager.Task::(_debug) Task a67a77ff-f969-47db-be08-054dc847e00c: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:07:51,192::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`f2384e86-1bcc-4d7f-b4ab-68026dc8dc6b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:07:51,192::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:07:51,193::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:07:51,193::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`f2384e86-1bcc-4d7f-b4ab-68026dc8dc6b`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:07:51,194::task::495::TaskManager.Task::(_debug) Task a67a77ff-f969-47db-be08-054dc847e00c: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:07:51,194::task::495::TaskManager.Task::(_debug) Task a67a77ff-f969-47db-be08-054dc847e00c: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:07:51,195::task::495::TaskManager.Task::(_debug) Task a67a77ff-f969-47db-be08-054dc847e00c: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:07:51,195::task::495::TaskManager.Task::(_debug) Task a67a77ff-f969-47db-be08-054dc847e00c: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:07:51,196::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:07:51,196::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:07:51,197::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:07:51,197::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:07:51,198::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:07:51,198::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:07:51,199::task::495::TaskManager.Task::(_debug) Task a67a77ff-f969-47db-be08-054dc847e00c: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:07:51,199::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:07:51,462::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0957757 s, 10.7 MB/s\n'; <rc> = 0 >Thread-15913549::DEBUG::2013-11-20 03:07:51,650::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15913549::DEBUG::2013-11-20 03:07:51,651::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::CPU stats not available >Thread-15913549::DEBUG::2013-11-20 03:07:51,651::libvirtvm::202::vm.Vm::(_getNetworkStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Network stats not available >Thread-15913549::DEBUG::2013-11-20 03:07:51,652::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913549::DEBUG::2013-11-20 03:07:51,653::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913549::DEBUG::2013-11-20 03:07:51,653::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913549::DEBUG::2013-11-20 03:07:51,654::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913549::DEBUG::2013-11-20 03:07:51,654::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913549::DEBUG::2013-11-20 03:07:51,655::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913549::DEBUG::2013-11-20 03:07:51,655::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913549::DEBUG::2013-11-20 03:07:51,656::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913549::DEBUG::2013-11-20 03:07:51,657::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '47120', 'displayIp': '0', 'displayPort': u'5904', 'session': 'Unknown', 'displaySecurePort': u'5905', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet3': {'state': 'unknown', 'macAddr': u'00:1a:4a:a8:64:08', 'speed': '1000', 'name': u'vnet3'}}, 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '0', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '0', 'truesize': '64424509440'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'truesize': '107374182400'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'truesize': '150323855360'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'truesize': '429496729600'}}, 'boot': 'cd', 'statsAge': '2.18', 'cpuIdle': '100.00', 'elapsedTime': '43', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913550::DEBUG::2013-11-20 03:07:51,663::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4b8ed9c8-dfb1-4de4-935d-6ba4009567b9',) {} >Thread-15913550::DEBUG::2013-11-20 03:07:51,664::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913550::DEBUG::2013-11-20 03:07:51,664::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913550::DEBUG::2013-11-20 03:07:51,665::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913550::DEBUG::2013-11-20 03:07:51,666::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913550::DEBUG::2013-11-20 03:07:51,666::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '46387', 'displayIp': '0', 'displayPort': u'5902', 'session': 'Unknown', 'displaySecurePort': u'5903', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet1': {'macAddr': u'00:1a:4a:a8:64:05', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet1'}, u'vnet2': {'macAddr': u'00:1a:4a:a8:64:0b', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet2'}}, 'vmId': '4b8ed9c8-dfb1-4de4-935d-6ba4009567b9', 'monitorResponse': '0', 'cpuUser': '75.79', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '30064771072', 'writeLatency': '0', 'imageID': 'b0980185-7de1-4e3b-bd76-4495169ab08b', 'flushLatency': '0', 'truesize': '30064771072'}, u'vdb': {'readLatency': '0', 'apparentsize': '826781204480', 'writeLatency': '0', 'imageID': '0b89347c-4654-42bb-a8d7-1a627130b7bd', 'flushLatency': '0', 'truesize': '826781204480'}}, 'boot': 'c', 'statsAge': '1.92', 'cpuIdle': '24.21', 'elapsedTime': '54', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '3601', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913551::DEBUG::2013-11-20 03:07:53,026::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913551::INFO::2013-11-20 03:07:53,027::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913551::DEBUG::2013-11-20 03:07:53,028::task::495::TaskManager.Task::(_debug) Task 7a7ac2b1-3ca1-4bbe-97d3-409fd7cf0aa7: moving from state init -> state preparing >Thread-15913551::DEBUG::2013-11-20 03:07:53,028::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913551::DEBUG::2013-11-20 03:07:53,029::task::495::TaskManager.Task::(_debug) Task 7a7ac2b1-3ca1-4bbe-97d3-409fd7cf0aa7: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913551::DEBUG::2013-11-20 03:07:53,030::task::495::TaskManager.Task::(_debug) Task 7a7ac2b1-3ca1-4bbe-97d3-409fd7cf0aa7: moving from state preparing -> state finished >Thread-15913551::DEBUG::2013-11-20 03:07:53,030::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913551::DEBUG::2013-11-20 03:07:53,031::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913551::DEBUG::2013-11-20 03:07:53,031::task::495::TaskManager.Task::(_debug) Task 7a7ac2b1-3ca1-4bbe-97d3-409fd7cf0aa7: ref 0 aborting False >Thread-15913551::INFO::2013-11-20 03:07:53,032::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913552::DEBUG::2013-11-20 03:07:53,043::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913552::INFO::2013-11-20 03:07:53,044::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913552::DEBUG::2013-11-20 03:07:53,045::task::495::TaskManager.Task::(_debug) Task 21a0d987-41e7-4f88-93c8-80c60fe40ed3: moving from state init -> state preparing >Thread-15913552::DEBUG::2013-11-20 03:07:53,046::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`0dfdfd46-42a1-497f-98db-484f871c9067`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913552::DEBUG::2013-11-20 03:07:53,046::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913552::DEBUG::2013-11-20 03:07:53,047::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913552::DEBUG::2013-11-20 03:07:53,048::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`0dfdfd46-42a1-497f-98db-484f871c9067`::Granted request >Thread-15913552::DEBUG::2013-11-20 03:07:53,049::task::495::TaskManager.Task::(_debug) Task 21a0d987-41e7-4f88-93c8-80c60fe40ed3: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913552::DEBUG::2013-11-20 03:07:53,049::task::495::TaskManager.Task::(_debug) Task 21a0d987-41e7-4f88-93c8-80c60fe40ed3: ref 1 aborting False >Thread-15913552::WARNING::2013-11-20 03:07:53,051::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913552::DEBUG::2013-11-20 03:07:53,052::task::495::TaskManager.Task::(_debug) Task 21a0d987-41e7-4f88-93c8-80c60fe40ed3: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913552::DEBUG::2013-11-20 03:07:53,053::task::495::TaskManager.Task::(_debug) Task 21a0d987-41e7-4f88-93c8-80c60fe40ed3: moving from state preparing -> state finished >Thread-15913552::DEBUG::2013-11-20 03:07:53,054::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913552::DEBUG::2013-11-20 03:07:53,054::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913552::DEBUG::2013-11-20 03:07:53,055::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913552::DEBUG::2013-11-20 03:07:53,055::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913552::DEBUG::2013-11-20 03:07:53,056::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913552::DEBUG::2013-11-20 03:07:53,057::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913552::DEBUG::2013-11-20 03:07:53,057::task::495::TaskManager.Task::(_debug) Task 21a0d987-41e7-4f88-93c8-80c60fe40ed3: ref 0 aborting False >Thread-15913552::INFO::2013-11-20 03:07:53,058::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:07:53,476::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:53,992::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0706125 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15913554::DEBUG::2013-11-20 03:07:54,802::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15913554::DEBUG::2013-11-20 03:07:54,803::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::CPU stats not available >Thread-15913554::DEBUG::2013-11-20 03:07:54,821::libvirtvm::202::vm.Vm::(_getNetworkStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Network stats not available >Thread-15913554::DEBUG::2013-11-20 03:07:54,821::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913554::DEBUG::2013-11-20 03:07:54,822::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913554::DEBUG::2013-11-20 03:07:54,822::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913554::DEBUG::2013-11-20 03:07:54,822::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913554::DEBUG::2013-11-20 03:07:54,822::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913554::DEBUG::2013-11-20 03:07:54,823::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913554::DEBUG::2013-11-20 03:07:54,823::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913554::DEBUG::2013-11-20 03:07:54,823::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913554::DEBUG::2013-11-20 03:07:54,824::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '47120', 'displayIp': '0', 'displayPort': u'5904', 'session': 'Unknown', 'displaySecurePort': u'5905', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet3': {'state': 'unknown', 'macAddr': u'00:1a:4a:a8:64:08', 'speed': '1000', 'name': u'vnet3'}}, 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '0', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '0', 'truesize': '64424509440'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'truesize': '107374182400'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'truesize': '150323855360'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'truesize': '429496729600'}}, 'boot': 'cd', 'statsAge': '1.53', 'cpuIdle': '100.00', 'elapsedTime': '46', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913555::DEBUG::2013-11-20 03:07:54,837::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4b8ed9c8-dfb1-4de4-935d-6ba4009567b9',) {} >Thread-15913555::DEBUG::2013-11-20 03:07:54,838::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913555::DEBUG::2013-11-20 03:07:54,838::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913555::DEBUG::2013-11-20 03:07:54,838::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913555::DEBUG::2013-11-20 03:07:54,839::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913555::DEBUG::2013-11-20 03:07:54,839::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '46387', 'displayIp': '0', 'displayPort': u'5902', 'session': 'Unknown', 'displaySecurePort': u'5903', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet1': {'macAddr': u'00:1a:4a:a8:64:05', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet1'}, u'vnet2': {'macAddr': u'00:1a:4a:a8:64:0b', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet2'}}, 'vmId': '4b8ed9c8-dfb1-4de4-935d-6ba4009567b9', 'monitorResponse': '0', 'cpuUser': '74.52', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '30064771072', 'writeLatency': '0', 'imageID': 'b0980185-7de1-4e3b-bd76-4495169ab08b', 'flushLatency': '0', 'truesize': '30064771072'}, u'vdb': {'readLatency': '0', 'apparentsize': '826781204480', 'writeLatency': '0', 'imageID': '0b89347c-4654-42bb-a8d7-1a627130b7bd', 'flushLatency': '0', 'truesize': '826781204480'}}, 'boot': 'c', 'statsAge': '0.73', 'cpuIdle': '25.48', 'elapsedTime': '57', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '3601', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:07:56,007::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:56,341::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0365219 s, 28.0 MB/s\n'; <rc> = 0 >Thread-15913557::DEBUG::2013-11-20 03:07:57,862::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15913557::DEBUG::2013-11-20 03:07:57,862::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::CPU stats not available >Thread-15913557::DEBUG::2013-11-20 03:07:57,863::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913557::DEBUG::2013-11-20 03:07:57,863::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913557::DEBUG::2013-11-20 03:07:57,864::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913557::DEBUG::2013-11-20 03:07:57,864::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913557::DEBUG::2013-11-20 03:07:57,864::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913557::DEBUG::2013-11-20 03:07:57,864::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913557::DEBUG::2013-11-20 03:07:57,865::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913557::DEBUG::2013-11-20 03:07:57,865::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913557::DEBUG::2013-11-20 03:07:57,866::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '47120', 'displayIp': '0', 'displayPort': u'5904', 'session': 'Unknown', 'displaySecurePort': u'5905', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet3': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet3'}}, 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '0', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '0', 'truesize': '64424509440'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'truesize': '107374182400'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'truesize': '150323855360'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'truesize': '429496729600'}}, 'boot': 'cd', 'statsAge': '0.59', 'cpuIdle': '100.00', 'elapsedTime': '49', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913558::DEBUG::2013-11-20 03:07:57,870::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4b8ed9c8-dfb1-4de4-935d-6ba4009567b9',) {} >Thread-15913558::DEBUG::2013-11-20 03:07:57,871::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913558::DEBUG::2013-11-20 03:07:57,871::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913558::DEBUG::2013-11-20 03:07:57,872::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913558::DEBUG::2013-11-20 03:07:57,872::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913558::DEBUG::2013-11-20 03:07:57,872::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '46387', 'displayIp': '0', 'displayPort': u'5902', 'session': 'Unknown', 'displaySecurePort': u'5903', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet1': {'macAddr': u'00:1a:4a:a8:64:05', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet1'}, u'vnet2': {'macAddr': u'00:1a:4a:a8:64:0b', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet2'}}, 'vmId': '4b8ed9c8-dfb1-4de4-935d-6ba4009567b9', 'monitorResponse': '0', 'cpuUser': '74.52', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '30064771072', 'writeLatency': '0', 'imageID': 'b0980185-7de1-4e3b-bd76-4495169ab08b', 'flushLatency': '0', 'truesize': '30064771072'}, u'vdb': {'readLatency': '0', 'apparentsize': '826781204480', 'writeLatency': '0', 'imageID': '0b89347c-4654-42bb-a8d7-1a627130b7bd', 'flushLatency': '0', 'truesize': '826781204480'}}, 'boot': 'c', 'statsAge': '0.70', 'cpuIdle': '25.48', 'elapsedTime': '60', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '3601', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:07:58,350::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:07:58,736::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0465669 s, 22.0 MB/s\n'; <rc> = 0 >Thread-15913519::DEBUG::2013-11-20 03:07:59,666::guestIF::97::vm.Vm::(_handleMessage) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::u'None' >Dummy-15912651::DEBUG::2013-11-20 03:08:00,750::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913560::DEBUG::2013-11-20 03:08:01,114::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15913560::DEBUG::2013-11-20 03:08:01,115::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::CPU stats not available >Thread-15913560::DEBUG::2013-11-20 03:08:01,116::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913560::DEBUG::2013-11-20 03:08:01,116::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913560::DEBUG::2013-11-20 03:08:01,117::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913560::DEBUG::2013-11-20 03:08:01,117::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913560::DEBUG::2013-11-20 03:08:01,117::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913560::DEBUG::2013-11-20 03:08:01,118::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913560::DEBUG::2013-11-20 03:08:01,118::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913560::DEBUG::2013-11-20 03:08:01,118::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913560::DEBUG::2013-11-20 03:08:01,119::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '47120', 'displayIp': '0', 'displayPort': u'5904', 'session': 'Unknown', 'displaySecurePort': u'5905', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet3': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet3'}}, 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '0', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '0', 'truesize': '64424509440'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'truesize': '107374182400'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'truesize': '150323855360'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'truesize': '429496729600'}}, 'boot': 'cd', 'statsAge': '1.84', 'cpuIdle': '100.00', 'elapsedTime': '53', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:08:01,329::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0896364 s, 11.4 MB/s\n'; <rc> = 0 >Thread-15913561::DEBUG::2013-11-20 03:08:03,287::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913561::INFO::2013-11-20 03:08:03,288::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913561::DEBUG::2013-11-20 03:08:03,288::task::495::TaskManager.Task::(_debug) Task 98e46aec-58ac-425d-8c64-1a7d11292d5d: moving from state init -> state preparing >Thread-15913561::DEBUG::2013-11-20 03:08:03,289::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913561::DEBUG::2013-11-20 03:08:03,289::task::495::TaskManager.Task::(_debug) Task 98e46aec-58ac-425d-8c64-1a7d11292d5d: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913561::DEBUG::2013-11-20 03:08:03,289::task::495::TaskManager.Task::(_debug) Task 98e46aec-58ac-425d-8c64-1a7d11292d5d: moving from state preparing -> state finished >Thread-15913561::DEBUG::2013-11-20 03:08:03,290::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913561::DEBUG::2013-11-20 03:08:03,290::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913561::DEBUG::2013-11-20 03:08:03,290::task::495::TaskManager.Task::(_debug) Task 98e46aec-58ac-425d-8c64-1a7d11292d5d: ref 0 aborting False >Thread-15913561::INFO::2013-11-20 03:08:03,291::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913562::DEBUG::2013-11-20 03:08:03,295::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913562::INFO::2013-11-20 03:08:03,295::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913562::DEBUG::2013-11-20 03:08:03,296::task::495::TaskManager.Task::(_debug) Task 590c291b-e647-4407-822b-fb1b2234547f: moving from state init -> state preparing >Thread-15913562::DEBUG::2013-11-20 03:08:03,296::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`dfce8ae3-d824-4608-846d-e2a0060d0c67`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913562::DEBUG::2013-11-20 03:08:03,297::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913562::DEBUG::2013-11-20 03:08:03,297::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913562::DEBUG::2013-11-20 03:08:03,298::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`dfce8ae3-d824-4608-846d-e2a0060d0c67`::Granted request >Thread-15913562::DEBUG::2013-11-20 03:08:03,298::task::495::TaskManager.Task::(_debug) Task 590c291b-e647-4407-822b-fb1b2234547f: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913562::DEBUG::2013-11-20 03:08:03,299::task::495::TaskManager.Task::(_debug) Task 590c291b-e647-4407-822b-fb1b2234547f: ref 1 aborting False >Thread-15913562::WARNING::2013-11-20 03:08:03,301::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913562::DEBUG::2013-11-20 03:08:03,310::task::495::TaskManager.Task::(_debug) Task 590c291b-e647-4407-822b-fb1b2234547f: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913562::DEBUG::2013-11-20 03:08:03,312::task::495::TaskManager.Task::(_debug) Task 590c291b-e647-4407-822b-fb1b2234547f: moving from state preparing -> state finished >Thread-15913562::DEBUG::2013-11-20 03:08:03,313::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913562::DEBUG::2013-11-20 03:08:03,314::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913562::DEBUG::2013-11-20 03:08:03,314::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913562::DEBUG::2013-11-20 03:08:03,315::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913562::DEBUG::2013-11-20 03:08:03,315::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913562::DEBUG::2013-11-20 03:08:03,315::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913562::DEBUG::2013-11-20 03:08:03,316::task::495::TaskManager.Task::(_debug) Task 590c291b-e647-4407-822b-fb1b2234547f: ref 0 aborting False >Thread-15913562::INFO::2013-11-20 03:08:03,316::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:08:03,347::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913563::INFO::2013-11-20 03:08:04,141::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913563::DEBUG::2013-11-20 03:08:04,142::task::495::TaskManager.Task::(_debug) Task e961d9cf-e7bc-40e8-8fbd-4e42b5f664d5: moving from state init -> state preparing >Thread-15913563::DEBUG::2013-11-20 03:08:04,143::task::495::TaskManager.Task::(_debug) Task e961d9cf-e7bc-40e8-8fbd-4e42b5f664d5: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000277996063232', 'lastCheck': 1384916879.1610751, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00135111808777', 'lastCheck': 1384916876.4529259, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000972986221313', 'lastCheck': 1384916877.8451419, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00042200088501', 'lastCheck': 1384916877.833935, 'valid': True, 'code': 0}} >Thread-15913563::DEBUG::2013-11-20 03:08:04,143::task::495::TaskManager.Task::(_debug) Task e961d9cf-e7bc-40e8-8fbd-4e42b5f664d5: moving from state preparing -> state finished >Thread-15913563::DEBUG::2013-11-20 03:08:04,144::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913563::DEBUG::2013-11-20 03:08:04,144::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913563::DEBUG::2013-11-20 03:08:04,144::task::495::TaskManager.Task::(_debug) Task e961d9cf-e7bc-40e8-8fbd-4e42b5f664d5: ref 0 aborting False >Thread-15913563::INFO::2013-11-20 03:08:04,145::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000277996063232', 'lastCheck': 1384916879.1610751, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00135111808777', 'lastCheck': 1384916876.4529259, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000972986221313', 'lastCheck': 1384916877.8451419, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00042200088501', 'lastCheck': 1384916877.833935, 'valid': True, 'code': 0}} >Thread-15913564::DEBUG::2013-11-20 03:08:04,193::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::CPU stats not available >Thread-15913564::DEBUG::2013-11-20 03:08:04,194::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913564::DEBUG::2013-11-20 03:08:04,194::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913564::DEBUG::2013-11-20 03:08:04,194::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913564::DEBUG::2013-11-20 03:08:04,195::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913564::DEBUG::2013-11-20 03:08:04,195::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913564::DEBUG::2013-11-20 03:08:04,195::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913564::DEBUG::2013-11-20 03:08:04,196::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913564::DEBUG::2013-11-20 03:08:04,196::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913564::DEBUG::2013-11-20 03:08:04,197::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913564::DEBUG::2013-11-20 03:08:04,198::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913564::DEBUG::2013-11-20 03:08:04,198::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913564::DEBUG::2013-11-20 03:08:04,199::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Dummy-15912651::DEBUG::2013-11-20 03:08:04,349::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0787409 s, 13.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:08:06,363::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:08:07,312::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0512357 s, 20.0 MB/s\n'; <rc> = 0 >Thread-15913566::DEBUG::2013-11-20 03:08:07,484::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15913566::DEBUG::2013-11-20 03:08:07,485::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913566::DEBUG::2013-11-20 03:08:07,485::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913566::DEBUG::2013-11-20 03:08:07,486::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913566::DEBUG::2013-11-20 03:08:07,486::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913566::DEBUG::2013-11-20 03:08:07,486::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913566::DEBUG::2013-11-20 03:08:07,487::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913566::DEBUG::2013-11-20 03:08:07,487::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913566::DEBUG::2013-11-20 03:08:07,487::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913566::DEBUG::2013-11-20 03:08:07,488::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '47120', 'displayIp': '0', 'displayPort': u'5904', 'session': 'Unknown', 'displaySecurePort': u'5905', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet3': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet3'}}, 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'monitorResponse': '0', 'cpuUser': '205.51', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '0', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '0', 'truesize': '64424509440'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'truesize': '107374182400'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'truesize': '150323855360'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'truesize': '429496729600'}}, 'boot': 'cd', 'statsAge': '0.19', 'cpuIdle': '0.00', 'elapsedTime': '59', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:08:09,321::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:08:10,126::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0773255 s, 13.2 MB/s\n'; <rc> = 0 >Thread-15913568::DEBUG::2013-11-20 03:08:10,517::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('6574d88c-00ac-46f4-99ce-a2726932bfda',) {} >Thread-15913568::DEBUG::2013-11-20 03:08:10,517::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913568::DEBUG::2013-11-20 03:08:10,518::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913568::DEBUG::2013-11-20 03:08:10,518::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913568::DEBUG::2013-11-20 03:08:10,518::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913568::DEBUG::2013-11-20 03:08:10,519::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913568::DEBUG::2013-11-20 03:08:10,519::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913568::DEBUG::2013-11-20 03:08:10,519::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913568::DEBUG::2013-11-20 03:08:10,520::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913568::DEBUG::2013-11-20 03:08:10,520::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '47120', 'displayIp': '0', 'displayPort': u'5904', 'session': 'Unknown', 'displaySecurePort': u'5905', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet3': {'macAddr': u'00:1a:4a:a8:64:08', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet3'}}, 'vmId': '6574d88c-00ac-46f4-99ce-a2726932bfda', 'monitorResponse': '0', 'cpuUser': '205.51', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '64424509440', 'writeLatency': '0', 'imageID': 'c4460480-32d3-469f-b0f4-c03bb6d74e41', 'flushLatency': '0', 'truesize': '64424509440'}, u'vdc': {'readLatency': '0', 'apparentsize': '107374182400', 'writeLatency': '0', 'imageID': '13efb45b-f4d2-4c94-bbdb-91ce9b9b7321', 'flushLatency': '0', 'truesize': '107374182400'}, u'vdb': {'readLatency': '0', 'apparentsize': '150323855360', 'writeLatency': '0', 'imageID': 'df8b790c-c762-4feb-9378-d64a321ce43c', 'flushLatency': '0', 'truesize': '150323855360'}, u'vdd': {'readLatency': '0', 'apparentsize': '429496729600', 'writeLatency': '0', 'imageID': 'd368307e-425e-423a-85b7-266c1e4a99ec', 'flushLatency': '0', 'truesize': '429496729600'}}, 'boot': 'cd', 'statsAge': '1.22', 'cpuIdle': '0.00', 'elapsedTime': '62', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:08:12,140::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:08:12,685::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0680285 s, 15.1 MB/s\n'; <rc> = 0 >Thread-15913569::DEBUG::2013-11-20 03:08:13,560::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913569::INFO::2013-11-20 03:08:13,561::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913569::DEBUG::2013-11-20 03:08:13,561::task::495::TaskManager.Task::(_debug) Task c3a25da3-2b87-443c-be84-597a27e73f8d: moving from state init -> state preparing >Thread-15913569::DEBUG::2013-11-20 03:08:13,562::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913569::DEBUG::2013-11-20 03:08:13,562::task::495::TaskManager.Task::(_debug) Task c3a25da3-2b87-443c-be84-597a27e73f8d: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913569::DEBUG::2013-11-20 03:08:13,563::task::495::TaskManager.Task::(_debug) Task c3a25da3-2b87-443c-be84-597a27e73f8d: moving from state preparing -> state finished >Thread-15913569::DEBUG::2013-11-20 03:08:13,563::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913569::DEBUG::2013-11-20 03:08:13,563::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913569::DEBUG::2013-11-20 03:08:13,564::task::495::TaskManager.Task::(_debug) Task c3a25da3-2b87-443c-be84-597a27e73f8d: ref 0 aborting False >Thread-15913569::INFO::2013-11-20 03:08:13,564::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913570::DEBUG::2013-11-20 03:08:13,568::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913570::INFO::2013-11-20 03:08:13,569::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913570::DEBUG::2013-11-20 03:08:13,569::task::495::TaskManager.Task::(_debug) Task 0cffa33d-0754-449f-85b7-8293e102767e: moving from state init -> state preparing >Thread-15913570::DEBUG::2013-11-20 03:08:13,570::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c455b58a-a9c1-4e40-a53d-e427416135fe`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913570::DEBUG::2013-11-20 03:08:13,570::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913570::DEBUG::2013-11-20 03:08:13,571::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913570::DEBUG::2013-11-20 03:08:13,571::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c455b58a-a9c1-4e40-a53d-e427416135fe`::Granted request >Thread-15913570::DEBUG::2013-11-20 03:08:13,572::task::495::TaskManager.Task::(_debug) Task 0cffa33d-0754-449f-85b7-8293e102767e: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913570::DEBUG::2013-11-20 03:08:13,572::task::495::TaskManager.Task::(_debug) Task 0cffa33d-0754-449f-85b7-8293e102767e: ref 1 aborting False >Thread-15913570::WARNING::2013-11-20 03:08:13,574::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913570::DEBUG::2013-11-20 03:08:13,575::task::495::TaskManager.Task::(_debug) Task 0cffa33d-0754-449f-85b7-8293e102767e: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913570::DEBUG::2013-11-20 03:08:13,576::task::495::TaskManager.Task::(_debug) Task 0cffa33d-0754-449f-85b7-8293e102767e: moving from state preparing -> state finished >Thread-15913570::DEBUG::2013-11-20 03:08:13,576::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913570::DEBUG::2013-11-20 03:08:13,577::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913570::DEBUG::2013-11-20 03:08:13,577::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913570::DEBUG::2013-11-20 03:08:13,578::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913570::DEBUG::2013-11-20 03:08:13,578::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913570::DEBUG::2013-11-20 03:08:13,579::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913570::DEBUG::2013-11-20 03:08:13,579::task::495::TaskManager.Task::(_debug) Task 0cffa33d-0754-449f-85b7-8293e102767e: ref 0 aborting False >Thread-15913570::INFO::2013-11-20 03:08:13,580::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:08:14,696::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:08:15,448::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0537866 s, 19.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:08:17,458::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:08:19,029::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0496902 s, 20.6 MB/s\n'; <rc> = 0 >Thread-15913573::INFO::2013-11-20 03:08:19,917::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913573::DEBUG::2013-11-20 03:08:19,918::task::495::TaskManager.Task::(_debug) Task b80b7b55-7679-49e4-8c9c-8b54d6fb8742: moving from state init -> state preparing >Thread-15913573::DEBUG::2013-11-20 03:08:19,919::task::495::TaskManager.Task::(_debug) Task b80b7b55-7679-49e4-8c9c-8b54d6fb8742: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000360012054443', 'lastCheck': 1384916899.3836329, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00088906288147', 'lastCheck': 1384916896.6111939, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0183851718903', 'lastCheck': 1384916897.970094, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00042200088501', 'lastCheck': 1384916897.9469571, 'valid': True, 'code': 0}} >Thread-15913573::DEBUG::2013-11-20 03:08:19,919::task::495::TaskManager.Task::(_debug) Task b80b7b55-7679-49e4-8c9c-8b54d6fb8742: moving from state preparing -> state finished >Thread-15913573::DEBUG::2013-11-20 03:08:19,920::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913573::DEBUG::2013-11-20 03:08:19,920::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913573::DEBUG::2013-11-20 03:08:19,920::task::495::TaskManager.Task::(_debug) Task b80b7b55-7679-49e4-8c9c-8b54d6fb8742: ref 0 aborting False >Thread-15913573::INFO::2013-11-20 03:08:19,921::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000360012054443', 'lastCheck': 1384916899.3836329, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00088906288147', 'lastCheck': 1384916896.6111939, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0183851718903', 'lastCheck': 1384916897.970094, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00042200088501', 'lastCheck': 1384916897.9469571, 'valid': True, 'code': 0}} >Thread-15913574::DEBUG::2013-11-20 03:08:19,933::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913574::DEBUG::2013-11-20 03:08:19,933::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913574::DEBUG::2013-11-20 03:08:19,934::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913574::DEBUG::2013-11-20 03:08:19,934::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913574::DEBUG::2013-11-20 03:08:19,934::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913574::DEBUG::2013-11-20 03:08:19,935::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913574::DEBUG::2013-11-20 03:08:19,935::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913574::DEBUG::2013-11-20 03:08:19,935::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913574::DEBUG::2013-11-20 03:08:19,936::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913574::DEBUG::2013-11-20 03:08:19,936::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk stats not available >Thread-15913574::DEBUG::2013-11-20 03:08:19,937::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Thread-15913574::DEBUG::2013-11-20 03:08:19,937::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::Disk latency not available >Dummy-15912651::DEBUG::2013-11-20 03:08:21,044::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913575::DEBUG::2013-11-20 03:08:21,232::clientIF::54::vds::(wrapper) [192.168.100.135]::call setVmTicket with ('4b8ed9c8-dfb1-4de4-935d-6ba4009567b9', 'FNUITVoWrGEk', '120') {} >Thread-15913575::DEBUG::2013-11-20 03:08:21,254::clientIF::59::vds::(wrapper) return setVmTicket with {'status': {'message': 'Done', 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:08:21,720::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0788343 s, 13.0 MB/s\n'; <rc> = 0 >Thread-15913518::INFO::2013-11-20 03:08:22,460::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:08:22,461::task::495::TaskManager.Task::(_debug) Task 831b2274-9ef3-4ec4-8060-f5380ea4c3a5: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:08:22,462::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e8b9c4b4-04b6-4a68-b2ef-ac414198ebaa`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:08:22,462::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:08:22,463::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:08:22,463::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e8b9c4b4-04b6-4a68-b2ef-ac414198ebaa`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:08:22,464::task::495::TaskManager.Task::(_debug) Task 831b2274-9ef3-4ec4-8060-f5380ea4c3a5: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:08:22,464::task::495::TaskManager.Task::(_debug) Task 831b2274-9ef3-4ec4-8060-f5380ea4c3a5: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:08:22,464::task::495::TaskManager.Task::(_debug) Task 831b2274-9ef3-4ec4-8060-f5380ea4c3a5: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:08:22,465::task::495::TaskManager.Task::(_debug) Task 831b2274-9ef3-4ec4-8060-f5380ea4c3a5: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:08:22,465::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:08:22,466::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:08:22,466::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:08:22,466::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:08:22,467::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:08:22,467::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:08:22,467::task::495::TaskManager.Task::(_debug) Task 831b2274-9ef3-4ec4-8060-f5380ea4c3a5: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:08:22,468::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:08:22,468::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:08:22,468::task::495::TaskManager.Task::(_debug) Task 0d59b8ac-ef9b-4b2e-9d2f-3b8f8705aa60: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:08:22,469::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`21ccf41b-aa1e-442b-92ea-c55edeb477a7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:08:22,469::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:08:22,470::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:08:22,470::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`21ccf41b-aa1e-442b-92ea-c55edeb477a7`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:08:22,471::task::495::TaskManager.Task::(_debug) Task 0d59b8ac-ef9b-4b2e-9d2f-3b8f8705aa60: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:08:22,471::task::495::TaskManager.Task::(_debug) Task 0d59b8ac-ef9b-4b2e-9d2f-3b8f8705aa60: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:08:22,471::task::495::TaskManager.Task::(_debug) Task 0d59b8ac-ef9b-4b2e-9d2f-3b8f8705aa60: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:08:22,472::task::495::TaskManager.Task::(_debug) Task 0d59b8ac-ef9b-4b2e-9d2f-3b8f8705aa60: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:08:22,472::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:08:22,473::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:08:22,473::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:08:22,473::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:08:22,474::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:08:22,474::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:08:22,474::task::495::TaskManager.Task::(_debug) Task 0d59b8ac-ef9b-4b2e-9d2f-3b8f8705aa60: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:08:22,475::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:08:23,730::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913577::DEBUG::2013-11-20 03:08:24,194::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913577::INFO::2013-11-20 03:08:24,196::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913577::DEBUG::2013-11-20 03:08:24,197::task::495::TaskManager.Task::(_debug) Task 89ffead0-6eee-4e2a-88bd-afd6d0400536: moving from state init -> state preparing >Thread-15913577::DEBUG::2013-11-20 03:08:24,198::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913577::DEBUG::2013-11-20 03:08:24,198::task::495::TaskManager.Task::(_debug) Task 89ffead0-6eee-4e2a-88bd-afd6d0400536: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913577::DEBUG::2013-11-20 03:08:24,198::task::495::TaskManager.Task::(_debug) Task 89ffead0-6eee-4e2a-88bd-afd6d0400536: moving from state preparing -> state finished >Thread-15913577::DEBUG::2013-11-20 03:08:24,199::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913577::DEBUG::2013-11-20 03:08:24,199::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913577::DEBUG::2013-11-20 03:08:24,200::task::495::TaskManager.Task::(_debug) Task 89ffead0-6eee-4e2a-88bd-afd6d0400536: ref 0 aborting False >Thread-15913577::INFO::2013-11-20 03:08:24,200::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913578::DEBUG::2013-11-20 03:08:24,205::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913578::INFO::2013-11-20 03:08:24,206::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913578::DEBUG::2013-11-20 03:08:24,207::task::495::TaskManager.Task::(_debug) Task 0652dd3a-4a38-4b10-b2bd-45e4410c588e: moving from state init -> state preparing >Thread-15913578::DEBUG::2013-11-20 03:08:24,208::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fc41f4c8-78dc-45de-9b68-b648fc31fc9f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913578::DEBUG::2013-11-20 03:08:24,209::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913578::DEBUG::2013-11-20 03:08:24,210::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913578::DEBUG::2013-11-20 03:08:24,210::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fc41f4c8-78dc-45de-9b68-b648fc31fc9f`::Granted request >Thread-15913578::DEBUG::2013-11-20 03:08:24,212::task::495::TaskManager.Task::(_debug) Task 0652dd3a-4a38-4b10-b2bd-45e4410c588e: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913578::DEBUG::2013-11-20 03:08:24,212::task::495::TaskManager.Task::(_debug) Task 0652dd3a-4a38-4b10-b2bd-45e4410c588e: ref 1 aborting False >Thread-15913578::WARNING::2013-11-20 03:08:24,214::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913578::DEBUG::2013-11-20 03:08:24,220::task::495::TaskManager.Task::(_debug) Task 0652dd3a-4a38-4b10-b2bd-45e4410c588e: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913578::DEBUG::2013-11-20 03:08:24,221::task::495::TaskManager.Task::(_debug) Task 0652dd3a-4a38-4b10-b2bd-45e4410c588e: moving from state preparing -> state finished >Thread-15913578::DEBUG::2013-11-20 03:08:24,221::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913578::DEBUG::2013-11-20 03:08:24,222::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913578::DEBUG::2013-11-20 03:08:24,223::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913578::DEBUG::2013-11-20 03:08:24,223::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913578::DEBUG::2013-11-20 03:08:24,224::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913578::DEBUG::2013-11-20 03:08:24,224::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913578::DEBUG::2013-11-20 03:08:24,224::task::495::TaskManager.Task::(_debug) Task 0652dd3a-4a38-4b10-b2bd-45e4410c588e: ref 0 aborting False >Thread-15913578::INFO::2013-11-20 03:08:24,225::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:08:24,660::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0608697 s, 16.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:08:26,670::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:08:27,098::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0526674 s, 19.4 MB/s\n'; <rc> = 0 >Thread-15912392::INFO::2013-11-20 03:08:28,949::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:08:28,949::task::495::TaskManager.Task::(_debug) Task 965bfc21-b78b-4178-a469-d15f901b10c3: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:08:28,950::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ad1162e6-c60d-4254-b247-28f7475f3e32`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:08:28,951::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:08:28,951::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:08:28,951::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ad1162e6-c60d-4254-b247-28f7475f3e32`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:08:28,952::task::495::TaskManager.Task::(_debug) Task 965bfc21-b78b-4178-a469-d15f901b10c3: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:08:28,952::task::495::TaskManager.Task::(_debug) Task 965bfc21-b78b-4178-a469-d15f901b10c3: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:08:28,953::task::495::TaskManager.Task::(_debug) Task 965bfc21-b78b-4178-a469-d15f901b10c3: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:08:28,953::task::495::TaskManager.Task::(_debug) Task 965bfc21-b78b-4178-a469-d15f901b10c3: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:08:28,953::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:08:28,954::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:08:28,954::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:08:28,955::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:08:28,955::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:08:28,955::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:08:28,956::task::495::TaskManager.Task::(_debug) Task 965bfc21-b78b-4178-a469-d15f901b10c3: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:08:28,956::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:08:29,108::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:08:30,225::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0476025 s, 21.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:08:32,237::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:08:32,795::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0429168 s, 23.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:08:34,804::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913582::DEBUG::2013-11-20 03:08:35,483::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913582::INFO::2013-11-20 03:08:35,484::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913582::DEBUG::2013-11-20 03:08:35,484::task::495::TaskManager.Task::(_debug) Task 6cda29ce-bb2c-4a7a-8682-6e161d3a96de: moving from state init -> state preparing >Thread-15913582::DEBUG::2013-11-20 03:08:35,485::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913582::DEBUG::2013-11-20 03:08:35,485::task::495::TaskManager.Task::(_debug) Task 6cda29ce-bb2c-4a7a-8682-6e161d3a96de: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913582::DEBUG::2013-11-20 03:08:35,486::task::495::TaskManager.Task::(_debug) Task 6cda29ce-bb2c-4a7a-8682-6e161d3a96de: moving from state preparing -> state finished >Thread-15913582::DEBUG::2013-11-20 03:08:35,486::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913582::DEBUG::2013-11-20 03:08:35,487::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913582::DEBUG::2013-11-20 03:08:35,487::task::495::TaskManager.Task::(_debug) Task 6cda29ce-bb2c-4a7a-8682-6e161d3a96de: ref 0 aborting False >Thread-15913582::INFO::2013-11-20 03:08:35,487::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913583::DEBUG::2013-11-20 03:08:35,499::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913583::INFO::2013-11-20 03:08:35,500::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913583::DEBUG::2013-11-20 03:08:35,501::task::495::TaskManager.Task::(_debug) Task 9279fd0b-39bb-41f9-8750-b4a6d270853a: moving from state init -> state preparing >Thread-15913583::DEBUG::2013-11-20 03:08:35,502::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3b6d5210-03c7-4eb5-a43d-532ba217d18c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913583::DEBUG::2013-11-20 03:08:35,503::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913583::DEBUG::2013-11-20 03:08:35,504::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913583::DEBUG::2013-11-20 03:08:35,504::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3b6d5210-03c7-4eb5-a43d-532ba217d18c`::Granted request >Thread-15913583::DEBUG::2013-11-20 03:08:35,506::task::495::TaskManager.Task::(_debug) Task 9279fd0b-39bb-41f9-8750-b4a6d270853a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913583::DEBUG::2013-11-20 03:08:35,506::task::495::TaskManager.Task::(_debug) Task 9279fd0b-39bb-41f9-8750-b4a6d270853a: ref 1 aborting False >Thread-15913583::WARNING::2013-11-20 03:08:35,508::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913583::DEBUG::2013-11-20 03:08:35,509::task::495::TaskManager.Task::(_debug) Task 9279fd0b-39bb-41f9-8750-b4a6d270853a: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913583::DEBUG::2013-11-20 03:08:35,510::task::495::TaskManager.Task::(_debug) Task 9279fd0b-39bb-41f9-8750-b4a6d270853a: moving from state preparing -> state finished >Thread-15913583::DEBUG::2013-11-20 03:08:35,511::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913583::DEBUG::2013-11-20 03:08:35,511::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913583::DEBUG::2013-11-20 03:08:35,512::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913583::DEBUG::2013-11-20 03:08:35,513::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913583::DEBUG::2013-11-20 03:08:35,513::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913583::DEBUG::2013-11-20 03:08:35,514::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913583::DEBUG::2013-11-20 03:08:35,515::task::495::TaskManager.Task::(_debug) Task 9279fd0b-39bb-41f9-8750-b4a6d270853a: ref 0 aborting False >Thread-15913583::INFO::2013-11-20 03:08:35,515::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:08:35,582::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0594919 s, 17.2 MB/s\n'; <rc> = 0 >Thread-15913584::INFO::2013-11-20 03:08:35,606::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913584::DEBUG::2013-11-20 03:08:35,606::task::495::TaskManager.Task::(_debug) Task a2d5174c-8702-4910-aebb-8b7c7748bea2: moving from state init -> state preparing >Thread-15913584::DEBUG::2013-11-20 03:08:35,607::task::495::TaskManager.Task::(_debug) Task a2d5174c-8702-4910-aebb-8b7c7748bea2: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000328779220581', 'lastCheck': 1384916909.396462, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000893115997314', 'lastCheck': 1384916906.6511099, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000971794128418', 'lastCheck': 1384916907.982836, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000352144241333', 'lastCheck': 1384916907.957339, 'valid': True, 'code': 0}} >Thread-15913584::DEBUG::2013-11-20 03:08:35,607::task::495::TaskManager.Task::(_debug) Task a2d5174c-8702-4910-aebb-8b7c7748bea2: moving from state preparing -> state finished >Thread-15913584::DEBUG::2013-11-20 03:08:35,608::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913584::DEBUG::2013-11-20 03:08:35,608::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913584::DEBUG::2013-11-20 03:08:35,608::task::495::TaskManager.Task::(_debug) Task a2d5174c-8702-4910-aebb-8b7c7748bea2: ref 0 aborting False >Thread-15913584::INFO::2013-11-20 03:08:35,615::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000328779220581', 'lastCheck': 1384916909.396462, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000893115997314', 'lastCheck': 1384916906.6511099, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000971794128418', 'lastCheck': 1384916907.982836, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000352144241333', 'lastCheck': 1384916907.957339, 'valid': True, 'code': 0}} >Thread-15913585::DEBUG::2013-11-20 03:08:35,653::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913585::DEBUG::2013-11-20 03:08:35,653::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913585::DEBUG::2013-11-20 03:08:35,654::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913585::DEBUG::2013-11-20 03:08:35,654::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913585::DEBUG::2013-11-20 03:08:35,654::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913585::DEBUG::2013-11-20 03:08:35,655::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913585::DEBUG::2013-11-20 03:08:35,655::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913585::DEBUG::2013-11-20 03:08:35,655::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Dummy-15912651::DEBUG::2013-11-20 03:08:37,592::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:08:38,020::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0500032 s, 20.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:08:40,030::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:08:41,355::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0471549 s, 21.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:08:43,364::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:08:44,038::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0463608 s, 22.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:08:46,047::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913519::DEBUG::2013-11-20 03:08:46,265::guestIF::97::vm.Vm::(_handleMessage) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::u'BRD-KLEE\\administrator' >Thread-15913589::DEBUG::2013-11-20 03:08:46,274::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913589::INFO::2013-11-20 03:08:46,277::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913589::DEBUG::2013-11-20 03:08:46,277::task::495::TaskManager.Task::(_debug) Task 9f461470-9635-48c8-89c0-554690e44d98: moving from state init -> state preparing >Thread-15913589::DEBUG::2013-11-20 03:08:46,279::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913589::DEBUG::2013-11-20 03:08:46,279::task::495::TaskManager.Task::(_debug) Task 9f461470-9635-48c8-89c0-554690e44d98: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913589::DEBUG::2013-11-20 03:08:46,280::task::495::TaskManager.Task::(_debug) Task 9f461470-9635-48c8-89c0-554690e44d98: moving from state preparing -> state finished >Thread-15913589::DEBUG::2013-11-20 03:08:46,281::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913589::DEBUG::2013-11-20 03:08:46,281::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913589::DEBUG::2013-11-20 03:08:46,282::task::495::TaskManager.Task::(_debug) Task 9f461470-9635-48c8-89c0-554690e44d98: ref 0 aborting False >Thread-15913589::INFO::2013-11-20 03:08:46,283::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913590::DEBUG::2013-11-20 03:08:46,288::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913590::INFO::2013-11-20 03:08:46,289::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913590::DEBUG::2013-11-20 03:08:46,289::task::495::TaskManager.Task::(_debug) Task 48974032-9511-4474-b75f-c733a94b1e2c: moving from state init -> state preparing >Thread-15913590::DEBUG::2013-11-20 03:08:46,291::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a9b7608a-c4fa-4706-bd39-6e7521bbf976`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913590::DEBUG::2013-11-20 03:08:46,291::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913590::DEBUG::2013-11-20 03:08:46,292::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913590::DEBUG::2013-11-20 03:08:46,293::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a9b7608a-c4fa-4706-bd39-6e7521bbf976`::Granted request >Thread-15913590::DEBUG::2013-11-20 03:08:46,294::task::495::TaskManager.Task::(_debug) Task 48974032-9511-4474-b75f-c733a94b1e2c: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913590::DEBUG::2013-11-20 03:08:46,294::task::495::TaskManager.Task::(_debug) Task 48974032-9511-4474-b75f-c733a94b1e2c: ref 1 aborting False >Thread-15913590::WARNING::2013-11-20 03:08:46,297::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913590::DEBUG::2013-11-20 03:08:46,298::task::495::TaskManager.Task::(_debug) Task 48974032-9511-4474-b75f-c733a94b1e2c: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913590::DEBUG::2013-11-20 03:08:46,299::task::495::TaskManager.Task::(_debug) Task 48974032-9511-4474-b75f-c733a94b1e2c: moving from state preparing -> state finished >Thread-15913590::DEBUG::2013-11-20 03:08:46,299::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913590::DEBUG::2013-11-20 03:08:46,300::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913590::DEBUG::2013-11-20 03:08:46,300::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913590::DEBUG::2013-11-20 03:08:46,301::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913590::DEBUG::2013-11-20 03:08:46,301::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913590::DEBUG::2013-11-20 03:08:46,302::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913590::DEBUG::2013-11-20 03:08:46,303::task::495::TaskManager.Task::(_debug) Task 48974032-9511-4474-b75f-c733a94b1e2c: ref 0 aborting False >Thread-15913590::INFO::2013-11-20 03:08:46,304::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:08:46,614::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0721798 s, 14.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:08:48,629::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:08:49,186::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0725625 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:08:51,201::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913546::INFO::2013-11-20 03:08:51,421::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913592::INFO::2013-11-20 03:08:51,426::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913546::DEBUG::2013-11-20 03:08:51,429::task::495::TaskManager.Task::(_debug) Task 0fbcc2bf-1335-47f2-afe4-ea2406307ab9: moving from state init -> state preparing >Thread-15913592::DEBUG::2013-11-20 03:08:51,429::task::495::TaskManager.Task::(_debug) Task 9393ceba-1623-4ea3-8d00-fa0e4c3078c7: moving from state init -> state preparing >Thread-15913592::DEBUG::2013-11-20 03:08:51,437::task::495::TaskManager.Task::(_debug) Task 9393ceba-1623-4ea3-8d00-fa0e4c3078c7: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000375986099243', 'lastCheck': 1384916929.5376949, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00121903419495', 'lastCheck': 1384916926.772265, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0011191368103', 'lastCheck': 1384916928.0153501, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000286102294922', 'lastCheck': 1384916927.977339, 'valid': True, 'code': 0}} >Thread-15913546::DEBUG::2013-11-20 03:08:51,439::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fde9c862-8b36-4495-bb6a-18cebc008fce`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913592::DEBUG::2013-11-20 03:08:51,440::task::495::TaskManager.Task::(_debug) Task 9393ceba-1623-4ea3-8d00-fa0e4c3078c7: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:08:51,440::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913592::DEBUG::2013-11-20 03:08:51,441::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913592::DEBUG::2013-11-20 03:08:51,442::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:08:51,442::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913592::DEBUG::2013-11-20 03:08:51,443::task::495::TaskManager.Task::(_debug) Task 9393ceba-1623-4ea3-8d00-fa0e4c3078c7: ref 0 aborting False >Thread-15913546::DEBUG::2013-11-20 03:08:51,443::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fde9c862-8b36-4495-bb6a-18cebc008fce`::Granted request >Thread-15913592::INFO::2013-11-20 03:08:51,444::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000375986099243', 'lastCheck': 1384916929.5376949, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00121903419495', 'lastCheck': 1384916926.772265, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0011191368103', 'lastCheck': 1384916928.0153501, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000286102294922', 'lastCheck': 1384916927.977339, 'valid': True, 'code': 0}} >Thread-15913546::DEBUG::2013-11-20 03:08:51,445::task::495::TaskManager.Task::(_debug) Task 0fbcc2bf-1335-47f2-afe4-ea2406307ab9: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:08:51,446::task::495::TaskManager.Task::(_debug) Task 0fbcc2bf-1335-47f2-afe4-ea2406307ab9: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:08:51,447::task::495::TaskManager.Task::(_debug) Task 0fbcc2bf-1335-47f2-afe4-ea2406307ab9: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:08:51,448::task::495::TaskManager.Task::(_debug) Task 0fbcc2bf-1335-47f2-afe4-ea2406307ab9: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:08:51,449::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:08:51,459::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:08:51,460::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:08:51,461::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:08:51,462::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:08:51,462::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:08:51,463::task::495::TaskManager.Task::(_debug) Task 0fbcc2bf-1335-47f2-afe4-ea2406307ab9: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:08:51,464::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:08:51,465::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:08:51,465::task::495::TaskManager.Task::(_debug) Task 39dc4fad-91d8-4ffa-88b5-1df65ffd1b47: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:08:51,466::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fab9ee33-20fe-40c7-8728-5f4cec5e81f4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:08:51,467::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:08:51,468::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:08:51,468::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fab9ee33-20fe-40c7-8728-5f4cec5e81f4`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:08:51,469::task::495::TaskManager.Task::(_debug) Task 39dc4fad-91d8-4ffa-88b5-1df65ffd1b47: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:08:51,470::task::495::TaskManager.Task::(_debug) Task 39dc4fad-91d8-4ffa-88b5-1df65ffd1b47: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:08:51,470::task::495::TaskManager.Task::(_debug) Task 39dc4fad-91d8-4ffa-88b5-1df65ffd1b47: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:08:51,471::task::495::TaskManager.Task::(_debug) Task 39dc4fad-91d8-4ffa-88b5-1df65ffd1b47: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:08:51,471::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:08:51,472::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:08:51,473::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:08:51,474::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:08:51,474::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:08:51,475::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:08:51,475::task::495::TaskManager.Task::(_debug) Task 39dc4fad-91d8-4ffa-88b5-1df65ffd1b47: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:08:51,476::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:08:51,477::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:08:51,477::task::495::TaskManager.Task::(_debug) Task bc65a2ff-8ba7-4227-addc-cba4b01489c4: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:08:51,478::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`6c889ec0-1135-4a78-8f89-cbd0c8590a53`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:08:51,479::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:08:51,480::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:08:51,480::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`6c889ec0-1135-4a78-8f89-cbd0c8590a53`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:08:51,481::task::495::TaskManager.Task::(_debug) Task bc65a2ff-8ba7-4227-addc-cba4b01489c4: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:08:51,482::task::495::TaskManager.Task::(_debug) Task bc65a2ff-8ba7-4227-addc-cba4b01489c4: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:08:51,482::task::495::TaskManager.Task::(_debug) Task bc65a2ff-8ba7-4227-addc-cba4b01489c4: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:08:51,483::task::495::TaskManager.Task::(_debug) Task bc65a2ff-8ba7-4227-addc-cba4b01489c4: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:08:51,483::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:08:51,484::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:08:51,485::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:08:51,485::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:08:51,486::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:08:51,487::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:08:51,488::task::495::TaskManager.Task::(_debug) Task bc65a2ff-8ba7-4227-addc-cba4b01489c4: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:08:51,489::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913593::DEBUG::2013-11-20 03:08:51,490::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913546::INFO::2013-11-20 03:08:51,491::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913593::DEBUG::2013-11-20 03:08:51,492::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913546::DEBUG::2013-11-20 03:08:51,492::task::495::TaskManager.Task::(_debug) Task 0807146f-7d87-434f-bdea-3238790e5377: moving from state init -> state preparing >Thread-15913593::DEBUG::2013-11-20 03:08:51,493::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913546::DEBUG::2013-11-20 03:08:51,494::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`25989cbf-a80b-44eb-bf45-e2f24b8c4f8a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913593::DEBUG::2013-11-20 03:08:51,494::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk stats not available >Thread-15913546::DEBUG::2013-11-20 03:08:51,495::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913593::DEBUG::2013-11-20 03:08:51,495::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913546::DEBUG::2013-11-20 03:08:51,496::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913593::DEBUG::2013-11-20 03:08:51,497::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913546::DEBUG::2013-11-20 03:08:51,497::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`25989cbf-a80b-44eb-bf45-e2f24b8c4f8a`::Granted request >Thread-15913593::DEBUG::2013-11-20 03:08:51,498::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913546::DEBUG::2013-11-20 03:08:51,499::task::495::TaskManager.Task::(_debug) Task 0807146f-7d87-434f-bdea-3238790e5377: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913593::DEBUG::2013-11-20 03:08:51,499::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::Disk latency not available >Thread-15913546::DEBUG::2013-11-20 03:08:51,500::task::495::TaskManager.Task::(_debug) Task 0807146f-7d87-434f-bdea-3238790e5377: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:08:51,512::task::495::TaskManager.Task::(_debug) Task 0807146f-7d87-434f-bdea-3238790e5377: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:08:51,513::task::495::TaskManager.Task::(_debug) Task 0807146f-7d87-434f-bdea-3238790e5377: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:08:51,513::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:08:51,514::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:08:51,515::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:08:51,515::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:08:51,516::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:08:51,516::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:08:51,517::task::495::TaskManager.Task::(_debug) Task 0807146f-7d87-434f-bdea-3238790e5377: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:08:51,517::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:08:51,737::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.072096 s, 14.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:08:53,752::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:08:54,328::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0767623 s, 13.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:08:56,351::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:08:56,898::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0844411 s, 12.1 MB/s\n'; <rc> = 0 >Thread-15913595::DEBUG::2013-11-20 03:08:57,368::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913595::INFO::2013-11-20 03:08:57,369::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913595::DEBUG::2013-11-20 03:08:57,370::task::495::TaskManager.Task::(_debug) Task 9b155268-4849-4fa2-bc41-c9ca2accbcb0: moving from state init -> state preparing >Thread-15913595::DEBUG::2013-11-20 03:08:57,371::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913595::DEBUG::2013-11-20 03:08:57,371::task::495::TaskManager.Task::(_debug) Task 9b155268-4849-4fa2-bc41-c9ca2accbcb0: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913595::DEBUG::2013-11-20 03:08:57,372::task::495::TaskManager.Task::(_debug) Task 9b155268-4849-4fa2-bc41-c9ca2accbcb0: moving from state preparing -> state finished >Thread-15913595::DEBUG::2013-11-20 03:08:57,373::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913595::DEBUG::2013-11-20 03:08:57,373::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913595::DEBUG::2013-11-20 03:08:57,374::task::495::TaskManager.Task::(_debug) Task 9b155268-4849-4fa2-bc41-c9ca2accbcb0: ref 0 aborting False >Thread-15913595::INFO::2013-11-20 03:08:57,374::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913596::DEBUG::2013-11-20 03:08:57,383::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913596::INFO::2013-11-20 03:08:57,384::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913596::DEBUG::2013-11-20 03:08:57,384::task::495::TaskManager.Task::(_debug) Task 247454d8-2929-4626-bf2f-b67d38bc2023: moving from state init -> state preparing >Thread-15913596::DEBUG::2013-11-20 03:08:57,386::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`76292a06-b5cd-4c35-8316-f7854aaab0ec`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913596::DEBUG::2013-11-20 03:08:57,386::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913596::DEBUG::2013-11-20 03:08:57,387::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913596::DEBUG::2013-11-20 03:08:57,388::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`76292a06-b5cd-4c35-8316-f7854aaab0ec`::Granted request >Thread-15913596::DEBUG::2013-11-20 03:08:57,389::task::495::TaskManager.Task::(_debug) Task 247454d8-2929-4626-bf2f-b67d38bc2023: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913596::DEBUG::2013-11-20 03:08:57,389::task::495::TaskManager.Task::(_debug) Task 247454d8-2929-4626-bf2f-b67d38bc2023: ref 1 aborting False >Thread-15913596::WARNING::2013-11-20 03:08:57,392::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913596::DEBUG::2013-11-20 03:08:57,393::task::495::TaskManager.Task::(_debug) Task 247454d8-2929-4626-bf2f-b67d38bc2023: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913596::DEBUG::2013-11-20 03:08:57,394::task::495::TaskManager.Task::(_debug) Task 247454d8-2929-4626-bf2f-b67d38bc2023: moving from state preparing -> state finished >Thread-15913596::DEBUG::2013-11-20 03:08:57,395::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913596::DEBUG::2013-11-20 03:08:57,395::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913596::DEBUG::2013-11-20 03:08:57,396::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913596::DEBUG::2013-11-20 03:08:57,396::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913596::DEBUG::2013-11-20 03:08:57,397::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913596::DEBUG::2013-11-20 03:08:57,397::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913596::DEBUG::2013-11-20 03:08:57,398::task::495::TaskManager.Task::(_debug) Task 247454d8-2929-4626-bf2f-b67d38bc2023: ref 0 aborting False >Thread-15913596::INFO::2013-11-20 03:08:57,399::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:08:58,919::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:08:59,442::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0749616 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:09:01,458::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:09:02,013::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0854365 s, 12.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:09:04,029::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:09:04,564::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.075179 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:09:06,579::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:09:07,076::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0806673 s, 12.7 MB/s\n'; <rc> = 0 >Thread-15913600::INFO::2013-11-20 03:09:07,294::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913600::DEBUG::2013-11-20 03:09:07,295::task::495::TaskManager.Task::(_debug) Task 355e4cc7-0860-4089-b0a6-b390a78c561d: moving from state init -> state preparing >Thread-15913600::DEBUG::2013-11-20 03:09:07,296::task::495::TaskManager.Task::(_debug) Task 355e4cc7-0860-4089-b0a6-b390a78c561d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000380039215088', 'lastCheck': 1384916939.5629361, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148892402649', 'lastCheck': 1384916947.2746351, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.027379989624', 'lastCheck': 1384916938.0571539, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000699996948242', 'lastCheck': 1384916937.9867001, 'valid': True, 'code': 0}} >Thread-15913600::DEBUG::2013-11-20 03:09:07,296::task::495::TaskManager.Task::(_debug) Task 355e4cc7-0860-4089-b0a6-b390a78c561d: moving from state preparing -> state finished >Thread-15913600::DEBUG::2013-11-20 03:09:07,297::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913600::DEBUG::2013-11-20 03:09:07,298::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913600::DEBUG::2013-11-20 03:09:07,298::task::495::TaskManager.Task::(_debug) Task 355e4cc7-0860-4089-b0a6-b390a78c561d: ref 0 aborting False >Thread-15913600::INFO::2013-11-20 03:09:07,299::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000380039215088', 'lastCheck': 1384916939.5629361, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00148892402649', 'lastCheck': 1384916947.2746351, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.027379989624', 'lastCheck': 1384916938.0571539, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000699996948242', 'lastCheck': 1384916937.9867001, 'valid': True, 'code': 0}} >Thread-15913602::DEBUG::2013-11-20 03:09:08,188::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913602::INFO::2013-11-20 03:09:08,189::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913602::DEBUG::2013-11-20 03:09:08,198::task::495::TaskManager.Task::(_debug) Task 6e30ef06-b303-4293-8c4f-a2b164c2dbf4: moving from state init -> state preparing >Thread-15913602::DEBUG::2013-11-20 03:09:08,199::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913602::DEBUG::2013-11-20 03:09:08,199::task::495::TaskManager.Task::(_debug) Task 6e30ef06-b303-4293-8c4f-a2b164c2dbf4: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913602::DEBUG::2013-11-20 03:09:08,200::task::495::TaskManager.Task::(_debug) Task 6e30ef06-b303-4293-8c4f-a2b164c2dbf4: moving from state preparing -> state finished >Thread-15913602::DEBUG::2013-11-20 03:09:08,201::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913602::DEBUG::2013-11-20 03:09:08,201::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913602::DEBUG::2013-11-20 03:09:08,202::task::495::TaskManager.Task::(_debug) Task 6e30ef06-b303-4293-8c4f-a2b164c2dbf4: ref 0 aborting False >Thread-15913602::INFO::2013-11-20 03:09:08,203::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913603::DEBUG::2013-11-20 03:09:08,213::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913603::INFO::2013-11-20 03:09:08,214::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913603::DEBUG::2013-11-20 03:09:08,215::task::495::TaskManager.Task::(_debug) Task eb200327-f6d0-4b2e-8c7f-6657a79c7870: moving from state init -> state preparing >Thread-15913603::DEBUG::2013-11-20 03:09:08,216::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d0ade912-c5a0-4867-b301-c0a1a15d9521`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913603::DEBUG::2013-11-20 03:09:08,217::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913603::DEBUG::2013-11-20 03:09:08,218::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913603::DEBUG::2013-11-20 03:09:08,219::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d0ade912-c5a0-4867-b301-c0a1a15d9521`::Granted request >Thread-15913603::DEBUG::2013-11-20 03:09:08,220::task::495::TaskManager.Task::(_debug) Task eb200327-f6d0-4b2e-8c7f-6657a79c7870: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913603::DEBUG::2013-11-20 03:09:08,221::task::495::TaskManager.Task::(_debug) Task eb200327-f6d0-4b2e-8c7f-6657a79c7870: ref 1 aborting False >Thread-15913603::WARNING::2013-11-20 03:09:08,223::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913603::DEBUG::2013-11-20 03:09:08,224::task::495::TaskManager.Task::(_debug) Task eb200327-f6d0-4b2e-8c7f-6657a79c7870: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913603::DEBUG::2013-11-20 03:09:08,226::task::495::TaskManager.Task::(_debug) Task eb200327-f6d0-4b2e-8c7f-6657a79c7870: moving from state preparing -> state finished >Thread-15913603::DEBUG::2013-11-20 03:09:08,226::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913603::DEBUG::2013-11-20 03:09:08,227::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913603::DEBUG::2013-11-20 03:09:08,228::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913603::DEBUG::2013-11-20 03:09:08,229::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913603::DEBUG::2013-11-20 03:09:08,229::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913603::DEBUG::2013-11-20 03:09:08,230::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913603::DEBUG::2013-11-20 03:09:08,231::task::495::TaskManager.Task::(_debug) Task eb200327-f6d0-4b2e-8c7f-6657a79c7870: ref 0 aborting False >Thread-15913603::INFO::2013-11-20 03:09:08,232::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-42840::DEBUG::2013-11-20 03:09:08,268::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-15913547::DEBUG::2013-11-20 03:09:08,752::guestIF::97::vm.Vm::(_handleMessage) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::u'None' >Dummy-15912651::DEBUG::2013-11-20 03:09:09,093::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-42840::DEBUG::2013-11-20 03:09:09,359::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:09:09,712::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0744745 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:09:11,726::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:09:12,287::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0739559 s, 13.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:09:14,302::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:09:14,865::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0766823 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:09:16,882::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:09:17,476::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0833164 s, 12.3 MB/s\n'; <rc> = 0 >Thread-15913607::DEBUG::2013-11-20 03:09:18,611::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913607::INFO::2013-11-20 03:09:18,612::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913607::DEBUG::2013-11-20 03:09:18,613::task::495::TaskManager.Task::(_debug) Task df625182-bc9c-4ad1-9dbc-59bf5b6d3de8: moving from state init -> state preparing >Thread-15913607::DEBUG::2013-11-20 03:09:18,614::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913607::DEBUG::2013-11-20 03:09:18,614::task::495::TaskManager.Task::(_debug) Task df625182-bc9c-4ad1-9dbc-59bf5b6d3de8: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913607::DEBUG::2013-11-20 03:09:18,614::task::495::TaskManager.Task::(_debug) Task df625182-bc9c-4ad1-9dbc-59bf5b6d3de8: moving from state preparing -> state finished >Thread-15913607::DEBUG::2013-11-20 03:09:18,615::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913607::DEBUG::2013-11-20 03:09:18,615::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913607::DEBUG::2013-11-20 03:09:18,615::task::495::TaskManager.Task::(_debug) Task df625182-bc9c-4ad1-9dbc-59bf5b6d3de8: ref 0 aborting False >Thread-15913607::INFO::2013-11-20 03:09:18,616::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913608::DEBUG::2013-11-20 03:09:18,668::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913608::INFO::2013-11-20 03:09:18,669::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913608::DEBUG::2013-11-20 03:09:18,670::task::495::TaskManager.Task::(_debug) Task 4a4faef5-2a5e-45f5-bafd-768940d4667a: moving from state init -> state preparing >Thread-15913608::DEBUG::2013-11-20 03:09:18,671::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`99c75046-c00d-49d8-842c-a61af16be77e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913608::DEBUG::2013-11-20 03:09:18,672::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913608::DEBUG::2013-11-20 03:09:18,672::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913608::DEBUG::2013-11-20 03:09:18,673::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`99c75046-c00d-49d8-842c-a61af16be77e`::Granted request >Thread-15913608::DEBUG::2013-11-20 03:09:18,674::task::495::TaskManager.Task::(_debug) Task 4a4faef5-2a5e-45f5-bafd-768940d4667a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913608::DEBUG::2013-11-20 03:09:18,674::task::495::TaskManager.Task::(_debug) Task 4a4faef5-2a5e-45f5-bafd-768940d4667a: ref 1 aborting False >Thread-15913608::WARNING::2013-11-20 03:09:18,676::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913608::DEBUG::2013-11-20 03:09:18,677::task::495::TaskManager.Task::(_debug) Task 4a4faef5-2a5e-45f5-bafd-768940d4667a: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913608::DEBUG::2013-11-20 03:09:18,678::task::495::TaskManager.Task::(_debug) Task 4a4faef5-2a5e-45f5-bafd-768940d4667a: moving from state preparing -> state finished >Thread-15913608::DEBUG::2013-11-20 03:09:18,679::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913608::DEBUG::2013-11-20 03:09:18,679::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913608::DEBUG::2013-11-20 03:09:18,680::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913608::DEBUG::2013-11-20 03:09:18,680::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913608::DEBUG::2013-11-20 03:09:18,681::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913608::DEBUG::2013-11-20 03:09:18,681::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913608::DEBUG::2013-11-20 03:09:18,682::task::495::TaskManager.Task::(_debug) Task 4a4faef5-2a5e-45f5-bafd-768940d4667a: ref 0 aborting False >Thread-15913608::INFO::2013-11-20 03:09:18,682::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:09:19,491::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:09:20,001::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0813159 s, 12.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:09:22,016::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:09:22,585::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0725957 s, 14.1 MB/s\n'; <rc> = 0 >Thread-15913610::INFO::2013-11-20 03:09:23,283::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913610::DEBUG::2013-11-20 03:09:23,284::task::495::TaskManager.Task::(_debug) Task c92eb1e8-e342-42bb-a515-87b0fb0b8335: moving from state init -> state preparing >Thread-15913610::DEBUG::2013-11-20 03:09:23,285::task::495::TaskManager.Task::(_debug) Task c92eb1e8-e342-42bb-a515-87b0fb0b8335: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000266075134277', 'lastCheck': 1384916960.1370821, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0264580249786', 'lastCheck': 1384916957.5682299, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.001384973526', 'lastCheck': 1384916958.5433421, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00109219551086', 'lastCheck': 1384916959.6700101, 'valid': True, 'code': 0}} >Thread-15913610::DEBUG::2013-11-20 03:09:23,292::task::495::TaskManager.Task::(_debug) Task c92eb1e8-e342-42bb-a515-87b0fb0b8335: moving from state preparing -> state finished >Thread-15913610::DEBUG::2013-11-20 03:09:23,292::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913610::DEBUG::2013-11-20 03:09:23,293::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913610::DEBUG::2013-11-20 03:09:23,294::task::495::TaskManager.Task::(_debug) Task c92eb1e8-e342-42bb-a515-87b0fb0b8335: ref 0 aborting False >Thread-15913610::INFO::2013-11-20 03:09:23,295::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000266075134277', 'lastCheck': 1384916960.1370821, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0264580249786', 'lastCheck': 1384916957.5682299, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.001384973526', 'lastCheck': 1384916958.5433421, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00109219551086', 'lastCheck': 1384916959.6700101, 'valid': True, 'code': 0}} >Thread-15913518::INFO::2013-11-20 03:09:23,718::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:09:23,719::task::495::TaskManager.Task::(_debug) Task 4e02bd12-41cb-41fa-b3dd-34d409daca6a: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:09:23,721::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`593cff0f-f8bf-4dff-8b09-2db196dd2f2d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:09:23,725::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:09:23,726::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:09:23,727::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`593cff0f-f8bf-4dff-8b09-2db196dd2f2d`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:09:23,728::task::495::TaskManager.Task::(_debug) Task 4e02bd12-41cb-41fa-b3dd-34d409daca6a: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:09:23,728::task::495::TaskManager.Task::(_debug) Task 4e02bd12-41cb-41fa-b3dd-34d409daca6a: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:09:23,729::task::495::TaskManager.Task::(_debug) Task 4e02bd12-41cb-41fa-b3dd-34d409daca6a: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:09:23,730::task::495::TaskManager.Task::(_debug) Task 4e02bd12-41cb-41fa-b3dd-34d409daca6a: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:09:23,730::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:09:23,731::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:09:23,731::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:09:23,732::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:09:23,733::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:09:23,733::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:09:23,734::task::495::TaskManager.Task::(_debug) Task 4e02bd12-41cb-41fa-b3dd-34d409daca6a: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:09:23,734::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:09:23,735::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:09:23,736::task::495::TaskManager.Task::(_debug) Task c5b964fc-3716-4bee-bbf4-d184ab058d23: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:09:23,737::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`99c4b2ed-20c2-47a7-81a8-fe195c36361d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:09:23,742::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:09:23,743::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:09:23,743::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`99c4b2ed-20c2-47a7-81a8-fe195c36361d`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:09:23,744::task::495::TaskManager.Task::(_debug) Task c5b964fc-3716-4bee-bbf4-d184ab058d23: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:09:23,745::task::495::TaskManager.Task::(_debug) Task c5b964fc-3716-4bee-bbf4-d184ab058d23: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:09:23,745::task::495::TaskManager.Task::(_debug) Task c5b964fc-3716-4bee-bbf4-d184ab058d23: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:09:23,746::task::495::TaskManager.Task::(_debug) Task c5b964fc-3716-4bee-bbf4-d184ab058d23: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:09:23,746::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:09:23,747::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:09:23,748::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:09:23,748::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:09:23,749::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:09:23,749::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:09:23,750::task::495::TaskManager.Task::(_debug) Task c5b964fc-3716-4bee-bbf4-d184ab058d23: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:09:23,750::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:09:24,600::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:09:25,068::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.050352 s, 20.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:09:27,082::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:09:27,641::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.081448 s, 12.6 MB/s\n'; <rc> = 0 >Thread-15913613::DEBUG::2013-11-20 03:09:29,093::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913613::INFO::2013-11-20 03:09:29,095::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913613::DEBUG::2013-11-20 03:09:29,096::task::495::TaskManager.Task::(_debug) Task 24337010-50aa-41d2-9344-69afc23f94d9: moving from state init -> state preparing >Thread-15913613::DEBUG::2013-11-20 03:09:29,097::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913613::DEBUG::2013-11-20 03:09:29,097::task::495::TaskManager.Task::(_debug) Task 24337010-50aa-41d2-9344-69afc23f94d9: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913613::DEBUG::2013-11-20 03:09:29,098::task::495::TaskManager.Task::(_debug) Task 24337010-50aa-41d2-9344-69afc23f94d9: moving from state preparing -> state finished >Thread-15913613::DEBUG::2013-11-20 03:09:29,098::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913613::DEBUG::2013-11-20 03:09:29,098::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913613::DEBUG::2013-11-20 03:09:29,099::task::495::TaskManager.Task::(_debug) Task 24337010-50aa-41d2-9344-69afc23f94d9: ref 0 aborting False >Thread-15913613::INFO::2013-11-20 03:09:29,099::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913614::DEBUG::2013-11-20 03:09:29,103::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913614::INFO::2013-11-20 03:09:29,103::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913614::DEBUG::2013-11-20 03:09:29,104::task::495::TaskManager.Task::(_debug) Task 2dfc18ca-3873-4682-97c4-6a8d03daf74e: moving from state init -> state preparing >Thread-15913614::DEBUG::2013-11-20 03:09:29,105::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`18047469-5c3f-4ea9-b61b-4fa4b9dc72e5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913614::DEBUG::2013-11-20 03:09:29,105::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913614::DEBUG::2013-11-20 03:09:29,106::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913614::DEBUG::2013-11-20 03:09:29,106::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`18047469-5c3f-4ea9-b61b-4fa4b9dc72e5`::Granted request >Thread-15913614::DEBUG::2013-11-20 03:09:29,111::task::495::TaskManager.Task::(_debug) Task 2dfc18ca-3873-4682-97c4-6a8d03daf74e: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913614::DEBUG::2013-11-20 03:09:29,111::task::495::TaskManager.Task::(_debug) Task 2dfc18ca-3873-4682-97c4-6a8d03daf74e: ref 1 aborting False >Thread-15913614::WARNING::2013-11-20 03:09:29,113::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913614::DEBUG::2013-11-20 03:09:29,113::task::495::TaskManager.Task::(_debug) Task 2dfc18ca-3873-4682-97c4-6a8d03daf74e: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913614::DEBUG::2013-11-20 03:09:29,114::task::495::TaskManager.Task::(_debug) Task 2dfc18ca-3873-4682-97c4-6a8d03daf74e: moving from state preparing -> state finished >Thread-15913614::DEBUG::2013-11-20 03:09:29,114::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913614::DEBUG::2013-11-20 03:09:29,115::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913614::DEBUG::2013-11-20 03:09:29,115::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913614::DEBUG::2013-11-20 03:09:29,115::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913614::DEBUG::2013-11-20 03:09:29,116::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913614::DEBUG::2013-11-20 03:09:29,116::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913614::DEBUG::2013-11-20 03:09:29,117::task::495::TaskManager.Task::(_debug) Task 2dfc18ca-3873-4682-97c4-6a8d03daf74e: ref 0 aborting False >Thread-15913614::INFO::2013-11-20 03:09:29,117::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:09:29,655::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 03:09:29,884::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >libvirtEventLoop::DEBUG::2013-11-20 03:09:29,886::vm::525::vm.Vm::(_rtcUpdate) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::new rtc offset 1 >Thread-15912392::DEBUG::2013-11-20 03:09:29,887::task::495::TaskManager.Task::(_debug) Task a3759285-1977-4426-a836-00ba6767410e: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:09:29,890::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`783af11b-6537-48b0-abc7-9458fc2c6acc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >libvirtEventLoop::DEBUG::2013-11-20 03:09:29,891::vm::525::vm.Vm::(_rtcUpdate) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::new rtc offset 1 >Thread-15912392::DEBUG::2013-11-20 03:09:29,891::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >libvirtEventLoop::DEBUG::2013-11-20 03:09:29,892::vm::525::vm.Vm::(_rtcUpdate) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::new rtc offset 3601 >Thread-15912392::DEBUG::2013-11-20 03:09:29,894::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >libvirtEventLoop::DEBUG::2013-11-20 03:09:29,895::vm::525::vm.Vm::(_rtcUpdate) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::new rtc offset 3601 >Thread-15912392::DEBUG::2013-11-20 03:09:29,895::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`783af11b-6537-48b0-abc7-9458fc2c6acc`::Granted request >libvirtEventLoop::DEBUG::2013-11-20 03:09:29,896::vm::525::vm.Vm::(_rtcUpdate) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::new rtc offset 3601 >Thread-15912392::DEBUG::2013-11-20 03:09:29,898::task::495::TaskManager.Task::(_debug) Task a3759285-1977-4426-a836-00ba6767410e: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:09:29,899::task::495::TaskManager.Task::(_debug) Task a3759285-1977-4426-a836-00ba6767410e: ref 1 aborting False >libvirtEventLoop::DEBUG::2013-11-20 03:09:29,899::vm::525::vm.Vm::(_rtcUpdate) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::new rtc offset 3601 >Thread-15912392::DEBUG::2013-11-20 03:09:29,900::task::495::TaskManager.Task::(_debug) Task a3759285-1977-4426-a836-00ba6767410e: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:09:29,901::task::495::TaskManager.Task::(_debug) Task a3759285-1977-4426-a836-00ba6767410e: moving from state preparing -> state finished >libvirtEventLoop::DEBUG::2013-11-20 03:09:29,901::vm::525::vm.Vm::(_rtcUpdate) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::new rtc offset 3601 >Thread-15912392::DEBUG::2013-11-20 03:09:29,902::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >libvirtEventLoop::DEBUG::2013-11-20 03:09:29,903::vm::525::vm.Vm::(_rtcUpdate) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::new rtc offset 3601 >Thread-15912392::DEBUG::2013-11-20 03:09:29,904::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >libvirtEventLoop::DEBUG::2013-11-20 03:09:29,905::vm::525::vm.Vm::(_rtcUpdate) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::new rtc offset 3601 >Thread-15912392::DEBUG::2013-11-20 03:09:29,906::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >libvirtEventLoop::DEBUG::2013-11-20 03:09:29,906::vm::525::vm.Vm::(_rtcUpdate) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::new rtc offset 3601 >Thread-15912392::DEBUG::2013-11-20 03:09:29,907::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:09:29,908::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >libvirtEventLoop::DEBUG::2013-11-20 03:09:29,908::vm::525::vm.Vm::(_rtcUpdate) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::new rtc offset 3601 >Thread-15912392::DEBUG::2013-11-20 03:09:29,909::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >libvirtEventLoop::DEBUG::2013-11-20 03:09:29,910::vm::525::vm.Vm::(_rtcUpdate) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::new rtc offset 3601 >Thread-15912392::DEBUG::2013-11-20 03:09:29,910::task::495::TaskManager.Task::(_debug) Task a3759285-1977-4426-a836-00ba6767410e: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:09:29,911::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >libvirtEventLoop::DEBUG::2013-11-20 03:09:29,912::vm::525::vm.Vm::(_rtcUpdate) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::new rtc offset 3601 >libvirtEventLoop::DEBUG::2013-11-20 03:09:29,913::vm::525::vm.Vm::(_rtcUpdate) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::new rtc offset 3601 >Dummy-15912651::DEBUG::2013-11-20 03:09:30,216::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.072386 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:09:32,231::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:09:32,765::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0723186 s, 14.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:09:34,781::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913547::DEBUG::2013-11-20 03:09:35,188::guestIF::97::vm.Vm::(_handleMessage) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::u'BRD-KLEE\\administrator' >Dummy-15912651::DEBUG::2013-11-20 03:09:35,324::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0708343 s, 14.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:09:37,339::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:09:37,865::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0944137 s, 10.8 MB/s\n'; <rc> = 0 >Thread-15913618::INFO::2013-11-20 03:09:39,627::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913618::DEBUG::2013-11-20 03:09:39,628::task::495::TaskManager.Task::(_debug) Task 53aeb014-4def-4611-9b2e-c8f10720dd2d: moving from state init -> state preparing >Thread-15913618::DEBUG::2013-11-20 03:09:39,629::task::495::TaskManager.Task::(_debug) Task 53aeb014-4def-4611-9b2e-c8f10720dd2d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000306844711304', 'lastCheck': 1384916970.3152349, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00189018249512', 'lastCheck': 1384916978.0192449, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00112295150757', 'lastCheck': 1384916979.0197389, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00469899177551', 'lastCheck': 1384916969.885715, 'valid': True, 'code': 0}} >Thread-15913618::DEBUG::2013-11-20 03:09:39,630::task::495::TaskManager.Task::(_debug) Task 53aeb014-4def-4611-9b2e-c8f10720dd2d: moving from state preparing -> state finished >Thread-15913618::DEBUG::2013-11-20 03:09:39,637::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913618::DEBUG::2013-11-20 03:09:39,638::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913618::DEBUG::2013-11-20 03:09:39,639::task::495::TaskManager.Task::(_debug) Task 53aeb014-4def-4611-9b2e-c8f10720dd2d: ref 0 aborting False >Thread-15913618::INFO::2013-11-20 03:09:39,640::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000306844711304', 'lastCheck': 1384916970.3152349, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00189018249512', 'lastCheck': 1384916978.0192449, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00112295150757', 'lastCheck': 1384916979.0197389, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00469899177551', 'lastCheck': 1384916969.885715, 'valid': True, 'code': 0}} >Thread-15913619::DEBUG::2013-11-20 03:09:39,648::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913619::INFO::2013-11-20 03:09:39,650::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913619::DEBUG::2013-11-20 03:09:39,655::task::495::TaskManager.Task::(_debug) Task a1ab11c0-4c3f-4903-89c1-56cd429d70a7: moving from state init -> state preparing >Thread-15913619::DEBUG::2013-11-20 03:09:39,656::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913619::DEBUG::2013-11-20 03:09:39,657::task::495::TaskManager.Task::(_debug) Task a1ab11c0-4c3f-4903-89c1-56cd429d70a7: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913619::DEBUG::2013-11-20 03:09:39,658::task::495::TaskManager.Task::(_debug) Task a1ab11c0-4c3f-4903-89c1-56cd429d70a7: moving from state preparing -> state finished >Thread-15913619::DEBUG::2013-11-20 03:09:39,663::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913619::DEBUG::2013-11-20 03:09:39,663::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913619::DEBUG::2013-11-20 03:09:39,664::task::495::TaskManager.Task::(_debug) Task a1ab11c0-4c3f-4903-89c1-56cd429d70a7: ref 0 aborting False >Thread-15913619::INFO::2013-11-20 03:09:39,665::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913620::DEBUG::2013-11-20 03:09:39,671::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913620::INFO::2013-11-20 03:09:39,672::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913620::DEBUG::2013-11-20 03:09:39,673::task::495::TaskManager.Task::(_debug) Task d095dd83-fc2a-4f24-8656-dd81ffdaa709: moving from state init -> state preparing >Thread-15913620::DEBUG::2013-11-20 03:09:39,676::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`33ffd04e-d14c-4b27-b67e-15d05dbf2128`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913620::DEBUG::2013-11-20 03:09:39,690::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913620::DEBUG::2013-11-20 03:09:39,691::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913620::DEBUG::2013-11-20 03:09:39,692::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`33ffd04e-d14c-4b27-b67e-15d05dbf2128`::Granted request >Thread-15913620::DEBUG::2013-11-20 03:09:39,694::task::495::TaskManager.Task::(_debug) Task d095dd83-fc2a-4f24-8656-dd81ffdaa709: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913620::DEBUG::2013-11-20 03:09:39,694::task::495::TaskManager.Task::(_debug) Task d095dd83-fc2a-4f24-8656-dd81ffdaa709: ref 1 aborting False >Thread-15913620::WARNING::2013-11-20 03:09:39,696::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913620::DEBUG::2013-11-20 03:09:39,697::task::495::TaskManager.Task::(_debug) Task d095dd83-fc2a-4f24-8656-dd81ffdaa709: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913620::DEBUG::2013-11-20 03:09:39,698::task::495::TaskManager.Task::(_debug) Task d095dd83-fc2a-4f24-8656-dd81ffdaa709: moving from state preparing -> state finished >Thread-15913620::DEBUG::2013-11-20 03:09:39,699::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913620::DEBUG::2013-11-20 03:09:39,699::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913620::DEBUG::2013-11-20 03:09:39,700::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913620::DEBUG::2013-11-20 03:09:39,701::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913620::DEBUG::2013-11-20 03:09:39,701::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913620::DEBUG::2013-11-20 03:09:39,702::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913620::DEBUG::2013-11-20 03:09:39,703::task::495::TaskManager.Task::(_debug) Task d095dd83-fc2a-4f24-8656-dd81ffdaa709: ref 0 aborting False >Thread-15913620::INFO::2013-11-20 03:09:39,703::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:09:39,879::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:09:40,463::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0765378 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:09:42,479::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:09:43,132::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0939918 s, 10.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:09:45,146::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:09:45,764::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0875845 s, 11.7 MB/s\n'; <rc> = 0 >Thread-15913624::DEBUG::2013-11-20 03:09:47,006::clientIF::54::vds::(wrapper) [192.168.100.135]::call create with ({'bridge': 'rhevm', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.2.0', 'vmId': '4760db3e-d3a8-4045-9153-12c36ec16ba2', 'transparentHugePages': 'true', 'spiceSslCipherSuite': 'DEFAULT', 'cpuType': 'Westmere', 'smp': '4', 'macAddr': '00:1a:4a:a8:64:04', 'boot': 'cd', 'custom': {}, 'vmType': 'kvm', 'memSize': 4096, 'smpCoresPerSocket': '2', 'vmName': 'KLEE-AOS01', 'spiceMonitors': '1', 'nice': '0', 'drives': [{'index': '0', 'domainID': '6b892067-ac7e-4039-a7be-5b7387525b16', 'format': 'raw', 'boot': 'true', 'volumeID': '5eaa8dde-10d8-45b5-9e10-5417c297c4be', 'imageID': '6f97dfa5-ee35-4f37-8495-1e6b3136b92c', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}], 'smartcardEnable': 'false', 'nicModel': 'pv', 'keyboardLayout': 'en-us', 'kvmEnable': 'true', 'timeOffset': '-1', 'spiceSecureChannels': 'smain,sinputs', 'display': 'qxl'},) {} >Thread-15913624::INFO::2013-11-20 03:09:47,413::clientIF::523::vds::(_getNetworkIp) network None: using 0 >Thread-15913624::INFO::2013-11-20 03:09:47,415::clientIF::656::vds::(create) vmContainerLock acquired by vm 4760db3e-d3a8-4045-9153-12c36ec16ba2 >Thread-15913625::DEBUG::2013-11-20 03:09:47,420::vm::342::vm.Vm::(_startUnderlyingVm) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Start >Thread-15913624::DEBUG::2013-11-20 03:09:47,421::clientIF::669::vds::(create) Total desktops after creation of 4760db3e-d3a8-4045-9153-12c36ec16ba2 is 4 >Thread-15913625::DEBUG::2013-11-20 03:09:47,422::vm::346::vm.Vm::(_startUnderlyingVm) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::_ongoingCreations acquired >Thread-15913624::DEBUG::2013-11-20 03:09:47,423::clientIF::59::vds::(wrapper) return create with {'status': {'message': 'Done', 'code': 0}, 'vmList': {'status': 'WaitForLaunch', 'bridge': 'rhevm', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.2.0', 'vmId': '4760db3e-d3a8-4045-9153-12c36ec16ba2', 'pid': '0', 'drives': [{'index': '0', 'domainID': '6b892067-ac7e-4039-a7be-5b7387525b16', 'format': 'raw', 'boot': 'true', 'volumeID': '5eaa8dde-10d8-45b5-9e10-5417c297c4be', 'imageID': '6f97dfa5-ee35-4f37-8495-1e6b3136b92c', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}], 'transparentHugePages': 'true', 'keyboardLayout': 'en-us', 'displayPort': '-1', 'displaySecurePort': '-1', 'spiceSslCipherSuite': 'DEFAULT', 'cpuType': 'Westmere', 'custom': {}, 'clientIp': '', 'nicModel': 'pv', 'macAddr': '00:1a:4a:a8:64:04', 'kvmEnable': 'true', 'boot': 'cd', 'smp': '4', 'vmType': 'kvm', 'smartcardEnable': 'false', 'timeOffset': '-1', 'memSize': 4096, 'displayIp': '0', 'spiceSecureChannels': 'smain,sinputs', 'smpCoresPerSocket': '2', 'vmName': 'KLEE-AOS01', 'display': 'qxl', 'spiceMonitors': '1', 'nice': '0'}} >Thread-15913625::INFO::2013-11-20 03:09:47,423::libvirtvm::1061::vm.Vm::(_run) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::VM wrapper has started >Thread-15913625::INFO::2013-11-20 03:09:47,428::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913625::DEBUG::2013-11-20 03:09:47,429::task::495::TaskManager.Task::(_debug) Task 7482bed7-f7a2-4f94-bfe4-19f216683321: moving from state init -> state preparing >Thread-15913625::DEBUG::2013-11-20 03:09:47,431::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c28c64be-0174-4eb4-998f-d7dced5d5899`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1690' at 'public_prepareVolume' >Thread-15913625::DEBUG::2013-11-20 03:09:47,432::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913625::DEBUG::2013-11-20 03:09:47,433::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913625::DEBUG::2013-11-20 03:09:47,433::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c28c64be-0174-4eb4-998f-d7dced5d5899`::Granted request >Thread-15913625::DEBUG::2013-11-20 03:09:47,434::task::495::TaskManager.Task::(_debug) Task 7482bed7-f7a2-4f94-bfe4-19f216683321: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913625::DEBUG::2013-11-20 03:09:47,435::task::495::TaskManager.Task::(_debug) Task 7482bed7-f7a2-4f94-bfe4-19f216683321: ref 1 aborting False >Thread-15913625::DEBUG::2013-11-20 03:09:47,436::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.6f97dfa5-ee35-4f37-8495-1e6b3136b92c`ReqID=`ead00eb7-045d-4261-80e9-e67860305216`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1695' at 'public_prepareVolume' >Thread-15913625::DEBUG::2013-11-20 03:09:47,437::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.6f97dfa5-ee35-4f37-8495-1e6b3136b92c' for lock type 'exclusive' >Thread-15913625::DEBUG::2013-11-20 03:09:47,439::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913625::DEBUG::2013-11-20 03:09:47,440::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:09:47,780::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913547::DEBUG::2013-11-20 03:09:48,034::guestIF::97::vm.Vm::(_handleMessage) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::u'None' >Thread-15913625::DEBUG::2013-11-20 03:09:48,423::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:09:48,521::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0763715 s, 13.4 MB/s\n'; <rc> = 0 >Thread-15913625::DEBUG::2013-11-20 03:09:48,758::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913625::DEBUG::2013-11-20 03:09:48,766::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=9 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15913625::DEBUG::2013-11-20 03:09:49,198::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000230876 s, 2.2 MB/s\n'; <rc> = 0 >Thread-15913625::DEBUG::2013-11-20 03:09:49,200::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000230876 s, 2.2 MB/s'], size: 512 >Thread-15913625::DEBUG::2013-11-20 03:09:49,216::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=9 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15913625::DEBUG::2013-11-20 03:09:49,632::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000243174 s, 2.1 MB/s\n'; <rc> = 0 >Thread-15913625::DEBUG::2013-11-20 03:09:49,633::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000243174 s, 2.1 MB/s'], size: 512 >Thread-15913625::INFO::2013-11-20 03:09:49,634::image::340::Storage.Image::(getChain) sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c chain=[<storage.blockVolume.BlockVolume instance at 0x7f487c1037e8>] >Thread-15913625::DEBUG::2013-11-20 03:09:49,635::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --autobackup n --available y 6b892067-ac7e-4039-a7be-5b7387525b16/5eaa8dde-10d8-45b5-9e10-5417c297c4be' (cwd None) >Thread-15913628::DEBUG::2013-11-20 03:09:50,028::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913628::INFO::2013-11-20 03:09:50,030::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913628::DEBUG::2013-11-20 03:09:50,030::task::495::TaskManager.Task::(_debug) Task 86700ddf-a639-47de-9df5-47adaa38ecb7: moving from state init -> state preparing >Thread-15913628::DEBUG::2013-11-20 03:09:50,031::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913628::DEBUG::2013-11-20 03:09:50,031::task::495::TaskManager.Task::(_debug) Task 86700ddf-a639-47de-9df5-47adaa38ecb7: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913628::DEBUG::2013-11-20 03:09:50,032::task::495::TaskManager.Task::(_debug) Task 86700ddf-a639-47de-9df5-47adaa38ecb7: moving from state preparing -> state finished >Thread-15913628::DEBUG::2013-11-20 03:09:50,032::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913628::DEBUG::2013-11-20 03:09:50,032::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913628::DEBUG::2013-11-20 03:09:50,038::task::495::TaskManager.Task::(_debug) Task 86700ddf-a639-47de-9df5-47adaa38ecb7: ref 0 aborting False >Thread-15913628::INFO::2013-11-20 03:09:50,039::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913629::DEBUG::2013-11-20 03:09:50,042::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913629::INFO::2013-11-20 03:09:50,043::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913629::DEBUG::2013-11-20 03:09:50,043::task::495::TaskManager.Task::(_debug) Task 30c4a7d4-7a3b-4431-b497-4def5c1ee50e: moving from state init -> state preparing >Thread-15913629::DEBUG::2013-11-20 03:09:50,044::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`cb168538-a85b-4ed0-9940-4e6201e2fa50`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913629::DEBUG::2013-11-20 03:09:50,045::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913629::DEBUG::2013-11-20 03:09:50,045::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913629::DEBUG::2013-11-20 03:09:50,046::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`cb168538-a85b-4ed0-9940-4e6201e2fa50`::Granted request >Thread-15913629::DEBUG::2013-11-20 03:09:50,046::task::495::TaskManager.Task::(_debug) Task 30c4a7d4-7a3b-4431-b497-4def5c1ee50e: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913629::DEBUG::2013-11-20 03:09:50,047::task::495::TaskManager.Task::(_debug) Task 30c4a7d4-7a3b-4431-b497-4def5c1ee50e: ref 1 aborting False >Thread-15913629::WARNING::2013-11-20 03:09:50,048::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913629::DEBUG::2013-11-20 03:09:50,049::task::495::TaskManager.Task::(_debug) Task 30c4a7d4-7a3b-4431-b497-4def5c1ee50e: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913629::DEBUG::2013-11-20 03:09:50,049::task::495::TaskManager.Task::(_debug) Task 30c4a7d4-7a3b-4431-b497-4def5c1ee50e: moving from state preparing -> state finished >Thread-15913629::DEBUG::2013-11-20 03:09:50,050::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913629::DEBUG::2013-11-20 03:09:50,050::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913629::DEBUG::2013-11-20 03:09:50,051::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913629::DEBUG::2013-11-20 03:09:50,051::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913629::DEBUG::2013-11-20 03:09:50,052::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913629::DEBUG::2013-11-20 03:09:50,052::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913629::DEBUG::2013-11-20 03:09:50,053::task::495::TaskManager.Task::(_debug) Task 30c4a7d4-7a3b-4431-b497-4def5c1ee50e: ref 0 aborting False >Thread-15913629::INFO::2013-11-20 03:09:50,054::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913519::DEBUG::2013-11-20 03:09:50,497::guestIF::97::vm.Vm::(_handleMessage) vmId=`4b8ed9c8-dfb1-4de4-935d-6ba4009567b9`::u'None' >Thread-15913625::DEBUG::2013-11-20 03:09:50,591::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913625::DEBUG::2013-11-20 03:09:50,592::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913625::DEBUG::2013-11-20 03:09:50,593::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913625::DEBUG::2013-11-20 03:09:50,594::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.5eaa8dde-10d8-45b5-9e10-5417c297c4be`ReqID=`50af7a47-c340-4ae0-a99a-a7d1fb2af5cf`::Request was made in '/usr/share/vdsm/storage/resourceFactories.py' line '149' at '__getResourceCandidatesList' >Thread-15913625::DEBUG::2013-11-20 03:09:50,595::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.5eaa8dde-10d8-45b5-9e10-5417c297c4be' for lock type 'exclusive' >Thread-15913625::DEBUG::2013-11-20 03:09:50,595::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.5eaa8dde-10d8-45b5-9e10-5417c297c4be' is free. Now locking as 'exclusive' (1 active user) >Thread-15913625::DEBUG::2013-11-20 03:09:50,596::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.5eaa8dde-10d8-45b5-9e10-5417c297c4be`ReqID=`50af7a47-c340-4ae0-a99a-a7d1fb2af5cf`::Granted request >Thread-15913625::DEBUG::2013-11-20 03:09:50,603::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.6f97dfa5-ee35-4f37-8495-1e6b3136b92c' is free. Now locking as 'exclusive' (1 active user) >Thread-15913625::DEBUG::2013-11-20 03:09:50,603::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.6f97dfa5-ee35-4f37-8495-1e6b3136b92c`ReqID=`ead00eb7-045d-4261-80e9-e67860305216`::Granted request >Thread-15913625::DEBUG::2013-11-20 03:09:50,604::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913625::DEBUG::2013-11-20 03:09:50,605::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:09:50,767::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913630::DEBUG::2013-11-20 03:09:51,140::clientIF::54::vds::(wrapper) [192.168.100.135]::call create with ({'bridge': 'rhevm', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.2.0', 'vmId': '0dc7e997-e884-4493-b701-790fb37b3d80', 'transparentHugePages': 'true', 'spiceSslCipherSuite': 'DEFAULT', 'cpuType': 'Westmere', 'smp': '4', 'macAddr': '00:1a:4a:a8:64:03', 'boot': 'cd', 'custom': {}, 'vmType': 'kvm', 'memSize': 4096, 'smpCoresPerSocket': '2', 'vmName': 'KLEE-AOS02', 'spiceMonitors': '1', 'nice': '0', 'drives': [{'index': '0', 'domainID': '6b892067-ac7e-4039-a7be-5b7387525b16', 'format': 'raw', 'boot': 'true', 'volumeID': '8506c15b-61ae-4830-84ed-c830955c25a3', 'imageID': 'b74dd33b-748d-48df-83f2-ebcc3523e117', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}], 'smartcardEnable': 'false', 'nicModel': 'pv', 'keyboardLayout': 'en-us', 'kvmEnable': 'true', 'timeOffset': '3600', 'spiceSecureChannels': 'smain,sinputs', 'display': 'qxl'},) {} >Thread-15913625::DEBUG::2013-11-20 03:09:51,586::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913625::DEBUG::2013-11-20 03:09:51,902::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913630::INFO::2013-11-20 03:09:51,904::clientIF::523::vds::(_getNetworkIp) network None: using 0 >Thread-15913630::INFO::2013-11-20 03:09:51,906::clientIF::656::vds::(create) vmContainerLock acquired by vm 0dc7e997-e884-4493-b701-790fb37b3d80 >Thread-15913625::DEBUG::2013-11-20 03:09:51,907::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --refresh 6b892067-ac7e-4039-a7be-5b7387525b16/5eaa8dde-10d8-45b5-9e10-5417c297c4be' (cwd None) >Thread-15913630::DEBUG::2013-11-20 03:09:52,150::clientIF::669::vds::(create) Total desktops after creation of 0dc7e997-e884-4493-b701-790fb37b3d80 is 5 >Thread-15913631::DEBUG::2013-11-20 03:09:52,151::vm::342::vm.Vm::(_startUnderlyingVm) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Start >Thread-15913630::DEBUG::2013-11-20 03:09:52,155::clientIF::59::vds::(wrapper) return create with {'status': {'message': 'Done', 'code': 0}, 'vmList': {'status': 'WaitForLaunch', 'bridge': 'rhevm', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.2.0', 'vmId': '0dc7e997-e884-4493-b701-790fb37b3d80', 'pid': '0', 'drives': [{'index': '0', 'domainID': '6b892067-ac7e-4039-a7be-5b7387525b16', 'format': 'raw', 'boot': 'true', 'volumeID': '8506c15b-61ae-4830-84ed-c830955c25a3', 'imageID': 'b74dd33b-748d-48df-83f2-ebcc3523e117', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}], 'transparentHugePages': 'true', 'keyboardLayout': 'en-us', 'displayPort': '-1', 'displaySecurePort': '-1', 'spiceSslCipherSuite': 'DEFAULT', 'cpuType': 'Westmere', 'custom': {}, 'clientIp': '', 'nicModel': 'pv', 'macAddr': '00:1a:4a:a8:64:03', 'kvmEnable': 'true', 'boot': 'cd', 'smp': '4', 'vmType': 'kvm', 'smartcardEnable': 'false', 'timeOffset': '3600', 'memSize': 4096, 'displayIp': '0', 'spiceSecureChannels': 'smain,sinputs', 'smpCoresPerSocket': '2', 'vmName': 'KLEE-AOS02', 'display': 'qxl', 'spiceMonitors': '1', 'nice': '0'}} >Dummy-15912651::DEBUG::2013-11-20 03:09:52,308::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.77864 s, 1.3 MB/s\n'; <rc> = 0 >Thread-15913625::DEBUG::2013-11-20 03:09:52,697::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913625::DEBUG::2013-11-20 03:09:52,699::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913625::DEBUG::2013-11-20 03:09:52,699::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913625::INFO::2013-11-20 03:09:52,700::volume::517::Storage.Volume::(prepare) Volume: preparing volume 6b892067-ac7e-4039-a7be-5b7387525b16/5eaa8dde-10d8-45b5-9e10-5417c297c4be >Thread-15913625::DEBUG::2013-11-20 03:09:52,701::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913625::DEBUG::2013-11-20 03:09:52,702::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15913546::INFO::2013-11-20 03:09:52,952::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:09:52,954::task::495::TaskManager.Task::(_debug) Task 0274b38a-476b-4ef3-aa65-43536912c356: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:09:52,956::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`331ec034-5368-4093-8506-21ba2ffbd5a7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:09:52,957::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:09:52,958::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:09:52,958::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`331ec034-5368-4093-8506-21ba2ffbd5a7`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:09:52,959::task::495::TaskManager.Task::(_debug) Task 0274b38a-476b-4ef3-aa65-43536912c356: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:09:52,960::task::495::TaskManager.Task::(_debug) Task 0274b38a-476b-4ef3-aa65-43536912c356: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:09:52,961::task::495::TaskManager.Task::(_debug) Task 0274b38a-476b-4ef3-aa65-43536912c356: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:09:52,962::task::495::TaskManager.Task::(_debug) Task 0274b38a-476b-4ef3-aa65-43536912c356: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:09:52,962::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:09:52,963::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:09:52,963::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:09:52,964::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:09:52,965::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:09:52,965::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:09:52,966::task::495::TaskManager.Task::(_debug) Task 0274b38a-476b-4ef3-aa65-43536912c356: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:09:52,967::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:09:52,967::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:09:52,968::task::495::TaskManager.Task::(_debug) Task 9d42237d-b171-4295-a663-42a68fbe1204: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:09:52,969::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fb41bbba-060d-43a9-9a77-45cf93a19d34`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:09:52,970::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:09:52,970::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:09:52,971::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fb41bbba-060d-43a9-9a77-45cf93a19d34`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:09:52,972::task::495::TaskManager.Task::(_debug) Task 9d42237d-b171-4295-a663-42a68fbe1204: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:09:52,973::task::495::TaskManager.Task::(_debug) Task 9d42237d-b171-4295-a663-42a68fbe1204: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:09:52,974::task::495::TaskManager.Task::(_debug) Task 9d42237d-b171-4295-a663-42a68fbe1204: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:09:52,974::task::495::TaskManager.Task::(_debug) Task 9d42237d-b171-4295-a663-42a68fbe1204: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:09:52,975::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:09:52,975::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:09:52,976::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:09:52,977::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:09:52,978::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:09:52,978::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:09:52,979::task::495::TaskManager.Task::(_debug) Task 9d42237d-b171-4295-a663-42a68fbe1204: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:09:52,979::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:09:52,980::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:09:52,981::task::495::TaskManager.Task::(_debug) Task 0223cbdf-f913-4341-a8ad-f5659e6f9189: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:09:52,982::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`289c1371-187e-485e-a939-d1d6ae56af8f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:09:52,983::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:09:52,984::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:09:52,984::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`289c1371-187e-485e-a939-d1d6ae56af8f`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:09:52,985::task::495::TaskManager.Task::(_debug) Task 0223cbdf-f913-4341-a8ad-f5659e6f9189: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:09:52,986::task::495::TaskManager.Task::(_debug) Task 0223cbdf-f913-4341-a8ad-f5659e6f9189: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:09:52,987::task::495::TaskManager.Task::(_debug) Task 0223cbdf-f913-4341-a8ad-f5659e6f9189: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:09:52,987::task::495::TaskManager.Task::(_debug) Task 0223cbdf-f913-4341-a8ad-f5659e6f9189: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:09:52,988::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:09:52,988::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:09:52,989::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:09:52,990::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:09:52,990::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:09:52,991::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:09:52,992::task::495::TaskManager.Task::(_debug) Task 0223cbdf-f913-4341-a8ad-f5659e6f9189: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:09:52,992::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:09:52,993::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:09:52,994::task::495::TaskManager.Task::(_debug) Task a74eef09-766d-472a-989c-09596a61cdb8: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:09:52,995::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`50299f1a-cd8a-40f0-8705-9d81c24f5a37`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:09:52,995::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:09:52,996::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:09:52,996::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`50299f1a-cd8a-40f0-8705-9d81c24f5a37`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:09:52,997::task::495::TaskManager.Task::(_debug) Task a74eef09-766d-472a-989c-09596a61cdb8: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:09:52,998::task::495::TaskManager.Task::(_debug) Task a74eef09-766d-472a-989c-09596a61cdb8: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:09:52,999::task::495::TaskManager.Task::(_debug) Task a74eef09-766d-472a-989c-09596a61cdb8: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:09:52,999::task::495::TaskManager.Task::(_debug) Task a74eef09-766d-472a-989c-09596a61cdb8: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:09:53,000::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:09:53,000::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:09:53,001::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:09:53,002::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:09:53,002::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:09:53,003::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:09:53,003::task::495::TaskManager.Task::(_debug) Task a74eef09-766d-472a-989c-09596a61cdb8: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:09:53,004::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913625::DEBUG::2013-11-20 03:09:53,617::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913625::DEBUG::2013-11-20 03:09:53,903::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913625::DEBUG::2013-11-20 03:09:53,905::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=9 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15913634::DEBUG::2013-11-20 03:09:54,090::clientIF::54::vds::(wrapper) [192.168.100.135]::call create with ({'bridge': 'rhevm', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.2.0', 'vmId': '11b5c641-e4f7-487e-9758-c42cb014e2d2', 'transparentHugePages': 'true', 'spiceSslCipherSuite': 'DEFAULT', 'cpuType': 'Westmere', 'smp': '2', 'macAddr': '00:1a:4a:a8:64:0d', 'boot': 'd', 'custom': {}, 'vmType': 'kvm', 'memSize': 2048, 'smpCoresPerSocket': '1', 'vmName': 'KLEE-Print01', 'spiceMonitors': '1', 'nice': '0', 'drives': [{'index': '0', 'domainID': '6b892067-ac7e-4039-a7be-5b7387525b16', 'format': 'cow', 'boot': 'true', 'volumeID': 'ff5e3845-0d0d-4037-a0de-a3f584a4aef5', 'imageID': 'cc863eeb-33db-4227-b4eb-9f4347299cf3', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}], 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'smartcardEnable': 'false', 'nicModel': 'pv', 'keyboardLayout': 'en-us', 'kvmEnable': 'true', 'timeOffset': '-1', 'spiceSecureChannels': 'smain,sinputs', 'display': 'qxl'},) {} >Dummy-15912651::DEBUG::2013-11-20 03:09:54,354::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913625::DEBUG::2013-11-20 03:09:54,670::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000185182 s, 2.8 MB/s\n'; <rc> = 0 >Thread-15913634::INFO::2013-11-20 03:09:54,673::clientIF::523::vds::(_getNetworkIp) network None: using 0 >Thread-15913634::INFO::2013-11-20 03:09:54,674::clientIF::656::vds::(create) vmContainerLock acquired by vm 11b5c641-e4f7-487e-9758-c42cb014e2d2 >Thread-15913625::DEBUG::2013-11-20 03:09:54,675::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000185182 s, 2.8 MB/s'], size: 512 >Thread-15913625::DEBUG::2013-11-20 03:09:54,679::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=9 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15913635::DEBUG::2013-11-20 03:09:54,931::vm::342::vm.Vm::(_startUnderlyingVm) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::Start >Thread-15913634::DEBUG::2013-11-20 03:09:54,941::clientIF::669::vds::(create) Total desktops after creation of 11b5c641-e4f7-487e-9758-c42cb014e2d2 is 6 >Thread-15913634::DEBUG::2013-11-20 03:09:54,951::clientIF::59::vds::(wrapper) return create with {'status': {'message': 'Done', 'code': 0}, 'vmList': {'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'bridge': 'rhevm', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.2.0', 'vmId': '11b5c641-e4f7-487e-9758-c42cb014e2d2', 'pid': '0', 'drives': [{'index': '0', 'domainID': '6b892067-ac7e-4039-a7be-5b7387525b16', 'format': 'cow', 'boot': 'true', 'volumeID': 'ff5e3845-0d0d-4037-a0de-a3f584a4aef5', 'imageID': 'cc863eeb-33db-4227-b4eb-9f4347299cf3', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}], 'transparentHugePages': 'true', 'keyboardLayout': 'en-us', 'displaySecurePort': '-1', 'displayPort': '-1', 'cpuType': 'Westmere', 'custom': {}, 'clientIp': '', 'nicModel': 'pv', 'macAddr': '00:1a:4a:a8:64:0d', 'kvmEnable': 'true', 'smartcardEnable': 'false', 'boot': 'd', 'smp': '2', 'vmType': 'kvm', 'spiceSslCipherSuite': 'DEFAULT', 'status': 'WaitForLaunch', 'timeOffset': '-1', 'memSize': 2048, 'displayIp': '0', 'spiceSecureChannels': 'smain,sinputs', 'smpCoresPerSocket': '1', 'vmName': 'KLEE-Print01', 'display': 'qxl', 'spiceMonitors': '1', 'nice': '0'}} >Dummy-15912651::DEBUG::2013-11-20 03:09:55,068::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0982567 s, 10.4 MB/s\n'; <rc> = 0 >Thread-15913625::DEBUG::2013-11-20 03:09:55,103::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000181294 s, 2.8 MB/s\n'; <rc> = 0 >Thread-15913625::DEBUG::2013-11-20 03:09:55,105::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000181294 s, 2.8 MB/s'], size: 512 >Thread-15913625::DEBUG::2013-11-20 03:09:55,107::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.5eaa8dde-10d8-45b5-9e10-5417c297c4be`ReqID=`d6b9aa42-6636-4b0f-bdd2-5fcf16dbac6b`::Request was made in '/usr/share/vdsm/storage/volume.py' line '533' at 'prepare' >Thread-15913625::DEBUG::2013-11-20 03:09:55,107::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.5eaa8dde-10d8-45b5-9e10-5417c297c4be' for lock type 'exclusive' >Thread-15913625::DEBUG::2013-11-20 03:09:55,109::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.5eaa8dde-10d8-45b5-9e10-5417c297c4be' is free. Now locking as 'exclusive' (1 active user) >Thread-15913625::DEBUG::2013-11-20 03:09:55,109::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.5eaa8dde-10d8-45b5-9e10-5417c297c4be`ReqID=`d6b9aa42-6636-4b0f-bdd2-5fcf16dbac6b`::Granted request >Thread-15913625::DEBUG::2013-11-20 03:09:55,111::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=9 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15913637::INFO::2013-11-20 03:09:55,388::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913637::DEBUG::2013-11-20 03:09:55,390::task::495::TaskManager.Task::(_debug) Task 8e52d9a3-3fb5-48a9-bc3d-b06a3f533ae9: moving from state init -> state preparing >Thread-15913637::DEBUG::2013-11-20 03:09:55,391::task::495::TaskManager.Task::(_debug) Task 8e52d9a3-3fb5-48a9-bc3d-b06a3f533ae9: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000442028045654', 'lastCheck': 1384916991.1371789, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00122809410095', 'lastCheck': 1384916988.2263241, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00136303901672', 'lastCheck': 1384916989.43997, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000420808792114', 'lastCheck': 1384916990.4352801, 'valid': True, 'code': 0}} >Thread-15913637::DEBUG::2013-11-20 03:09:55,392::task::495::TaskManager.Task::(_debug) Task 8e52d9a3-3fb5-48a9-bc3d-b06a3f533ae9: moving from state preparing -> state finished >Thread-15913637::DEBUG::2013-11-20 03:09:55,393::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913637::DEBUG::2013-11-20 03:09:55,393::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913637::DEBUG::2013-11-20 03:09:55,394::task::495::TaskManager.Task::(_debug) Task 8e52d9a3-3fb5-48a9-bc3d-b06a3f533ae9: ref 0 aborting False >Thread-15913637::INFO::2013-11-20 03:09:55,394::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000442028045654', 'lastCheck': 1384916991.1371789, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00122809410095', 'lastCheck': 1384916988.2263241, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00136303901672', 'lastCheck': 1384916989.43997, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000420808792114', 'lastCheck': 1384916990.4352801, 'valid': True, 'code': 0}} >Thread-15913625::DEBUG::2013-11-20 03:09:55,601::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.00018082 s, 2.8 MB/s\n'; <rc> = 0 >Thread-15913625::DEBUG::2013-11-20 03:09:55,602::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.00018082 s, 2.8 MB/s'], size: 512 >Thread-15913625::DEBUG::2013-11-20 03:09:55,604::task::495::TaskManager.Task::(_debug) Task 7482bed7-f7a2-4f94-bfe4-19f216683321: finished: None >Thread-15913625::DEBUG::2013-11-20 03:09:55,604::task::495::TaskManager.Task::(_debug) Task 7482bed7-f7a2-4f94-bfe4-19f216683321: moving from state preparing -> state finished >Thread-15913625::DEBUG::2013-11-20 03:09:55,605::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913625::DEBUG::2013-11-20 03:09:55,606::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913625::DEBUG::2013-11-20 03:09:55,606::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913625::DEBUG::2013-11-20 03:09:55,607::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913625::DEBUG::2013-11-20 03:09:55,608::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913625::DEBUG::2013-11-20 03:09:55,608::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913625::DEBUG::2013-11-20 03:09:55,609::task::495::TaskManager.Task::(_debug) Task 7482bed7-f7a2-4f94-bfe4-19f216683321: ref 0 aborting False >Thread-15913625::INFO::2013-11-20 03:09:55,610::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15913625::INFO::2013-11-20 03:09:55,611::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913625::DEBUG::2013-11-20 03:09:55,611::task::495::TaskManager.Task::(_debug) Task c64c0f5e-fec3-48d9-a7dd-5efaf46e4f36: moving from state init -> state preparing >Thread-15913625::DEBUG::2013-11-20 03:09:55,613::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f3baf50a-4ecc-413b-b377-0454eb1bc246`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1666' at 'public_getVolumePath' >Thread-15913625::DEBUG::2013-11-20 03:09:55,613::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913625::DEBUG::2013-11-20 03:09:55,614::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913625::DEBUG::2013-11-20 03:09:55,615::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f3baf50a-4ecc-413b-b377-0454eb1bc246`::Granted request >Thread-15913625::DEBUG::2013-11-20 03:09:55,615::task::495::TaskManager.Task::(_debug) Task c64c0f5e-fec3-48d9-a7dd-5efaf46e4f36: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913625::DEBUG::2013-11-20 03:09:55,616::task::495::TaskManager.Task::(_debug) Task c64c0f5e-fec3-48d9-a7dd-5efaf46e4f36: ref 1 aborting False >Thread-15913625::DEBUG::2013-11-20 03:09:55,617::task::495::TaskManager.Task::(_debug) Task c64c0f5e-fec3-48d9-a7dd-5efaf46e4f36: finished: {'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/6f97dfa5-ee35-4f37-8495-1e6b3136b92c/5eaa8dde-10d8-45b5-9e10-5417c297c4be'} >Thread-15913625::DEBUG::2013-11-20 03:09:55,618::task::495::TaskManager.Task::(_debug) Task c64c0f5e-fec3-48d9-a7dd-5efaf46e4f36: moving from state preparing -> state finished >Thread-15913625::DEBUG::2013-11-20 03:09:55,618::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913625::DEBUG::2013-11-20 03:09:55,619::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913625::DEBUG::2013-11-20 03:09:55,620::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913625::DEBUG::2013-11-20 03:09:55,620::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913625::DEBUG::2013-11-20 03:09:55,621::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913625::DEBUG::2013-11-20 03:09:55,621::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913625::DEBUG::2013-11-20 03:09:55,622::task::495::TaskManager.Task::(_debug) Task c64c0f5e-fec3-48d9-a7dd-5efaf46e4f36: ref 0 aborting False >Thread-15913625::INFO::2013-11-20 03:09:55,623::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, Return response: {'status': {'message': 'OK', 'code': 0}, 'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/6f97dfa5-ee35-4f37-8495-1e6b3136b92c/5eaa8dde-10d8-45b5-9e10-5417c297c4be'} >Thread-15913625::INFO::2013-11-20 03:09:55,623::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913625::DEBUG::2013-11-20 03:09:55,624::task::495::TaskManager.Task::(_debug) Task 30e6a4a0-966d-4155-adfb-4bfef8aadde3: moving from state init -> state preparing >Thread-15913625::DEBUG::2013-11-20 03:09:55,625::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`dcb1f46d-8c60-4a6e-b371-2e9c57c3efd1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913625::DEBUG::2013-11-20 03:09:55,626::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913625::DEBUG::2013-11-20 03:09:55,626::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913625::DEBUG::2013-11-20 03:09:55,627::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`dcb1f46d-8c60-4a6e-b371-2e9c57c3efd1`::Granted request >Thread-15913625::DEBUG::2013-11-20 03:09:55,628::task::495::TaskManager.Task::(_debug) Task 30e6a4a0-966d-4155-adfb-4bfef8aadde3: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913625::DEBUG::2013-11-20 03:09:55,628::task::495::TaskManager.Task::(_debug) Task 30e6a4a0-966d-4155-adfb-4bfef8aadde3: ref 1 aborting False >Thread-15913625::DEBUG::2013-11-20 03:09:55,629::task::495::TaskManager.Task::(_debug) Task 30e6a4a0-966d-4155-adfb-4bfef8aadde3: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913625::DEBUG::2013-11-20 03:09:55,630::task::495::TaskManager.Task::(_debug) Task 30e6a4a0-966d-4155-adfb-4bfef8aadde3: moving from state preparing -> state finished >Thread-15913625::DEBUG::2013-11-20 03:09:55,630::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913625::DEBUG::2013-11-20 03:09:55,631::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913625::DEBUG::2013-11-20 03:09:55,632::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913625::DEBUG::2013-11-20 03:09:55,632::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913625::DEBUG::2013-11-20 03:09:55,633::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913625::DEBUG::2013-11-20 03:09:55,633::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913625::DEBUG::2013-11-20 03:09:55,634::task::495::TaskManager.Task::(_debug) Task 30e6a4a0-966d-4155-adfb-4bfef8aadde3: ref 0 aborting False >Thread-15913625::INFO::2013-11-20 03:09:55,634::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913625::INFO::2013-11-20 03:09:55,635::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16) >Thread-15913625::DEBUG::2013-11-20 03:09:55,636::task::495::TaskManager.Task::(_debug) Task c8bac55b-a828-488b-ac81-2452f484433f: moving from state init -> state preparing >Thread-15913625::INFO::2013-11-20 03:09:55,636::blockSD::674::Storage.StorageDomain::(validate) sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 >Thread-15913625::DEBUG::2013-11-20 03:09:55,637::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913625::DEBUG::2013-11-20 03:09:55,638::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913625::DEBUG::2013-11-20 03:09:55,639::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913625::DEBUG::2013-11-20 03:09:55,640::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913625::DEBUG::2013-11-20 03:09:55,641::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15913625::DEBUG::2013-11-20 03:09:55,642::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15913625::DEBUG::2013-11-20 03:09:56,349::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913625::DEBUG::2013-11-20 03:09:56,353::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15913625::DEBUG::2013-11-20 03:09:56,357::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'_SHA_CKSUM=716ea7492d9134a4b13de27ac61a336ad474727c', u'POOL_SPM_ID=2', u'POOL_SPM_LVER=287'] >Thread-15913625::DEBUG::2013-11-20 03:09:56,360::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`13713d99-cc7d-482a-97ac-58caad5f4ff3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1297' at 'public_getStorageDomainInfo' >Thread-15913625::DEBUG::2013-11-20 03:09:56,361::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913625::DEBUG::2013-11-20 03:09:56,362::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913625::DEBUG::2013-11-20 03:09:56,362::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`13713d99-cc7d-482a-97ac-58caad5f4ff3`::Granted request >Thread-15913625::DEBUG::2013-11-20 03:09:56,363::task::495::TaskManager.Task::(_debug) Task c8bac55b-a828-488b-ac81-2452f484433f: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913625::DEBUG::2013-11-20 03:09:56,364::task::495::TaskManager.Task::(_debug) Task c8bac55b-a828-488b-ac81-2452f484433f: ref 1 aborting False >Thread-15913625::WARNING::2013-11-20 03:09:56,366::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913625::DEBUG::2013-11-20 03:09:56,367::task::495::TaskManager.Task::(_debug) Task c8bac55b-a828-488b-ac81-2452f484433f: finished: {'info': {'uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'vguuid': 'aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', 'lver': 287, 'state': 'OK', 'version': '2', 'role': 'Master', 'master_ver': 1, 'spm_id': 2, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'OS'}} >Thread-15913625::DEBUG::2013-11-20 03:09:56,372::task::495::TaskManager.Task::(_debug) Task c8bac55b-a828-488b-ac81-2452f484433f: moving from state preparing -> state finished >Thread-15913625::DEBUG::2013-11-20 03:09:56,373::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913625::DEBUG::2013-11-20 03:09:56,374::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913625::DEBUG::2013-11-20 03:09:56,375::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913625::DEBUG::2013-11-20 03:09:56,375::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913625::DEBUG::2013-11-20 03:09:56,376::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913625::DEBUG::2013-11-20 03:09:56,377::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913625::DEBUG::2013-11-20 03:09:56,377::task::495::TaskManager.Task::(_debug) Task c8bac55b-a828-488b-ac81-2452f484433f: ref 0 aborting False >Thread-15913625::INFO::2013-11-20 03:09:56,378::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'vguuid': 'aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', 'lver': 287, 'state': 'OK', 'version': '2', 'role': 'Master', 'master_ver': 1, 'spm_id': 2, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'OS'}} >Thread-15913625::DEBUG::2013-11-20 03:09:56,398::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/libexec/vdsm/hooks/before_vm_start/50_vhostmd' (cwd None) >Thread-15913625::DEBUG::2013-11-20 03:09:56,907::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913625::INFO::2013-11-20 03:09:56,908::hooks::51::root::(_runHooksDir) >Thread-15913625::DEBUG::2013-11-20 03:09:56,909::libvirtvm::1077::vm.Vm::(_run) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::<?xml version="1.0" encoding="utf-8"?> ><domain type="kvm"> > <name>KLEE-AOS01</name> > <uuid>4760db3e-d3a8-4045-9153-12c36ec16ba2</uuid> > <memory>4194304</memory> > <currentMemory>4194304</currentMemory> > <vcpu>4</vcpu> > <devices> > <disk device="disk" type="block"> > <source dev="/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/6f97dfa5-ee35-4f37-8495-1e6b3136b92c/5eaa8dde-10d8-45b5-9e10-5417c297c4be"/> > <target bus="virtio" dev="vda"/> > <serial>37-8495-1e6b3136b92c</serial> > <driver cache="none" error_policy="stop" io="native" name="qemu" type="raw"/> > </disk> > <disk device="cdrom" type="file"> > <target bus="ide" dev="hdc"/> > </disk> > <controller index="0" ports="16" type="virtio-serial"/> > <channel type="unix"> > <target name="com.redhat.rhevm.vdsm" type="virtio"/> > <source mode="bind" path="/var/lib/libvirt/qemu/channels/KLEE-AOS01.com.redhat.rhevm.vdsm"/> > </channel> > <memballoon model="none"/> > <interface type="bridge"> > <mac address="00:1a:4a:a8:64:04"/> > <model type="virtio"/> > <source bridge="rhevm"/> > </interface> > <input bus="ps2" type="mouse"/> > <video> > <model heads="1" type="qxl" vram="65536"/> > </video> > <channel type="spicevmc"> > <target name="com.redhat.spice.0" type="virtio"/> > </channel> > <graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice"> > <channel mode="secure" name="main"/> > <channel mode="secure" name="inputs"/> > </graphics> > </devices> > <os> > <type arch="x86_64" machine="rhel6.2.0">hvm</type> > <boot dev="hd"/> > <boot dev="cdrom"/> > <smbios mode="sysinfo"/> > </os> > <sysinfo type="smbios"> > <system> > <entry name="manufacturer">Red Hat</entry> > <entry name="product">RHEV Hypervisor</entry> > <entry name="version">6.3-20121012.0.el6_3</entry> > <entry name="serial">30343536-3138-5A43-4A32-333630373442_AC:16:2D:76:EB:F0</entry> > <entry name="uuid">4760db3e-d3a8-4045-9153-12c36ec16ba2</entry> > </system> > </sysinfo> > <clock adjustment="-1" offset="variable"> > <timer name="rtc" tickpolicy="catchup"/> > </clock> > <features> > <acpi/> > </features> > <cpu match="exact"> > <model>Westmere</model> > <topology cores="2" sockets="2" threads="1"/> > </cpu> ></domain> > >Dummy-15912651::DEBUG::2013-11-20 03:09:57,087::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:09:57,703::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0843804 s, 12.1 MB/s\n'; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-20 03:09:58,382::libvirtvm::1401::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::event Started detail 0 opaque None >Thread-15913625::DEBUG::2013-11-20 03:09:58,435::utils::307::vm.Vm::(start) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Start statistics collection >Thread-15913639::DEBUG::2013-11-20 03:09:58,436::utils::336::vm.Vm::(run) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Stats thread started >Thread-15913640::DEBUG::2013-11-20 03:09:58,439::guestIF::47::root::(_connect) Attempting connection to /var/lib/libvirt/qemu/channels/KLEE-AOS01.com.redhat.rhevm.vdsm >Thread-15913639::INFO::2013-11-20 03:09:58,441::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913640::DEBUG::2013-11-20 03:09:58,442::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /usr/share/vdsm/prepare-vmchannel /var/lib/libvirt/qemu/channels/KLEE-AOS01.com.redhat.rhevm.vdsm' (cwd None) >Thread-15913639::DEBUG::2013-11-20 03:09:58,443::task::495::TaskManager.Task::(_debug) Task a1d0828f-420f-4416-aea1-531294dab989: moving from state init -> state preparing >Thread-15913625::WARNING::2013-11-20 03:09:58,445::libvirtvm::1124::vm.Vm::(_readPauseCode) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::_readPauseCode unsupported by libvirt vm >Thread-15913639::DEBUG::2013-11-20 03:09:58,677::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`93dd46a9-c5bb-4191-a11a-600c6dd980b2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913625::DEBUG::2013-11-20 03:09:58,679::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /usr/share/vdsm/get-vm-pid KLEE-AOS01' (cwd None) >Thread-15913639::DEBUG::2013-11-20 03:09:58,680::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:09:59,104::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:09:59,109::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`93dd46a9-c5bb-4191-a11a-600c6dd980b2`::Granted request >Thread-15913640::DEBUG::2013-11-20 03:09:59,112::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913639::DEBUG::2013-11-20 03:09:59,115::task::495::TaskManager.Task::(_debug) Task a1d0828f-420f-4416-aea1-531294dab989: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913640::DEBUG::2013-11-20 03:09:59,116::guestIF::53::root::(_connect) connected to /var/lib/libvirt/qemu/channels/KLEE-AOS01.com.redhat.rhevm.vdsm >Thread-15913639::DEBUG::2013-11-20 03:09:59,116::task::495::TaskManager.Task::(_debug) Task a1d0828f-420f-4416-aea1-531294dab989: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:09:59,118::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913639::DEBUG::2013-11-20 03:09:59,120::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15913625::DEBUG::2013-11-20 03:09:59,497::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913625::DEBUG::2013-11-20 03:09:59,513::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /sbin/service ksmtuned retune' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:09:59,831::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913639::DEBUG::2013-11-20 03:10:00,233::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913625::DEBUG::2013-11-20 03:10:00,249::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913625::DEBUG::2013-11-20 03:10:00,251::vm::358::vm.Vm::(_startUnderlyingVm) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::_ongoingCreations released >Thread-15913631::DEBUG::2013-11-20 03:10:00,251::vm::346::vm.Vm::(_startUnderlyingVm) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::_ongoingCreations acquired >Thread-15913631::INFO::2013-11-20 03:10:00,316::libvirtvm::1061::vm.Vm::(_run) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::VM wrapper has started >Thread-15913631::INFO::2013-11-20 03:10:00,389::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913631::DEBUG::2013-11-20 03:10:00,452::task::495::TaskManager.Task::(_debug) Task 6d5199b3-fd75-4ada-a50e-c85fdc68047f: moving from state init -> state preparing >Thread-15913631::DEBUG::2013-11-20 03:10:00,559::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`47f0e53e-faa3-4b8c-87e7-0fdeaebe2b25`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1690' at 'public_prepareVolume' >Thread-15913631::DEBUG::2013-11-20 03:10:00,570::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:10:00,618::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913631::DEBUG::2013-11-20 03:10:00,619::resourceManager::481::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' found in shared state and queue is empty, Joining current shared lock (2 active users) >Thread-15913639::DEBUG::2013-11-20 03:10:00,622::task::495::TaskManager.Task::(_debug) Task a1d0828f-420f-4416-aea1-531294dab989: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913631::DEBUG::2013-11-20 03:10:00,623::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`47f0e53e-faa3-4b8c-87e7-0fdeaebe2b25`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:10:00,624::task::495::TaskManager.Task::(_debug) Task a1d0828f-420f-4416-aea1-531294dab989: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:10:00,626::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913631::DEBUG::2013-11-20 03:10:00,628::task::495::TaskManager.Task::(_debug) Task 6d5199b3-fd75-4ada-a50e-c85fdc68047f: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:10:00,629::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913631::DEBUG::2013-11-20 03:10:00,630::task::495::TaskManager.Task::(_debug) Task 6d5199b3-fd75-4ada-a50e-c85fdc68047f: ref 1 aborting False >Thread-15913631::DEBUG::2013-11-20 03:10:00,635::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.b74dd33b-748d-48df-83f2-ebcc3523e117`ReqID=`dc135158-444d-46f7-8e44-87c5210d8fb3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1695' at 'public_prepareVolume' >Thread-15913639::DEBUG::2013-11-20 03:10:00,635::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913631::DEBUG::2013-11-20 03:10:00,636::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.b74dd33b-748d-48df-83f2-ebcc3523e117' for lock type 'exclusive' >Thread-15913631::DEBUG::2013-11-20 03:10:00,639::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913639::DEBUG::2013-11-20 03:10:00,640::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (1 active users) >Thread-15913631::DEBUG::2013-11-20 03:10:00,644::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15913639::DEBUG::2013-11-20 03:10:00,646::task::495::TaskManager.Task::(_debug) Task a1d0828f-420f-4416-aea1-531294dab989: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:10:00,647::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913642::DEBUG::2013-11-20 03:10:00,999::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913642::INFO::2013-11-20 03:10:01,001::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913642::DEBUG::2013-11-20 03:10:01,005::task::495::TaskManager.Task::(_debug) Task 7348e82f-4073-44fe-9662-9399756680ed: moving from state init -> state preparing >Thread-15913642::DEBUG::2013-11-20 03:10:01,009::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913642::DEBUG::2013-11-20 03:10:01,010::task::495::TaskManager.Task::(_debug) Task 7348e82f-4073-44fe-9662-9399756680ed: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913642::DEBUG::2013-11-20 03:10:01,013::task::495::TaskManager.Task::(_debug) Task 7348e82f-4073-44fe-9662-9399756680ed: moving from state preparing -> state finished >Thread-15913642::DEBUG::2013-11-20 03:10:01,016::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913642::DEBUG::2013-11-20 03:10:01,017::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913642::DEBUG::2013-11-20 03:10:01,019::task::495::TaskManager.Task::(_debug) Task 7348e82f-4073-44fe-9662-9399756680ed: ref 0 aborting False >Thread-15913642::INFO::2013-11-20 03:10:01,020::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913643::DEBUG::2013-11-20 03:10:01,031::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913643::INFO::2013-11-20 03:10:01,033::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913643::DEBUG::2013-11-20 03:10:01,035::task::495::TaskManager.Task::(_debug) Task abe16ff2-2867-44bc-be19-2e7cb31b4d32: moving from state init -> state preparing >Thread-15913643::DEBUG::2013-11-20 03:10:01,037::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b60010a2-628f-4c1c-8672-41b50f138692`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913643::DEBUG::2013-11-20 03:10:01,038::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913643::DEBUG::2013-11-20 03:10:01,043::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913643::DEBUG::2013-11-20 03:10:01,044::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b60010a2-628f-4c1c-8672-41b50f138692`::Granted request >Thread-15913643::DEBUG::2013-11-20 03:10:01,047::task::495::TaskManager.Task::(_debug) Task abe16ff2-2867-44bc-be19-2e7cb31b4d32: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913643::DEBUG::2013-11-20 03:10:01,049::task::495::TaskManager.Task::(_debug) Task abe16ff2-2867-44bc-be19-2e7cb31b4d32: ref 1 aborting False >Thread-15913643::WARNING::2013-11-20 03:10:01,054::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913643::DEBUG::2013-11-20 03:10:01,056::task::495::TaskManager.Task::(_debug) Task abe16ff2-2867-44bc-be19-2e7cb31b4d32: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913643::DEBUG::2013-11-20 03:10:01,058::task::495::TaskManager.Task::(_debug) Task abe16ff2-2867-44bc-be19-2e7cb31b4d32: moving from state preparing -> state finished >Thread-15913643::DEBUG::2013-11-20 03:10:01,059::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913643::DEBUG::2013-11-20 03:10:01,060::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913643::DEBUG::2013-11-20 03:10:01,067::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913643::DEBUG::2013-11-20 03:10:01,068::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913643::DEBUG::2013-11-20 03:10:01,069::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913643::DEBUG::2013-11-20 03:10:01,071::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913643::DEBUG::2013-11-20 03:10:01,074::task::495::TaskManager.Task::(_debug) Task abe16ff2-2867-44bc-be19-2e7cb31b4d32: ref 0 aborting False >Thread-15913643::INFO::2013-11-20 03:10:01,075::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:10:01,117::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.618191 s, 1.7 MB/s\n'; <rc> = 0 >Thread-15913631::DEBUG::2013-11-20 03:10:01,630::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913631::DEBUG::2013-11-20 03:10:01,884::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913631::DEBUG::2013-11-20 03:10:01,890::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=11 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15913645::DEBUG::2013-11-20 03:10:02,143::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4760db3e-d3a8-4045-9153-12c36ec16ba2',) {} >Thread-15913645::DEBUG::2013-11-20 03:10:02,144::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::CPU stats not available >Thread-15913645::DEBUG::2013-11-20 03:10:02,145::libvirtvm::202::vm.Vm::(_getNetworkStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Network stats not available >Thread-15913645::DEBUG::2013-11-20 03:10:02,146::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk stats not available >Thread-15913645::DEBUG::2013-11-20 03:10:02,147::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk latency not available >Thread-15913645::DEBUG::2013-11-20 03:10:02,148::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '49279', 'displayIp': '0', 'displayPort': u'5906', 'session': 'Unknown', 'displaySecurePort': u'5907', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet4': {'state': 'unknown', 'macAddr': u'00:1a:4a:a8:64:04', 'speed': '1000', 'name': u'vnet4'}}, 'vmId': '4760db3e-d3a8-4045-9153-12c36ec16ba2', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '85899345920', 'writeLatency': '0', 'imageID': '6f97dfa5-ee35-4f37-8495-1e6b3136b92c', 'flushLatency': '0', 'truesize': '85899345920'}}, 'boot': 'cd', 'statsAge': '3.70', 'cpuIdle': '100.00', 'elapsedTime': '14', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913631::DEBUG::2013-11-20 03:10:02,340::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.00763903 s, 67.0 kB/s\n'; <rc> = 0 >Thread-15913631::DEBUG::2013-11-20 03:10:02,341::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.00763903 s, 67.0 kB/s'], size: 512 >Thread-15913631::DEBUG::2013-11-20 03:10:02,343::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=11 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15913631::DEBUG::2013-11-20 03:10:02,752::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000338672 s, 1.5 MB/s\n'; <rc> = 0 >Thread-15913631::DEBUG::2013-11-20 03:10:02,754::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000338672 s, 1.5 MB/s'], size: 512 >Thread-15913631::INFO::2013-11-20 03:10:02,755::image::340::Storage.Image::(getChain) sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 chain=[<storage.blockVolume.BlockVolume instance at 0x7f47180c4128>] >Thread-15913631::DEBUG::2013-11-20 03:10:02,756::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --autobackup n --available y 6b892067-ac7e-4039-a7be-5b7387525b16/8506c15b-61ae-4830-84ed-c830955c25a3' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:10:03,138::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:10:03,605::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0640672 s, 16.0 MB/s\n'; <rc> = 0 >Thread-15913631::DEBUG::2013-11-20 03:10:03,654::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913631::DEBUG::2013-11-20 03:10:03,656::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913631::DEBUG::2013-11-20 03:10:03,656::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913631::DEBUG::2013-11-20 03:10:03,658::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.8506c15b-61ae-4830-84ed-c830955c25a3`ReqID=`cefafd7f-8cd8-4c76-a571-146d0882160a`::Request was made in '/usr/share/vdsm/storage/resourceFactories.py' line '149' at '__getResourceCandidatesList' >Thread-15913631::DEBUG::2013-11-20 03:10:03,659::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.8506c15b-61ae-4830-84ed-c830955c25a3' for lock type 'exclusive' >Thread-15913631::DEBUG::2013-11-20 03:10:03,660::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.8506c15b-61ae-4830-84ed-c830955c25a3' is free. Now locking as 'exclusive' (1 active user) >Thread-15913631::DEBUG::2013-11-20 03:10:03,660::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.8506c15b-61ae-4830-84ed-c830955c25a3`ReqID=`cefafd7f-8cd8-4c76-a571-146d0882160a`::Granted request >Thread-15913631::DEBUG::2013-11-20 03:10:03,661::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.b74dd33b-748d-48df-83f2-ebcc3523e117' is free. Now locking as 'exclusive' (1 active user) >Thread-15913631::DEBUG::2013-11-20 03:10:03,662::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.b74dd33b-748d-48df-83f2-ebcc3523e117`ReqID=`dc135158-444d-46f7-8e44-87c5210d8fb3`::Granted request >Thread-15913631::DEBUG::2013-11-20 03:10:03,664::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913631::DEBUG::2013-11-20 03:10:03,666::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15913631::DEBUG::2013-11-20 03:10:04,504::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913631::DEBUG::2013-11-20 03:10:04,712::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913631::DEBUG::2013-11-20 03:10:04,713::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --refresh 6b892067-ac7e-4039-a7be-5b7387525b16/8506c15b-61ae-4830-84ed-c830955c25a3' (cwd None) >Thread-15913647::DEBUG::2013-11-20 03:10:05,239::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4760db3e-d3a8-4045-9153-12c36ec16ba2',) {} >Thread-15913647::DEBUG::2013-11-20 03:10:05,240::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::CPU stats not available >Thread-15913647::DEBUG::2013-11-20 03:10:05,241::libvirtvm::202::vm.Vm::(_getNetworkStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Network stats not available >Thread-15913647::DEBUG::2013-11-20 03:10:05,241::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk stats not available >Thread-15913647::DEBUG::2013-11-20 03:10:05,242::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk latency not available >Thread-15913647::DEBUG::2013-11-20 03:10:05,243::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '49279', 'displayIp': '0', 'displayPort': u'5906', 'session': 'Unknown', 'displaySecurePort': u'5907', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet4': {'state': 'unknown', 'macAddr': u'00:1a:4a:a8:64:04', 'speed': '1000', 'name': u'vnet4'}}, 'vmId': '4760db3e-d3a8-4045-9153-12c36ec16ba2', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '85899345920', 'writeLatency': '0', 'imageID': '6f97dfa5-ee35-4f37-8495-1e6b3136b92c', 'flushLatency': '0', 'truesize': '85899345920'}}, 'boot': 'cd', 'statsAge': '0.25', 'cpuIdle': '100.00', 'elapsedTime': '17', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:10:05,615::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913631::DEBUG::2013-11-20 03:10:06,000::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913631::DEBUG::2013-11-20 03:10:06,001::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913631::DEBUG::2013-11-20 03:10:06,002::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913631::INFO::2013-11-20 03:10:06,003::volume::517::Storage.Volume::(prepare) Volume: preparing volume 6b892067-ac7e-4039-a7be-5b7387525b16/8506c15b-61ae-4830-84ed-c830955c25a3 >Thread-15913631::DEBUG::2013-11-20 03:10:06,004::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913631::DEBUG::2013-11-20 03:10:06,005::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:10:06,251::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.252287 s, 4.1 MB/s\n'; <rc> = 0 >Thread-15913631::DEBUG::2013-11-20 03:10:06,971::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913631::DEBUG::2013-11-20 03:10:07,296::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913631::DEBUG::2013-11-20 03:10:07,298::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=11 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15913631::DEBUG::2013-11-20 03:10:07,877::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000240376 s, 2.1 MB/s\n'; <rc> = 0 >Thread-15913631::DEBUG::2013-11-20 03:10:07,879::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000240376 s, 2.1 MB/s'], size: 512 >Thread-15913631::DEBUG::2013-11-20 03:10:07,880::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=11 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:10:08,263::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913649::DEBUG::2013-11-20 03:10:08,502::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4760db3e-d3a8-4045-9153-12c36ec16ba2',) {} >Thread-15913649::DEBUG::2013-11-20 03:10:08,503::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::CPU stats not available >Thread-15913631::DEBUG::2013-11-20 03:10:08,505::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000230126 s, 2.2 MB/s\n'; <rc> = 0 >Thread-15913649::DEBUG::2013-11-20 03:10:08,505::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk stats not available >Thread-15913631::DEBUG::2013-11-20 03:10:08,506::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000230126 s, 2.2 MB/s'], size: 512 >Thread-15913649::DEBUG::2013-11-20 03:10:08,506::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk latency not available >Thread-15913631::DEBUG::2013-11-20 03:10:08,509::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.8506c15b-61ae-4830-84ed-c830955c25a3`ReqID=`8221faf1-4e78-4a4c-8772-cedc7fd01c58`::Request was made in '/usr/share/vdsm/storage/volume.py' line '533' at 'prepare' >Thread-15913649::DEBUG::2013-11-20 03:10:08,509::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '49279', 'displayIp': '0', 'displayPort': u'5906', 'session': 'Unknown', 'displaySecurePort': u'5907', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet4': {'macAddr': u'00:1a:4a:a8:64:04', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet4'}}, 'vmId': '4760db3e-d3a8-4045-9153-12c36ec16ba2', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '85899345920', 'writeLatency': '0', 'imageID': '6f97dfa5-ee35-4f37-8495-1e6b3136b92c', 'flushLatency': '0', 'truesize': '85899345920'}}, 'boot': 'cd', 'statsAge': '1.48', 'cpuIdle': '100.00', 'elapsedTime': '21', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913631::DEBUG::2013-11-20 03:10:08,510::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.8506c15b-61ae-4830-84ed-c830955c25a3' for lock type 'exclusive' >Thread-15913631::DEBUG::2013-11-20 03:10:08,512::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.8506c15b-61ae-4830-84ed-c830955c25a3' is free. Now locking as 'exclusive' (1 active user) >Thread-15913631::DEBUG::2013-11-20 03:10:08,513::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.8506c15b-61ae-4830-84ed-c830955c25a3`ReqID=`8221faf1-4e78-4a4c-8772-cedc7fd01c58`::Granted request >Thread-15913631::DEBUG::2013-11-20 03:10:08,514::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=11 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:10:09,057::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.095956 s, 10.7 MB/s\n'; <rc> = 0 >Thread-15913631::DEBUG::2013-11-20 03:10:09,106::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000153089 s, 3.3 MB/s\n'; <rc> = 0 >Thread-15913631::DEBUG::2013-11-20 03:10:09,107::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000153089 s, 3.3 MB/s'], size: 512 >Thread-15913631::DEBUG::2013-11-20 03:10:09,108::task::495::TaskManager.Task::(_debug) Task 6d5199b3-fd75-4ada-a50e-c85fdc68047f: finished: None >Thread-15913631::DEBUG::2013-11-20 03:10:09,109::task::495::TaskManager.Task::(_debug) Task 6d5199b3-fd75-4ada-a50e-c85fdc68047f: moving from state preparing -> state finished >Thread-15913631::DEBUG::2013-11-20 03:10:09,109::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913631::DEBUG::2013-11-20 03:10:09,110::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913631::DEBUG::2013-11-20 03:10:09,111::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913631::DEBUG::2013-11-20 03:10:09,111::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913631::DEBUG::2013-11-20 03:10:09,112::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913631::DEBUG::2013-11-20 03:10:09,112::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913631::DEBUG::2013-11-20 03:10:09,112::task::495::TaskManager.Task::(_debug) Task 6d5199b3-fd75-4ada-a50e-c85fdc68047f: ref 0 aborting False >Thread-15913631::INFO::2013-11-20 03:10:09,113::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15913631::INFO::2013-11-20 03:10:09,114::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913631::DEBUG::2013-11-20 03:10:09,114::task::495::TaskManager.Task::(_debug) Task ffe2360f-0e81-4786-b017-07d00f15862c: moving from state init -> state preparing >Thread-15913631::DEBUG::2013-11-20 03:10:09,115::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`1811646a-5859-4197-8ed9-009a1e9d0929`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1666' at 'public_getVolumePath' >Thread-15913631::DEBUG::2013-11-20 03:10:09,115::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913631::DEBUG::2013-11-20 03:10:09,116::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913631::DEBUG::2013-11-20 03:10:09,116::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`1811646a-5859-4197-8ed9-009a1e9d0929`::Granted request >Thread-15913631::DEBUG::2013-11-20 03:10:09,117::task::495::TaskManager.Task::(_debug) Task ffe2360f-0e81-4786-b017-07d00f15862c: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913631::DEBUG::2013-11-20 03:10:09,118::task::495::TaskManager.Task::(_debug) Task ffe2360f-0e81-4786-b017-07d00f15862c: ref 1 aborting False >Thread-15913631::DEBUG::2013-11-20 03:10:09,118::task::495::TaskManager.Task::(_debug) Task ffe2360f-0e81-4786-b017-07d00f15862c: finished: {'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/b74dd33b-748d-48df-83f2-ebcc3523e117/8506c15b-61ae-4830-84ed-c830955c25a3'} >Thread-15913631::DEBUG::2013-11-20 03:10:09,119::task::495::TaskManager.Task::(_debug) Task ffe2360f-0e81-4786-b017-07d00f15862c: moving from state preparing -> state finished >Thread-15913631::DEBUG::2013-11-20 03:10:09,119::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913631::DEBUG::2013-11-20 03:10:09,120::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913631::DEBUG::2013-11-20 03:10:09,120::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913631::DEBUG::2013-11-20 03:10:09,120::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913631::DEBUG::2013-11-20 03:10:09,121::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913631::DEBUG::2013-11-20 03:10:09,121::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913631::DEBUG::2013-11-20 03:10:09,122::task::495::TaskManager.Task::(_debug) Task ffe2360f-0e81-4786-b017-07d00f15862c: ref 0 aborting False >Thread-15913631::INFO::2013-11-20 03:10:09,122::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, Return response: {'status': {'message': 'OK', 'code': 0}, 'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/b74dd33b-748d-48df-83f2-ebcc3523e117/8506c15b-61ae-4830-84ed-c830955c25a3'} >Thread-15913631::INFO::2013-11-20 03:10:09,122::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913631::DEBUG::2013-11-20 03:10:09,123::task::495::TaskManager.Task::(_debug) Task 3e95a198-743d-447e-9e03-401d6f7c901f: moving from state init -> state preparing >Thread-15913631::DEBUG::2013-11-20 03:10:09,124::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`621c0327-8cc9-4d69-8b4d-b70099e1d312`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913631::DEBUG::2013-11-20 03:10:09,124::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913631::DEBUG::2013-11-20 03:10:09,125::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913631::DEBUG::2013-11-20 03:10:09,125::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`621c0327-8cc9-4d69-8b4d-b70099e1d312`::Granted request >Thread-15913631::DEBUG::2013-11-20 03:10:09,125::task::495::TaskManager.Task::(_debug) Task 3e95a198-743d-447e-9e03-401d6f7c901f: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913631::DEBUG::2013-11-20 03:10:09,126::task::495::TaskManager.Task::(_debug) Task 3e95a198-743d-447e-9e03-401d6f7c901f: ref 1 aborting False >Thread-15913631::DEBUG::2013-11-20 03:10:09,126::task::495::TaskManager.Task::(_debug) Task 3e95a198-743d-447e-9e03-401d6f7c901f: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913631::DEBUG::2013-11-20 03:10:09,126::task::495::TaskManager.Task::(_debug) Task 3e95a198-743d-447e-9e03-401d6f7c901f: moving from state preparing -> state finished >Thread-15913631::DEBUG::2013-11-20 03:10:09,127::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913631::DEBUG::2013-11-20 03:10:09,127::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913631::DEBUG::2013-11-20 03:10:09,128::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913631::DEBUG::2013-11-20 03:10:09,128::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913631::DEBUG::2013-11-20 03:10:09,128::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913631::DEBUG::2013-11-20 03:10:09,129::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913631::DEBUG::2013-11-20 03:10:09,129::task::495::TaskManager.Task::(_debug) Task 3e95a198-743d-447e-9e03-401d6f7c901f: ref 0 aborting False >Thread-15913631::INFO::2013-11-20 03:10:09,130::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913631::INFO::2013-11-20 03:10:09,130::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16) >Thread-15913631::DEBUG::2013-11-20 03:10:09,130::task::495::TaskManager.Task::(_debug) Task 567eb2d2-3196-4994-94b1-1f4075e91607: moving from state init -> state preparing >Thread-15913631::INFO::2013-11-20 03:10:09,131::blockSD::674::Storage.StorageDomain::(validate) sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 >Thread-15913631::DEBUG::2013-11-20 03:10:09,131::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913631::DEBUG::2013-11-20 03:10:09,132::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913631::DEBUG::2013-11-20 03:10:09,132::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913631::DEBUG::2013-11-20 03:10:09,133::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913631::DEBUG::2013-11-20 03:10:09,134::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15913631::DEBUG::2013-11-20 03:10:09,134::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15913631::DEBUG::2013-11-20 03:10:10,540::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913631::DEBUG::2013-11-20 03:10:10,542::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15913631::DEBUG::2013-11-20 03:10:10,543::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'_SHA_CKSUM=716ea7492d9134a4b13de27ac61a336ad474727c', u'POOL_SPM_ID=2', u'POOL_SPM_LVER=287'] >Thread-15913631::DEBUG::2013-11-20 03:10:10,544::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`25f25ed8-8051-48ba-ae5f-5d9cab618c17`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1297' at 'public_getStorageDomainInfo' >Thread-15913631::DEBUG::2013-11-20 03:10:10,545::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913631::DEBUG::2013-11-20 03:10:10,545::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913631::DEBUG::2013-11-20 03:10:10,546::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`25f25ed8-8051-48ba-ae5f-5d9cab618c17`::Granted request >Thread-15913631::DEBUG::2013-11-20 03:10:10,546::task::495::TaskManager.Task::(_debug) Task 567eb2d2-3196-4994-94b1-1f4075e91607: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913631::DEBUG::2013-11-20 03:10:10,547::task::495::TaskManager.Task::(_debug) Task 567eb2d2-3196-4994-94b1-1f4075e91607: ref 1 aborting False >Thread-15913631::WARNING::2013-11-20 03:10:10,548::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913631::DEBUG::2013-11-20 03:10:10,549::task::495::TaskManager.Task::(_debug) Task 567eb2d2-3196-4994-94b1-1f4075e91607: finished: {'info': {'uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'vguuid': 'aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', 'lver': 287, 'state': 'OK', 'version': '2', 'role': 'Master', 'master_ver': 1, 'spm_id': 2, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'OS'}} >Thread-15913631::DEBUG::2013-11-20 03:10:10,549::task::495::TaskManager.Task::(_debug) Task 567eb2d2-3196-4994-94b1-1f4075e91607: moving from state preparing -> state finished >Thread-15913631::DEBUG::2013-11-20 03:10:10,549::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913631::DEBUG::2013-11-20 03:10:10,550::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913631::DEBUG::2013-11-20 03:10:10,550::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913631::DEBUG::2013-11-20 03:10:10,551::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913631::DEBUG::2013-11-20 03:10:10,551::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913631::DEBUG::2013-11-20 03:10:10,551::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913631::DEBUG::2013-11-20 03:10:10,552::task::495::TaskManager.Task::(_debug) Task 567eb2d2-3196-4994-94b1-1f4075e91607: ref 0 aborting False >Thread-15913631::INFO::2013-11-20 03:10:10,552::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'vguuid': 'aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', 'lver': 287, 'state': 'OK', 'version': '2', 'role': 'Master', 'master_ver': 1, 'spm_id': 2, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'OS'}} >Thread-15913631::DEBUG::2013-11-20 03:10:10,577::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/libexec/vdsm/hooks/before_vm_start/50_vhostmd' (cwd None) >Thread-15913631::DEBUG::2013-11-20 03:10:10,989::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913631::INFO::2013-11-20 03:10:10,990::hooks::51::root::(_runHooksDir) >Thread-15913631::DEBUG::2013-11-20 03:10:10,991::libvirtvm::1077::vm.Vm::(_run) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::<?xml version="1.0" encoding="utf-8"?> ><domain type="kvm"> > <name>KLEE-AOS02</name> > <uuid>0dc7e997-e884-4493-b701-790fb37b3d80</uuid> > <memory>4194304</memory> > <currentMemory>4194304</currentMemory> > <vcpu>4</vcpu> > <devices> > <disk device="disk" type="block"> > <source dev="/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/b74dd33b-748d-48df-83f2-ebcc3523e117/8506c15b-61ae-4830-84ed-c830955c25a3"/> > <target bus="virtio" dev="vda"/> > <serial>df-83f2-ebcc3523e117</serial> > <driver cache="none" error_policy="stop" io="native" name="qemu" type="raw"/> > </disk> > <disk device="cdrom" type="file"> > <target bus="ide" dev="hdc"/> > </disk> > <controller index="0" ports="16" type="virtio-serial"/> > <channel type="unix"> > <target name="com.redhat.rhevm.vdsm" type="virtio"/> > <source mode="bind" path="/var/lib/libvirt/qemu/channels/KLEE-AOS02.com.redhat.rhevm.vdsm"/> > </channel> > <memballoon model="none"/> > <interface type="bridge"> > <mac address="00:1a:4a:a8:64:03"/> > <model type="virtio"/> > <source bridge="rhevm"/> > </interface> > <input bus="ps2" type="mouse"/> > <video> > <model heads="1" type="qxl" vram="65536"/> > </video> > <channel type="spicevmc"> > <target name="com.redhat.spice.0" type="virtio"/> > </channel> > <graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice"> > <channel mode="secure" name="main"/> > <channel mode="secure" name="inputs"/> > </graphics> > </devices> > <os> > <type arch="x86_64" machine="rhel6.2.0">hvm</type> > <boot dev="hd"/> > <boot dev="cdrom"/> > <smbios mode="sysinfo"/> > </os> > <sysinfo type="smbios"> > <system> > <entry name="manufacturer">Red Hat</entry> > <entry name="product">RHEV Hypervisor</entry> > <entry name="version">6.3-20121012.0.el6_3</entry> > <entry name="serial">30343536-3138-5A43-4A32-333630373442_AC:16:2D:76:EB:F0</entry> > <entry name="uuid">0dc7e997-e884-4493-b701-790fb37b3d80</entry> > </system> > </sysinfo> > <clock adjustment="3600" offset="variable"> > <timer name="rtc" tickpolicy="catchup"/> > </clock> > <features> > <acpi/> > </features> > <cpu match="exact"> > <model>Westmere</model> > <topology cores="2" sockets="2" threads="1"/> > </cpu> ></domain> > >Dummy-15912651::DEBUG::2013-11-20 03:10:11,068::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913650::DEBUG::2013-11-20 03:10:11,275::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913650::INFO::2013-11-20 03:10:11,276::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913650::DEBUG::2013-11-20 03:10:11,277::task::495::TaskManager.Task::(_debug) Task fab6245e-6ae9-4f48-8eae-ecfdab4234d6: moving from state init -> state preparing >Thread-15913650::DEBUG::2013-11-20 03:10:11,278::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913650::DEBUG::2013-11-20 03:10:11,278::task::495::TaskManager.Task::(_debug) Task fab6245e-6ae9-4f48-8eae-ecfdab4234d6: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913650::DEBUG::2013-11-20 03:10:11,279::task::495::TaskManager.Task::(_debug) Task fab6245e-6ae9-4f48-8eae-ecfdab4234d6: moving from state preparing -> state finished >Thread-15913650::DEBUG::2013-11-20 03:10:11,279::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913650::DEBUG::2013-11-20 03:10:11,279::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913650::DEBUG::2013-11-20 03:10:11,280::task::495::TaskManager.Task::(_debug) Task fab6245e-6ae9-4f48-8eae-ecfdab4234d6: ref 0 aborting False >Thread-15913650::INFO::2013-11-20 03:10:11,280::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913651::DEBUG::2013-11-20 03:10:11,284::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913651::INFO::2013-11-20 03:10:11,285::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913651::DEBUG::2013-11-20 03:10:11,285::task::495::TaskManager.Task::(_debug) Task eb0fa976-5ca8-4463-a20f-ceaeae71d2c7: moving from state init -> state preparing >Thread-15913651::DEBUG::2013-11-20 03:10:11,286::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d318ff2f-1cb7-47ea-963c-d07a3c56ec5c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913651::DEBUG::2013-11-20 03:10:11,287::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913651::DEBUG::2013-11-20 03:10:11,287::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913651::DEBUG::2013-11-20 03:10:11,288::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d318ff2f-1cb7-47ea-963c-d07a3c56ec5c`::Granted request >Thread-15913651::DEBUG::2013-11-20 03:10:11,288::task::495::TaskManager.Task::(_debug) Task eb0fa976-5ca8-4463-a20f-ceaeae71d2c7: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913651::DEBUG::2013-11-20 03:10:11,289::task::495::TaskManager.Task::(_debug) Task eb0fa976-5ca8-4463-a20f-ceaeae71d2c7: ref 1 aborting False >Thread-15913651::WARNING::2013-11-20 03:10:11,290::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913651::DEBUG::2013-11-20 03:10:11,291::task::495::TaskManager.Task::(_debug) Task eb0fa976-5ca8-4463-a20f-ceaeae71d2c7: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913651::DEBUG::2013-11-20 03:10:11,292::task::495::TaskManager.Task::(_debug) Task eb0fa976-5ca8-4463-a20f-ceaeae71d2c7: moving from state preparing -> state finished >Thread-15913651::DEBUG::2013-11-20 03:10:11,292::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913651::DEBUG::2013-11-20 03:10:11,292::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913651::DEBUG::2013-11-20 03:10:11,293::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913651::DEBUG::2013-11-20 03:10:11,293::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913651::DEBUG::2013-11-20 03:10:11,293::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913651::DEBUG::2013-11-20 03:10:11,294::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913651::DEBUG::2013-11-20 03:10:11,294::task::495::TaskManager.Task::(_debug) Task eb0fa976-5ca8-4463-a20f-ceaeae71d2c7: ref 0 aborting False >Thread-15913651::INFO::2013-11-20 03:10:11,295::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913652::INFO::2013-11-20 03:10:11,530::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913652::DEBUG::2013-11-20 03:10:11,530::task::495::TaskManager.Task::(_debug) Task e4b7b303-4a9f-46ae-b03d-8089670c629b: moving from state init -> state preparing >Thread-15913652::DEBUG::2013-11-20 03:10:11,531::task::495::TaskManager.Task::(_debug) Task e4b7b303-4a9f-46ae-b03d-8089670c629b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000564098358154', 'lastCheck': 1384917001.5123861, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00155115127563', 'lastCheck': 1384917009.94769, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00176382064819', 'lastCheck': 1384917010.944814, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000507831573486', 'lastCheck': 1384917001.212626, 'valid': True, 'code': 0}} >Thread-15913652::DEBUG::2013-11-20 03:10:11,532::task::495::TaskManager.Task::(_debug) Task e4b7b303-4a9f-46ae-b03d-8089670c629b: moving from state preparing -> state finished >Thread-15913652::DEBUG::2013-11-20 03:10:11,532::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913652::DEBUG::2013-11-20 03:10:11,533::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913652::DEBUG::2013-11-20 03:10:11,533::task::495::TaskManager.Task::(_debug) Task e4b7b303-4a9f-46ae-b03d-8089670c629b: ref 0 aborting False >Thread-15913652::INFO::2013-11-20 03:10:11,534::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000564098358154', 'lastCheck': 1384917001.5123861, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00155115127563', 'lastCheck': 1384917009.94769, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00176382064819', 'lastCheck': 1384917010.944814, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000507831573486', 'lastCheck': 1384917001.212626, 'valid': True, 'code': 0}} >Thread-15913653::DEBUG::2013-11-20 03:10:11,548::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::CPU stats not available >Thread-15913653::DEBUG::2013-11-20 03:10:11,550::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk stats not available >Thread-15913653::DEBUG::2013-11-20 03:10:11,551::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk latency not available >Dummy-15912651::DEBUG::2013-11-20 03:10:11,591::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0585906 s, 17.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:10:13,602::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:10:14,360::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0504733 s, 20.3 MB/s\n'; <rc> = 0 >Thread-15913655::DEBUG::2013-11-20 03:10:14,723::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4760db3e-d3a8-4045-9153-12c36ec16ba2',) {} >Thread-15913655::DEBUG::2013-11-20 03:10:14,724::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::CPU stats not available >Thread-15913655::DEBUG::2013-11-20 03:10:14,724::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk stats not available >Thread-15913655::DEBUG::2013-11-20 03:10:14,725::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk latency not available >Thread-15913655::DEBUG::2013-11-20 03:10:14,725::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '49279', 'displayIp': '0', 'displayPort': u'5906', 'session': 'Unknown', 'displaySecurePort': u'5907', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet4': {'macAddr': u'00:1a:4a:a8:64:04', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet4'}}, 'vmId': '4760db3e-d3a8-4045-9153-12c36ec16ba2', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '85899345920', 'writeLatency': '0', 'imageID': '6f97dfa5-ee35-4f37-8495-1e6b3136b92c', 'flushLatency': '0', 'truesize': '85899345920'}}, 'boot': 'cd', 'statsAge': '3.70', 'cpuIdle': '100.00', 'elapsedTime': '27', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:10:16,370::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:10:17,309::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0671163 s, 15.3 MB/s\n'; <rc> = 0 >Thread-15913657::DEBUG::2013-11-20 03:10:17,749::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4760db3e-d3a8-4045-9153-12c36ec16ba2',) {} >Thread-15913657::DEBUG::2013-11-20 03:10:17,750::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::CPU stats not available >Thread-15913657::DEBUG::2013-11-20 03:10:17,750::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk stats not available >Thread-15913657::DEBUG::2013-11-20 03:10:17,751::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk latency not available >Thread-15913657::DEBUG::2013-11-20 03:10:17,751::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '49279', 'displayIp': '0', 'displayPort': u'5906', 'session': 'Unknown', 'displaySecurePort': u'5907', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet4': {'macAddr': u'00:1a:4a:a8:64:04', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet4'}}, 'vmId': '4760db3e-d3a8-4045-9153-12c36ec16ba2', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '85899345920', 'writeLatency': '0', 'imageID': '6f97dfa5-ee35-4f37-8495-1e6b3136b92c', 'flushLatency': '0', 'truesize': '85899345920'}}, 'boot': 'cd', 'statsAge': '6.73', 'cpuIdle': '100.00', 'elapsedTime': '30', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:10:19,319::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:10:19,749::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0465877 s, 22.0 MB/s\n'; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-20 03:10:20,250::libvirtvm::1401::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::event Started detail 0 opaque None >Thread-15913631::DEBUG::2013-11-20 03:10:20,299::utils::307::vm.Vm::(start) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Start statistics collection >Thread-15913658::DEBUG::2013-11-20 03:10:20,300::utils::336::vm.Vm::(run) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Stats thread started >Thread-15913659::DEBUG::2013-11-20 03:10:20,302::guestIF::47::root::(_connect) Attempting connection to /var/lib/libvirt/qemu/channels/KLEE-AOS02.com.redhat.rhevm.vdsm >Thread-15913658::INFO::2013-11-20 03:10:20,302::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913659::DEBUG::2013-11-20 03:10:20,303::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /usr/share/vdsm/prepare-vmchannel /var/lib/libvirt/qemu/channels/KLEE-AOS02.com.redhat.rhevm.vdsm' (cwd None) >Thread-15913658::DEBUG::2013-11-20 03:10:20,304::task::495::TaskManager.Task::(_debug) Task 41bf2d30-79d8-4dee-93e3-f07eddda1b5b: moving from state init -> state preparing >Thread-15913631::WARNING::2013-11-20 03:10:20,305::libvirtvm::1124::vm.Vm::(_readPauseCode) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::_readPauseCode unsupported by libvirt vm >Thread-15913658::DEBUG::2013-11-20 03:10:20,306::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`734c584a-fc62-4f29-94f0-e32c91056105`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913631::DEBUG::2013-11-20 03:10:20,464::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /usr/share/vdsm/get-vm-pid KLEE-AOS02' (cwd None) >Thread-15913658::DEBUG::2013-11-20 03:10:20,465::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:10:20,627::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:10:20,628::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`734c584a-fc62-4f29-94f0-e32c91056105`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:10:20,651::task::495::TaskManager.Task::(_debug) Task 41bf2d30-79d8-4dee-93e3-f07eddda1b5b: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:10:20,651::task::495::TaskManager.Task::(_debug) Task 41bf2d30-79d8-4dee-93e3-f07eddda1b5b: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:10:20,652::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913658::DEBUG::2013-11-20 03:10:20,654::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15913659::DEBUG::2013-11-20 03:10:21,073::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913631::DEBUG::2013-11-20 03:10:21,079::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913659::DEBUG::2013-11-20 03:10:21,081::guestIF::53::root::(_connect) connected to /var/lib/libvirt/qemu/channels/KLEE-AOS02.com.redhat.rhevm.vdsm >Thread-15913661::DEBUG::2013-11-20 03:10:21,083::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4760db3e-d3a8-4045-9153-12c36ec16ba2',) {} >Thread-15913661::DEBUG::2013-11-20 03:10:21,085::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::CPU stats not available >Thread-15913661::DEBUG::2013-11-20 03:10:21,085::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk stats not available >Thread-15913661::DEBUG::2013-11-20 03:10:21,086::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk latency not available >Thread-15913661::DEBUG::2013-11-20 03:10:21,087::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '49279', 'displayIp': '0', 'displayPort': u'5906', 'session': 'Unknown', 'displaySecurePort': u'5907', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet4': {'macAddr': u'00:1a:4a:a8:64:04', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet4'}}, 'vmId': '4760db3e-d3a8-4045-9153-12c36ec16ba2', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '85899345920', 'writeLatency': '0', 'imageID': '6f97dfa5-ee35-4f37-8495-1e6b3136b92c', 'flushLatency': '0', 'truesize': '85899345920'}}, 'boot': 'cd', 'statsAge': '1.87', 'cpuIdle': '100.00', 'elapsedTime': '33', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913631::DEBUG::2013-11-20 03:10:21,094::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /sbin/service ksmtuned retune' (cwd None) >Thread-15913662::DEBUG::2013-11-20 03:10:21,536::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913662::INFO::2013-11-20 03:10:21,538::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913662::DEBUG::2013-11-20 03:10:21,539::task::495::TaskManager.Task::(_debug) Task 5542c1e1-8a07-4431-9834-7a5edd02314b: moving from state init -> state preparing >Thread-15913662::DEBUG::2013-11-20 03:10:21,540::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913662::DEBUG::2013-11-20 03:10:21,540::task::495::TaskManager.Task::(_debug) Task 5542c1e1-8a07-4431-9834-7a5edd02314b: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913662::DEBUG::2013-11-20 03:10:21,540::task::495::TaskManager.Task::(_debug) Task 5542c1e1-8a07-4431-9834-7a5edd02314b: moving from state preparing -> state finished >Thread-15913662::DEBUG::2013-11-20 03:10:21,541::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913662::DEBUG::2013-11-20 03:10:21,541::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913662::DEBUG::2013-11-20 03:10:21,542::task::495::TaskManager.Task::(_debug) Task 5542c1e1-8a07-4431-9834-7a5edd02314b: ref 0 aborting False >Thread-15913662::INFO::2013-11-20 03:10:21,542::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913663::DEBUG::2013-11-20 03:10:21,546::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913663::INFO::2013-11-20 03:10:21,547::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913663::DEBUG::2013-11-20 03:10:21,547::task::495::TaskManager.Task::(_debug) Task 6b5726fe-4d38-441c-9e92-c7fba7e554f5: moving from state init -> state preparing >Thread-15913663::DEBUG::2013-11-20 03:10:21,549::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d8026d2e-2274-4a6b-9c37-8c5f7d1794d8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913663::DEBUG::2013-11-20 03:10:21,549::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913663::DEBUG::2013-11-20 03:10:21,550::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913663::DEBUG::2013-11-20 03:10:21,550::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d8026d2e-2274-4a6b-9c37-8c5f7d1794d8`::Granted request >Thread-15913663::DEBUG::2013-11-20 03:10:21,551::task::495::TaskManager.Task::(_debug) Task 6b5726fe-4d38-441c-9e92-c7fba7e554f5: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913663::DEBUG::2013-11-20 03:10:21,552::task::495::TaskManager.Task::(_debug) Task 6b5726fe-4d38-441c-9e92-c7fba7e554f5: ref 1 aborting False >Thread-15913663::WARNING::2013-11-20 03:10:21,554::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913663::DEBUG::2013-11-20 03:10:21,554::task::495::TaskManager.Task::(_debug) Task 6b5726fe-4d38-441c-9e92-c7fba7e554f5: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913663::DEBUG::2013-11-20 03:10:21,555::task::495::TaskManager.Task::(_debug) Task 6b5726fe-4d38-441c-9e92-c7fba7e554f5: moving from state preparing -> state finished >Thread-15913663::DEBUG::2013-11-20 03:10:21,556::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913663::DEBUG::2013-11-20 03:10:21,556::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913663::DEBUG::2013-11-20 03:10:21,556::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913663::DEBUG::2013-11-20 03:10:21,557::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913663::DEBUG::2013-11-20 03:10:21,557::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913663::DEBUG::2013-11-20 03:10:21,557::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913663::DEBUG::2013-11-20 03:10:21,558::task::495::TaskManager.Task::(_debug) Task 6b5726fe-4d38-441c-9e92-c7fba7e554f5: ref 0 aborting False >Thread-15913663::INFO::2013-11-20 03:10:21,558::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:10:21,760::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913658::DEBUG::2013-11-20 03:10:22,077::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913658::DEBUG::2013-11-20 03:10:22,245::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913631::DEBUG::2013-11-20 03:10:22,247::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913658::DEBUG::2013-11-20 03:10:22,248::task::495::TaskManager.Task::(_debug) Task 41bf2d30-79d8-4dee-93e3-f07eddda1b5b: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:10:22,249::task::495::TaskManager.Task::(_debug) Task 41bf2d30-79d8-4dee-93e3-f07eddda1b5b: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:10:22,250::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913631::DEBUG::2013-11-20 03:10:22,250::vm::358::vm.Vm::(_startUnderlyingVm) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::_ongoingCreations released >Thread-15913635::DEBUG::2013-11-20 03:10:22,261::vm::346::vm.Vm::(_startUnderlyingVm) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::_ongoingCreations acquired >Thread-15913658::DEBUG::2013-11-20 03:10:22,276::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913635::INFO::2013-11-20 03:10:22,282::libvirtvm::1061::vm.Vm::(_run) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::VM wrapper has started >Thread-15913635::INFO::2013-11-20 03:10:22,315::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913658::DEBUG::2013-11-20 03:10:22,318::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913635::DEBUG::2013-11-20 03:10:22,320::task::495::TaskManager.Task::(_debug) Task 548931ca-5754-4a34-a2c3-0c27b288a873: moving from state init -> state preparing >Thread-15913635::DEBUG::2013-11-20 03:10:22,344::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a94a39a5-d1dc-44fa-9e90-2b09aabb964f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1690' at 'public_prepareVolume' >Thread-15913658::DEBUG::2013-11-20 03:10:22,349::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913635::DEBUG::2013-11-20 03:10:22,349::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:10:22,353::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:10:22,361::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913635::DEBUG::2013-11-20 03:10:22,361::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:10:22,362::task::495::TaskManager.Task::(_debug) Task 41bf2d30-79d8-4dee-93e3-f07eddda1b5b: ref 0 aborting False >Thread-15913635::DEBUG::2013-11-20 03:10:22,363::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a94a39a5-d1dc-44fa-9e90-2b09aabb964f`::Granted request >Thread-15913658::INFO::2013-11-20 03:10:22,363::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913635::DEBUG::2013-11-20 03:10:22,369::task::495::TaskManager.Task::(_debug) Task 548931ca-5754-4a34-a2c3-0c27b288a873: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913635::DEBUG::2013-11-20 03:10:22,377::task::495::TaskManager.Task::(_debug) Task 548931ca-5754-4a34-a2c3-0c27b288a873: ref 1 aborting False >Thread-15913635::DEBUG::2013-11-20 03:10:22,411::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.cc863eeb-33db-4227-b4eb-9f4347299cf3`ReqID=`7225744f-a9b1-48fe-81ea-26e74febb937`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1695' at 'public_prepareVolume' >Thread-15913635::DEBUG::2013-11-20 03:10:22,413::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.cc863eeb-33db-4227-b4eb-9f4347299cf3' for lock type 'exclusive' >Thread-15913635::DEBUG::2013-11-20 03:10:22,421::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913635::DEBUG::2013-11-20 03:10:22,455::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:10:22,714::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.429326 s, 2.4 MB/s\n'; <rc> = 0 >Thread-15913635::DEBUG::2013-11-20 03:10:23,426::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913635::DEBUG::2013-11-20 03:10:24,600::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913635::DEBUG::2013-11-20 03:10:24,602::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=23 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15913665::DEBUG::2013-11-20 03:10:24,603::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4760db3e-d3a8-4045-9153-12c36ec16ba2',) {} >Thread-15913665::DEBUG::2013-11-20 03:10:24,604::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk stats not available >Dummy-15912651::DEBUG::2013-11-20 03:10:24,814::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913665::DEBUG::2013-11-20 03:10:24,816::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk latency not available >Thread-15913665::DEBUG::2013-11-20 03:10:24,818::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '49279', 'displayIp': '0', 'displayPort': u'5906', 'session': 'Unknown', 'displaySecurePort': u'5907', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet4': {'macAddr': u'00:1a:4a:a8:64:04', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet4'}}, 'vmId': '4760db3e-d3a8-4045-9153-12c36ec16ba2', 'monitorResponse': '0', 'cpuUser': '292.93', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '85899345920', 'writeLatency': '0', 'imageID': '6f97dfa5-ee35-4f37-8495-1e6b3136b92c', 'flushLatency': '0', 'truesize': '85899345920'}}, 'boot': 'cd', 'statsAge': '1.17', 'cpuIdle': '0.00', 'elapsedTime': '37', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913666::DEBUG::2013-11-20 03:10:24,994::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('0dc7e997-e884-4493-b701-790fb37b3d80',) {} >Thread-15913666::DEBUG::2013-11-20 03:10:24,996::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::CPU stats not available >Thread-15913666::DEBUG::2013-11-20 03:10:24,997::libvirtvm::202::vm.Vm::(_getNetworkStats) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Network stats not available >Thread-15913666::DEBUG::2013-11-20 03:10:24,997::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk stats not available >Thread-15913666::DEBUG::2013-11-20 03:10:24,998::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk latency not available >Thread-15913666::DEBUG::2013-11-20 03:10:24,999::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '49674', 'displayIp': '0', 'displayPort': u'5908', 'session': 'Unknown', 'displaySecurePort': u'5909', 'timeOffset': '3600', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet5': {'state': 'unknown', 'macAddr': u'00:1a:4a:a8:64:03', 'speed': '1000', 'name': u'vnet5'}}, 'vmId': '0dc7e997-e884-4493-b701-790fb37b3d80', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '85899345920', 'writeLatency': '0', 'imageID': 'b74dd33b-748d-48df-83f2-ebcc3523e117', 'flushLatency': '0', 'truesize': '85899345920'}}, 'boot': 'cd', 'statsAge': '0.56', 'cpuIdle': '100.00', 'elapsedTime': '33', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913635::DEBUG::2013-11-20 03:10:26,432::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.0116596 s, 43.9 kB/s\n'; <rc> = 0 >Thread-15913635::DEBUG::2013-11-20 03:10:26,433::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.0116596 s, 43.9 kB/s'], size: 512 >Thread-15913635::DEBUG::2013-11-20 03:10:26,435::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=23 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15913635::DEBUG::2013-11-20 03:10:27,025::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000144369 s, 3.5 MB/s\n'; <rc> = 0 >Thread-15913635::DEBUG::2013-11-20 03:10:27,026::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000144369 s, 3.5 MB/s'], size: 512 >Thread-15913635::INFO::2013-11-20 03:10:27,027::image::340::Storage.Image::(getChain) sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 chain=[<storage.blockVolume.BlockVolume instance at 0x7f488c120a28>] >Thread-15913635::DEBUG::2013-11-20 03:10:27,029::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --autobackup n --available y 6b892067-ac7e-4039-a7be-5b7387525b16/ff5e3845-0d0d-4037-a0de-a3f584a4aef5' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:10:27,539::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.112241 s, 9.1 MB/s\n'; <rc> = 0 >Thread-15913667::INFO::2013-11-20 03:10:28,200::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913667::DEBUG::2013-11-20 03:10:28,200::task::495::TaskManager.Task::(_debug) Task 46fc00d6-dddb-46fb-9e6e-de1f19c82cd5: moving from state init -> state preparing >Thread-15913667::DEBUG::2013-11-20 03:10:28,201::task::495::TaskManager.Task::(_debug) Task 46fc00d6-dddb-46fb-9e6e-de1f19c82cd5: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000663995742798', 'lastCheck': 1384917022.0669079, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00814414024353', 'lastCheck': 1384917019.9834399, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0175859928131', 'lastCheck': 1384917021.095197, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00330805778503', 'lastCheck': 1384917022.0647261, 'valid': True, 'code': 0}} >Thread-15913667::DEBUG::2013-11-20 03:10:28,202::task::495::TaskManager.Task::(_debug) Task 46fc00d6-dddb-46fb-9e6e-de1f19c82cd5: moving from state preparing -> state finished >Thread-15913667::DEBUG::2013-11-20 03:10:28,202::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913667::DEBUG::2013-11-20 03:10:28,203::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913667::DEBUG::2013-11-20 03:10:28,204::task::495::TaskManager.Task::(_debug) Task 46fc00d6-dddb-46fb-9e6e-de1f19c82cd5: ref 0 aborting False >Thread-15913667::INFO::2013-11-20 03:10:28,204::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000663995742798', 'lastCheck': 1384917022.0669079, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00814414024353', 'lastCheck': 1384917019.9834399, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0175859928131', 'lastCheck': 1384917021.095197, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00330805778503', 'lastCheck': 1384917022.0647261, 'valid': True, 'code': 0}} >Thread-15913668::DEBUG::2013-11-20 03:10:28,231::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::CPU stats not available >Thread-15913668::DEBUG::2013-11-20 03:10:28,231::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk stats not available >Thread-15913668::DEBUG::2013-11-20 03:10:28,232::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk latency not available >Thread-15913668::DEBUG::2013-11-20 03:10:28,232::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk stats not available >Thread-15913668::DEBUG::2013-11-20 03:10:28,233::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk latency not available >Thread-15913635::DEBUG::2013-11-20 03:10:29,112::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913635::DEBUG::2013-11-20 03:10:29,113::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913635::DEBUG::2013-11-20 03:10:29,114::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913635::DEBUG::2013-11-20 03:10:29,114::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.ff5e3845-0d0d-4037-a0de-a3f584a4aef5`ReqID=`1fa2644d-3391-4ad2-a3b1-c2ae45b4c5bc`::Request was made in '/usr/share/vdsm/storage/resourceFactories.py' line '149' at '__getResourceCandidatesList' >Thread-15913635::DEBUG::2013-11-20 03:10:29,115::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.ff5e3845-0d0d-4037-a0de-a3f584a4aef5' for lock type 'exclusive' >Thread-15913635::DEBUG::2013-11-20 03:10:29,115::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.ff5e3845-0d0d-4037-a0de-a3f584a4aef5' is free. Now locking as 'exclusive' (1 active user) >Thread-15913635::DEBUG::2013-11-20 03:10:29,116::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.ff5e3845-0d0d-4037-a0de-a3f584a4aef5`ReqID=`1fa2644d-3391-4ad2-a3b1-c2ae45b4c5bc`::Granted request >Thread-15913635::DEBUG::2013-11-20 03:10:29,116::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.cc863eeb-33db-4227-b4eb-9f4347299cf3' is free. Now locking as 'exclusive' (1 active user) >Thread-15913635::DEBUG::2013-11-20 03:10:29,117::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.cc863eeb-33db-4227-b4eb-9f4347299cf3`ReqID=`7225744f-a9b1-48fe-81ea-26e74febb937`::Granted request >Thread-15913635::DEBUG::2013-11-20 03:10:29,118::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913635::DEBUG::2013-11-20 03:10:29,119::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:10:29,561::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:10:29,992::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0515127 s, 19.9 MB/s\n'; <rc> = 0 >Thread-15913635::DEBUG::2013-11-20 03:10:30,424::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913635::DEBUG::2013-11-20 03:10:30,688::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913635::DEBUG::2013-11-20 03:10:30,690::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --refresh 6b892067-ac7e-4039-a7be-5b7387525b16/ff5e3845-0d0d-4037-a0de-a3f584a4aef5' (cwd None) >Thread-15913635::DEBUG::2013-11-20 03:10:31,352::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913635::DEBUG::2013-11-20 03:10:31,353::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913635::DEBUG::2013-11-20 03:10:31,354::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913635::INFO::2013-11-20 03:10:31,354::volume::517::Storage.Volume::(prepare) Volume: preparing volume 6b892067-ac7e-4039-a7be-5b7387525b16/ff5e3845-0d0d-4037-a0de-a3f584a4aef5 >Thread-15913635::DEBUG::2013-11-20 03:10:31,354::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913635::DEBUG::2013-11-20 03:10:31,355::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15913670::DEBUG::2013-11-20 03:10:31,584::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4760db3e-d3a8-4045-9153-12c36ec16ba2',) {} >Thread-15913670::DEBUG::2013-11-20 03:10:31,585::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk stats not available >Thread-15913670::DEBUG::2013-11-20 03:10:31,585::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk latency not available >Thread-15913670::DEBUG::2013-11-20 03:10:31,586::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '49279', 'displayIp': '0', 'displayPort': u'5906', 'session': 'Unknown', 'displaySecurePort': u'5907', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet4': {'macAddr': u'00:1a:4a:a8:64:04', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet4'}}, 'vmId': '4760db3e-d3a8-4045-9153-12c36ec16ba2', 'monitorResponse': '0', 'cpuUser': '292.93', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '85899345920', 'writeLatency': '0', 'imageID': '6f97dfa5-ee35-4f37-8495-1e6b3136b92c', 'flushLatency': '0', 'truesize': '85899345920'}}, 'boot': 'cd', 'statsAge': '0.06', 'cpuIdle': '0.00', 'elapsedTime': '44', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913671::DEBUG::2013-11-20 03:10:31,591::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('0dc7e997-e884-4493-b701-790fb37b3d80',) {} >Thread-15913671::DEBUG::2013-11-20 03:10:31,591::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::CPU stats not available >Thread-15913671::DEBUG::2013-11-20 03:10:31,592::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk stats not available >Thread-15913671::DEBUG::2013-11-20 03:10:31,592::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk latency not available >Thread-15913671::DEBUG::2013-11-20 03:10:31,593::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '49674', 'displayIp': '0', 'displayPort': u'5908', 'session': 'Unknown', 'displaySecurePort': u'5909', 'timeOffset': '3600', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:03', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'vmId': '0dc7e997-e884-4493-b701-790fb37b3d80', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '85899345920', 'writeLatency': '0', 'imageID': 'b74dd33b-748d-48df-83f2-ebcc3523e117', 'flushLatency': '0', 'truesize': '85899345920'}}, 'boot': 'cd', 'statsAge': '0.74', 'cpuIdle': '100.00', 'elapsedTime': '39', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913672::DEBUG::2013-11-20 03:10:31,746::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913672::INFO::2013-11-20 03:10:31,747::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913672::DEBUG::2013-11-20 03:10:31,747::task::495::TaskManager.Task::(_debug) Task e3952903-b5c1-41c0-b8b8-747b3be9432e: moving from state init -> state preparing >Thread-15913672::DEBUG::2013-11-20 03:10:31,748::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913672::DEBUG::2013-11-20 03:10:31,748::task::495::TaskManager.Task::(_debug) Task e3952903-b5c1-41c0-b8b8-747b3be9432e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913672::DEBUG::2013-11-20 03:10:31,748::task::495::TaskManager.Task::(_debug) Task e3952903-b5c1-41c0-b8b8-747b3be9432e: moving from state preparing -> state finished >Thread-15913672::DEBUG::2013-11-20 03:10:31,749::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913672::DEBUG::2013-11-20 03:10:31,749::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913672::DEBUG::2013-11-20 03:10:31,749::task::495::TaskManager.Task::(_debug) Task e3952903-b5c1-41c0-b8b8-747b3be9432e: ref 0 aborting False >Thread-15913672::INFO::2013-11-20 03:10:31,750::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913673::DEBUG::2013-11-20 03:10:31,754::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913673::INFO::2013-11-20 03:10:31,754::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913673::DEBUG::2013-11-20 03:10:31,755::task::495::TaskManager.Task::(_debug) Task b147d4d9-00f1-48f6-b5d8-c66f3fec224d: moving from state init -> state preparing >Thread-15913673::DEBUG::2013-11-20 03:10:31,756::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`28f9d381-8bce-42be-a68e-06b38b3d4812`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913673::DEBUG::2013-11-20 03:10:31,756::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913673::DEBUG::2013-11-20 03:10:31,757::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913673::DEBUG::2013-11-20 03:10:31,757::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`28f9d381-8bce-42be-a68e-06b38b3d4812`::Granted request >Thread-15913673::DEBUG::2013-11-20 03:10:31,758::task::495::TaskManager.Task::(_debug) Task b147d4d9-00f1-48f6-b5d8-c66f3fec224d: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913673::DEBUG::2013-11-20 03:10:31,758::task::495::TaskManager.Task::(_debug) Task b147d4d9-00f1-48f6-b5d8-c66f3fec224d: ref 1 aborting False >Thread-15913673::WARNING::2013-11-20 03:10:31,760::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913673::DEBUG::2013-11-20 03:10:31,761::task::495::TaskManager.Task::(_debug) Task b147d4d9-00f1-48f6-b5d8-c66f3fec224d: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913673::DEBUG::2013-11-20 03:10:31,761::task::495::TaskManager.Task::(_debug) Task b147d4d9-00f1-48f6-b5d8-c66f3fec224d: moving from state preparing -> state finished >Thread-15913673::DEBUG::2013-11-20 03:10:31,762::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913673::DEBUG::2013-11-20 03:10:31,762::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913673::DEBUG::2013-11-20 03:10:31,763::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913673::DEBUG::2013-11-20 03:10:31,763::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913673::DEBUG::2013-11-20 03:10:31,763::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913673::DEBUG::2013-11-20 03:10:31,764::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913673::DEBUG::2013-11-20 03:10:31,765::task::495::TaskManager.Task::(_debug) Task b147d4d9-00f1-48f6-b5d8-c66f3fec224d: ref 0 aborting False >Thread-15913673::INFO::2013-11-20 03:10:31,766::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913635::DEBUG::2013-11-20 03:10:31,899::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:10:32,004::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913635::DEBUG::2013-11-20 03:10:32,126::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913635::DEBUG::2013-11-20 03:10:32,309::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=23 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15912392::INFO::2013-11-20 03:10:32,320::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:10:32,490::task::495::TaskManager.Task::(_debug) Task 68abad21-304e-44e1-a883-e695f695a361: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:10:32,494::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`beca5588-0ce9-48c5-b9e8-b2a480e0a792`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:10:32,494::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:10:32,495::resourceManager::481::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' found in shared state and queue is empty, Joining current shared lock (2 active users) >Thread-15912392::DEBUG::2013-11-20 03:10:32,496::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`beca5588-0ce9-48c5-b9e8-b2a480e0a792`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:10:32,497::task::495::TaskManager.Task::(_debug) Task 68abad21-304e-44e1-a883-e695f695a361: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:10:32,497::task::495::TaskManager.Task::(_debug) Task 68abad21-304e-44e1-a883-e695f695a361: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:10:32,498::task::495::TaskManager.Task::(_debug) Task 68abad21-304e-44e1-a883-e695f695a361: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:10:32,499::task::495::TaskManager.Task::(_debug) Task 68abad21-304e-44e1-a883-e695f695a361: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:10:32,499::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:10:32,500::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:10:32,501::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:10:32,501::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (1 active users) >Thread-15912392::DEBUG::2013-11-20 03:10:32,502::task::495::TaskManager.Task::(_debug) Task 68abad21-304e-44e1-a883-e695f695a361: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:10:32,503::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913635::DEBUG::2013-11-20 03:10:32,664::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000191607 s, 2.7 MB/s\n'; <rc> = 0 >Thread-15913635::DEBUG::2013-11-20 03:10:32,664::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000191607 s, 2.7 MB/s'], size: 512 >Thread-15913635::DEBUG::2013-11-20 03:10:32,665::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=23 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15913518::INFO::2013-11-20 03:10:32,894::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:10:32,895::task::495::TaskManager.Task::(_debug) Task a887684d-d0f5-477d-9c9b-4aaec383c496: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:10:32,897::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b368d73f-0dbe-4f29-b2c3-8eedda2aac68`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:10:32,898::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:10:32,898::resourceManager::481::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' found in shared state and queue is empty, Joining current shared lock (2 active users) >Thread-15913518::DEBUG::2013-11-20 03:10:32,899::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b368d73f-0dbe-4f29-b2c3-8eedda2aac68`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:10:32,900::task::495::TaskManager.Task::(_debug) Task a887684d-d0f5-477d-9c9b-4aaec383c496: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:10:32,900::task::495::TaskManager.Task::(_debug) Task a887684d-d0f5-477d-9c9b-4aaec383c496: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:10:32,901::task::495::TaskManager.Task::(_debug) Task a887684d-d0f5-477d-9c9b-4aaec383c496: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:10:32,901::task::495::TaskManager.Task::(_debug) Task a887684d-d0f5-477d-9c9b-4aaec383c496: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:10:32,901::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:10:32,902::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:10:32,902::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:10:32,903::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (1 active users) >Thread-15913518::DEBUG::2013-11-20 03:10:32,903::task::495::TaskManager.Task::(_debug) Task a887684d-d0f5-477d-9c9b-4aaec383c496: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:10:32,904::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:10:32,904::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:10:32,905::task::495::TaskManager.Task::(_debug) Task 68e3ab14-cf4e-4148-a793-af3e8181b17a: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:10:32,905::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f86b241d-bf72-4873-84c5-b9e0038a2fb4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:10:32,906::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:10:32,906::resourceManager::481::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' found in shared state and queue is empty, Joining current shared lock (2 active users) >Thread-15913518::DEBUG::2013-11-20 03:10:32,906::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f86b241d-bf72-4873-84c5-b9e0038a2fb4`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:10:32,907::task::495::TaskManager.Task::(_debug) Task 68e3ab14-cf4e-4148-a793-af3e8181b17a: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:10:32,907::task::495::TaskManager.Task::(_debug) Task 68e3ab14-cf4e-4148-a793-af3e8181b17a: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:10:32,908::task::495::TaskManager.Task::(_debug) Task 68e3ab14-cf4e-4148-a793-af3e8181b17a: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:10:32,908::task::495::TaskManager.Task::(_debug) Task 68e3ab14-cf4e-4148-a793-af3e8181b17a: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:10:32,909::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:10:32,909::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:10:32,909::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:10:32,910::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (1 active users) >Thread-15913518::DEBUG::2013-11-20 03:10:32,910::task::495::TaskManager.Task::(_debug) Task 68e3ab14-cf4e-4148-a793-af3e8181b17a: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:10:32,911::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:10:33,043::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0633458 s, 16.2 MB/s\n'; <rc> = 0 >Thread-15913635::DEBUG::2013-11-20 03:10:33,185::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000138831 s, 3.7 MB/s\n'; <rc> = 0 >Thread-15913635::DEBUG::2013-11-20 03:10:33,186::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000138831 s, 3.7 MB/s'], size: 512 >Thread-15913635::DEBUG::2013-11-20 03:10:33,187::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.ff5e3845-0d0d-4037-a0de-a3f584a4aef5`ReqID=`9b75a5f6-83da-4282-95c0-855f1457aa30`::Request was made in '/usr/share/vdsm/storage/volume.py' line '533' at 'prepare' >Thread-15913635::DEBUG::2013-11-20 03:10:33,187::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.ff5e3845-0d0d-4037-a0de-a3f584a4aef5' for lock type 'exclusive' >Thread-15913635::DEBUG::2013-11-20 03:10:33,188::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.ff5e3845-0d0d-4037-a0de-a3f584a4aef5' is free. Now locking as 'exclusive' (1 active user) >Thread-15913635::DEBUG::2013-11-20 03:10:33,188::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.ff5e3845-0d0d-4037-a0de-a3f584a4aef5`ReqID=`9b75a5f6-83da-4282-95c0-855f1457aa30`::Granted request >Thread-15913635::DEBUG::2013-11-20 03:10:33,189::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=23 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15913635::DEBUG::2013-11-20 03:10:34,456::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000151006 s, 3.4 MB/s\n'; <rc> = 0 >Thread-15913635::DEBUG::2013-11-20 03:10:34,457::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000151006 s, 3.4 MB/s'], size: 512 >Thread-15913635::DEBUG::2013-11-20 03:10:34,458::task::495::TaskManager.Task::(_debug) Task 548931ca-5754-4a34-a2c3-0c27b288a873: finished: None >Thread-15913635::DEBUG::2013-11-20 03:10:34,458::task::495::TaskManager.Task::(_debug) Task 548931ca-5754-4a34-a2c3-0c27b288a873: moving from state preparing -> state finished >Thread-15913635::DEBUG::2013-11-20 03:10:34,459::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913635::DEBUG::2013-11-20 03:10:34,459::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913635::DEBUG::2013-11-20 03:10:34,460::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913635::DEBUG::2013-11-20 03:10:34,460::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913635::DEBUG::2013-11-20 03:10:34,460::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913635::DEBUG::2013-11-20 03:10:34,461::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913635::DEBUG::2013-11-20 03:10:34,461::task::495::TaskManager.Task::(_debug) Task 548931ca-5754-4a34-a2c3-0c27b288a873: ref 0 aborting False >Thread-15913635::INFO::2013-11-20 03:10:34,462::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15913635::INFO::2013-11-20 03:10:34,462::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913635::DEBUG::2013-11-20 03:10:34,463::task::495::TaskManager.Task::(_debug) Task c061563d-0e3d-4316-adbd-b8ba55de8659: moving from state init -> state preparing >Thread-15913635::DEBUG::2013-11-20 03:10:34,463::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`77a8483d-1a24-40e0-956a-7257bb62e175`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1666' at 'public_getVolumePath' >Thread-15913635::DEBUG::2013-11-20 03:10:34,464::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913635::DEBUG::2013-11-20 03:10:34,464::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913635::DEBUG::2013-11-20 03:10:34,465::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`77a8483d-1a24-40e0-956a-7257bb62e175`::Granted request >Thread-15913635::DEBUG::2013-11-20 03:10:34,465::task::495::TaskManager.Task::(_debug) Task c061563d-0e3d-4316-adbd-b8ba55de8659: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913635::DEBUG::2013-11-20 03:10:34,466::task::495::TaskManager.Task::(_debug) Task c061563d-0e3d-4316-adbd-b8ba55de8659: ref 1 aborting False >Thread-15913635::DEBUG::2013-11-20 03:10:34,466::task::495::TaskManager.Task::(_debug) Task c061563d-0e3d-4316-adbd-b8ba55de8659: finished: {'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/cc863eeb-33db-4227-b4eb-9f4347299cf3/ff5e3845-0d0d-4037-a0de-a3f584a4aef5'} >Thread-15913635::DEBUG::2013-11-20 03:10:34,467::task::495::TaskManager.Task::(_debug) Task c061563d-0e3d-4316-adbd-b8ba55de8659: moving from state preparing -> state finished >Thread-15913635::DEBUG::2013-11-20 03:10:34,467::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913635::DEBUG::2013-11-20 03:10:34,467::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913635::DEBUG::2013-11-20 03:10:34,468::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913635::DEBUG::2013-11-20 03:10:34,468::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913635::DEBUG::2013-11-20 03:10:34,469::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913635::DEBUG::2013-11-20 03:10:34,469::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913635::DEBUG::2013-11-20 03:10:34,469::task::495::TaskManager.Task::(_debug) Task c061563d-0e3d-4316-adbd-b8ba55de8659: ref 0 aborting False >Thread-15913635::INFO::2013-11-20 03:10:34,470::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, Return response: {'status': {'message': 'OK', 'code': 0}, 'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/cc863eeb-33db-4227-b4eb-9f4347299cf3/ff5e3845-0d0d-4037-a0de-a3f584a4aef5'} >Thread-15913635::INFO::2013-11-20 03:10:34,471::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913635::DEBUG::2013-11-20 03:10:34,472::task::495::TaskManager.Task::(_debug) Task a7e9eb23-6840-474d-a4a4-6e3a71a5044a: moving from state init -> state preparing >Thread-15913635::DEBUG::2013-11-20 03:10:34,473::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`70e974a9-90c7-4235-986c-4053c3f10c23`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913635::DEBUG::2013-11-20 03:10:34,473::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913635::DEBUG::2013-11-20 03:10:34,474::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913635::DEBUG::2013-11-20 03:10:34,474::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`70e974a9-90c7-4235-986c-4053c3f10c23`::Granted request >Thread-15913635::DEBUG::2013-11-20 03:10:34,475::task::495::TaskManager.Task::(_debug) Task a7e9eb23-6840-474d-a4a4-6e3a71a5044a: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913635::DEBUG::2013-11-20 03:10:34,475::task::495::TaskManager.Task::(_debug) Task a7e9eb23-6840-474d-a4a4-6e3a71a5044a: ref 1 aborting False >Thread-15913635::DEBUG::2013-11-20 03:10:34,475::task::495::TaskManager.Task::(_debug) Task a7e9eb23-6840-474d-a4a4-6e3a71a5044a: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913635::DEBUG::2013-11-20 03:10:34,476::task::495::TaskManager.Task::(_debug) Task a7e9eb23-6840-474d-a4a4-6e3a71a5044a: moving from state preparing -> state finished >Thread-15913635::DEBUG::2013-11-20 03:10:34,476::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913635::DEBUG::2013-11-20 03:10:34,476::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913635::DEBUG::2013-11-20 03:10:34,477::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913635::DEBUG::2013-11-20 03:10:34,477::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913635::DEBUG::2013-11-20 03:10:34,477::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913635::DEBUG::2013-11-20 03:10:34,478::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913635::DEBUG::2013-11-20 03:10:34,478::task::495::TaskManager.Task::(_debug) Task a7e9eb23-6840-474d-a4a4-6e3a71a5044a: ref 0 aborting False >Thread-15913635::INFO::2013-11-20 03:10:34,478::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913635::INFO::2013-11-20 03:10:34,479::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16) >Thread-15913635::DEBUG::2013-11-20 03:10:34,479::task::495::TaskManager.Task::(_debug) Task cd6890f4-5509-4ba1-b908-c0d148b6db36: moving from state init -> state preparing >Thread-15913635::INFO::2013-11-20 03:10:34,480::blockSD::674::Storage.StorageDomain::(validate) sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 >Thread-15913635::DEBUG::2013-11-20 03:10:34,480::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913635::DEBUG::2013-11-20 03:10:34,481::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913635::DEBUG::2013-11-20 03:10:34,481::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15913635::DEBUG::2013-11-20 03:10:34,482::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15913635::DEBUG::2013-11-20 03:10:34,482::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15913635::DEBUG::2013-11-20 03:10:34,483::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:10:35,963::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913675::DEBUG::2013-11-20 03:10:35,993::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4760db3e-d3a8-4045-9153-12c36ec16ba2',) {} >Thread-15913675::DEBUG::2013-11-20 03:10:35,999::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk stats not available >Thread-15913675::DEBUG::2013-11-20 03:10:36,194::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk latency not available >Thread-15913675::DEBUG::2013-11-20 03:10:36,196::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '49279', 'displayIp': '0', 'displayPort': u'5906', 'session': 'Unknown', 'displaySecurePort': u'5907', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet4': {'macAddr': u'00:1a:4a:a8:64:04', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet4'}}, 'vmId': '4760db3e-d3a8-4045-9153-12c36ec16ba2', 'monitorResponse': '0', 'cpuUser': '292.93', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '85899345920', 'writeLatency': '0', 'imageID': '6f97dfa5-ee35-4f37-8495-1e6b3136b92c', 'flushLatency': '0', 'truesize': '85899345920'}}, 'boot': 'cd', 'statsAge': '0.04', 'cpuIdle': '0.00', 'elapsedTime': '48', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913676::DEBUG::2013-11-20 03:10:36,203::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('0dc7e997-e884-4493-b701-790fb37b3d80',) {} >Thread-15913676::DEBUG::2013-11-20 03:10:36,204::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::CPU stats not available >Thread-15913676::DEBUG::2013-11-20 03:10:36,204::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk stats not available >Thread-15913676::DEBUG::2013-11-20 03:10:36,205::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk latency not available >Thread-15913676::DEBUG::2013-11-20 03:10:36,205::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '49674', 'displayIp': '0', 'displayPort': u'5908', 'session': 'Unknown', 'displaySecurePort': u'5909', 'timeOffset': '3600', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:03', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'vmId': '0dc7e997-e884-4493-b701-790fb37b3d80', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '85899345920', 'writeLatency': '0', 'imageID': 'b74dd33b-748d-48df-83f2-ebcc3523e117', 'flushLatency': '0', 'truesize': '85899345920'}}, 'boot': 'cd', 'statsAge': '0.26', 'cpuIdle': '100.00', 'elapsedTime': '44', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:10:37,648::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0438783 s, 23.3 MB/s\n'; <rc> = 0 >Thread-15913635::DEBUG::2013-11-20 03:10:37,810::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913635::DEBUG::2013-11-20 03:10:37,812::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15913635::DEBUG::2013-11-20 03:10:37,813::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'_SHA_CKSUM=716ea7492d9134a4b13de27ac61a336ad474727c', u'POOL_SPM_ID=2', u'POOL_SPM_LVER=287'] >Thread-15913635::DEBUG::2013-11-20 03:10:37,815::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`16ed6b6d-1659-4db5-a15d-cae793c774d0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1297' at 'public_getStorageDomainInfo' >Thread-15913635::DEBUG::2013-11-20 03:10:37,815::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913635::DEBUG::2013-11-20 03:10:37,816::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913635::DEBUG::2013-11-20 03:10:37,816::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`16ed6b6d-1659-4db5-a15d-cae793c774d0`::Granted request >Thread-15913635::DEBUG::2013-11-20 03:10:37,817::task::495::TaskManager.Task::(_debug) Task cd6890f4-5509-4ba1-b908-c0d148b6db36: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913635::DEBUG::2013-11-20 03:10:37,817::task::495::TaskManager.Task::(_debug) Task cd6890f4-5509-4ba1-b908-c0d148b6db36: ref 1 aborting False >Thread-15913635::WARNING::2013-11-20 03:10:37,819::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913635::DEBUG::2013-11-20 03:10:37,820::task::495::TaskManager.Task::(_debug) Task cd6890f4-5509-4ba1-b908-c0d148b6db36: finished: {'info': {'uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'vguuid': 'aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', 'lver': 287, 'state': 'OK', 'version': '2', 'role': 'Master', 'master_ver': 1, 'spm_id': 2, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'OS'}} >Thread-15913635::DEBUG::2013-11-20 03:10:37,820::task::495::TaskManager.Task::(_debug) Task cd6890f4-5509-4ba1-b908-c0d148b6db36: moving from state preparing -> state finished >Thread-15913635::DEBUG::2013-11-20 03:10:37,821::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913635::DEBUG::2013-11-20 03:10:37,821::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913635::DEBUG::2013-11-20 03:10:37,821::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913635::DEBUG::2013-11-20 03:10:37,822::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913635::DEBUG::2013-11-20 03:10:37,822::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913635::DEBUG::2013-11-20 03:10:37,823::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913635::DEBUG::2013-11-20 03:10:37,823::task::495::TaskManager.Task::(_debug) Task cd6890f4-5509-4ba1-b908-c0d148b6db36: ref 0 aborting False >Thread-15913635::INFO::2013-11-20 03:10:37,824::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'vguuid': 'aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', 'lver': 287, 'state': 'OK', 'version': '2', 'role': 'Master', 'master_ver': 1, 'spm_id': 2, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'OS'}} >Thread-15913635::DEBUG::2013-11-20 03:10:37,835::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/libexec/vdsm/hooks/before_vm_start/50_vhostmd' (cwd None) >Thread-15913635::DEBUG::2013-11-20 03:10:38,329::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913635::INFO::2013-11-20 03:10:38,330::hooks::51::root::(_runHooksDir) >Thread-15913635::DEBUG::2013-11-20 03:10:38,337::libvirtvm::1077::vm.Vm::(_run) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::<?xml version="1.0" encoding="utf-8"?> ><domain type="kvm"> > <name>KLEE-Print01</name> > <uuid>11b5c641-e4f7-487e-9758-c42cb014e2d2</uuid> > <memory>2097152</memory> > <currentMemory>2097152</currentMemory> > <vcpu>2</vcpu> > <devices> > <disk device="disk" type="block"> > <source dev="/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/cc863eeb-33db-4227-b4eb-9f4347299cf3/ff5e3845-0d0d-4037-a0de-a3f584a4aef5"/> > <target bus="virtio" dev="vda"/> > <serial>27-b4eb-9f4347299cf3</serial> > <driver cache="none" error_policy="stop" io="native" name="qemu" type="qcow2"/> > </disk> > <disk device="cdrom" type="file"> > <source file="/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO"/> > <target bus="ide" dev="hdc"/> > </disk> > <controller index="0" ports="16" type="virtio-serial"/> > <channel type="unix"> > <target name="com.redhat.rhevm.vdsm" type="virtio"/> > <source mode="bind" path="/var/lib/libvirt/qemu/channels/KLEE-Print01.com.redhat.rhevm.vdsm"/> > </channel> > <memballoon model="none"/> > <interface type="bridge"> > <mac address="00:1a:4a:a8:64:0d"/> > <model type="virtio"/> > <source bridge="rhevm"/> > </interface> > <input bus="ps2" type="mouse"/> > <video> > <model heads="1" type="qxl" vram="65536"/> > </video> > <channel type="spicevmc"> > <target name="com.redhat.spice.0" type="virtio"/> > </channel> > <graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice"> > <channel mode="secure" name="main"/> > <channel mode="secure" name="inputs"/> > </graphics> > </devices> > <os> > <type arch="x86_64" machine="rhel6.2.0">hvm</type> > <boot dev="cdrom"/> > <smbios mode="sysinfo"/> > </os> > <sysinfo type="smbios"> > <system> > <entry name="manufacturer">Red Hat</entry> > <entry name="product">RHEV Hypervisor</entry> > <entry name="version">6.3-20121012.0.el6_3</entry> > <entry name="serial">30343536-3138-5A43-4A32-333630373442_AC:16:2D:76:EB:F0</entry> > <entry name="uuid">11b5c641-e4f7-487e-9758-c42cb014e2d2</entry> > </system> > </sysinfo> > <clock adjustment="-1" offset="variable"> > <timer name="rtc" tickpolicy="catchup"/> > </clock> > <features> > <acpi/> > </features> > <cpu match="exact"> > <model>Westmere</model> > <topology cores="1" sockets="2" threads="1"/> > </cpu> ></domain> > >Thread-15913678::DEBUG::2013-11-20 03:10:39,228::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4760db3e-d3a8-4045-9153-12c36ec16ba2',) {} >Thread-15913678::DEBUG::2013-11-20 03:10:39,229::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk stats not available >Thread-15913678::DEBUG::2013-11-20 03:10:39,229::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk latency not available >Thread-15913678::DEBUG::2013-11-20 03:10:39,230::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '49279', 'displayIp': '0', 'displayPort': u'5906', 'session': 'Unknown', 'displaySecurePort': u'5907', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet4': {'macAddr': u'00:1a:4a:a8:64:04', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet4'}}, 'vmId': '4760db3e-d3a8-4045-9153-12c36ec16ba2', 'monitorResponse': '0', 'cpuUser': '403.48', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '85899345920', 'writeLatency': '0', 'imageID': '6f97dfa5-ee35-4f37-8495-1e6b3136b92c', 'flushLatency': '0', 'truesize': '85899345920'}}, 'boot': 'cd', 'statsAge': '1.20', 'cpuIdle': '0.00', 'elapsedTime': '51', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913679::DEBUG::2013-11-20 03:10:39,234::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('0dc7e997-e884-4493-b701-790fb37b3d80',) {} >Thread-15913679::DEBUG::2013-11-20 03:10:39,235::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::CPU stats not available >Thread-15913679::DEBUG::2013-11-20 03:10:39,235::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk stats not available >Thread-15913679::DEBUG::2013-11-20 03:10:39,235::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk latency not available >Thread-15913679::DEBUG::2013-11-20 03:10:39,236::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '49674', 'displayIp': '0', 'displayPort': u'5908', 'session': 'Unknown', 'displaySecurePort': u'5909', 'timeOffset': '3600', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:03', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'vmId': '0dc7e997-e884-4493-b701-790fb37b3d80', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '85899345920', 'writeLatency': '0', 'imageID': 'b74dd33b-748d-48df-83f2-ebcc3523e117', 'flushLatency': '0', 'truesize': '85899345920'}}, 'boot': 'cd', 'statsAge': '0.21', 'cpuIdle': '100.00', 'elapsedTime': '47', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:10:39,657::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:10:40,114::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0474614 s, 21.6 MB/s\n'; <rc> = 0 >Thread-15913680::DEBUG::2013-11-20 03:10:42,060::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913680::INFO::2013-11-20 03:10:42,061::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913680::DEBUG::2013-11-20 03:10:42,062::task::495::TaskManager.Task::(_debug) Task 17ccea81-c0df-4296-9841-dc4c98705ce2: moving from state init -> state preparing >Thread-15913680::DEBUG::2013-11-20 03:10:42,062::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913680::DEBUG::2013-11-20 03:10:42,063::task::495::TaskManager.Task::(_debug) Task 17ccea81-c0df-4296-9841-dc4c98705ce2: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913680::DEBUG::2013-11-20 03:10:42,063::task::495::TaskManager.Task::(_debug) Task 17ccea81-c0df-4296-9841-dc4c98705ce2: moving from state preparing -> state finished >Thread-15913680::DEBUG::2013-11-20 03:10:42,063::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913680::DEBUG::2013-11-20 03:10:42,064::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913680::DEBUG::2013-11-20 03:10:42,064::task::495::TaskManager.Task::(_debug) Task 17ccea81-c0df-4296-9841-dc4c98705ce2: ref 0 aborting False >Thread-15913680::INFO::2013-11-20 03:10:42,065::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913681::DEBUG::2013-11-20 03:10:42,068::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913681::INFO::2013-11-20 03:10:42,069::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913681::DEBUG::2013-11-20 03:10:42,069::task::495::TaskManager.Task::(_debug) Task 496be271-d34a-4515-9d3e-09649851cbc0: moving from state init -> state preparing >Thread-15913681::DEBUG::2013-11-20 03:10:42,070::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a669af6a-45da-4402-970b-4165a35f7bf1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913681::DEBUG::2013-11-20 03:10:42,071::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913681::DEBUG::2013-11-20 03:10:42,071::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913681::DEBUG::2013-11-20 03:10:42,072::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a669af6a-45da-4402-970b-4165a35f7bf1`::Granted request >Thread-15913681::DEBUG::2013-11-20 03:10:42,072::task::495::TaskManager.Task::(_debug) Task 496be271-d34a-4515-9d3e-09649851cbc0: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913681::DEBUG::2013-11-20 03:10:42,073::task::495::TaskManager.Task::(_debug) Task 496be271-d34a-4515-9d3e-09649851cbc0: ref 1 aborting False >Thread-15913681::WARNING::2013-11-20 03:10:42,074::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913681::DEBUG::2013-11-20 03:10:42,075::task::495::TaskManager.Task::(_debug) Task 496be271-d34a-4515-9d3e-09649851cbc0: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913681::DEBUG::2013-11-20 03:10:42,076::task::495::TaskManager.Task::(_debug) Task 496be271-d34a-4515-9d3e-09649851cbc0: moving from state preparing -> state finished >Thread-15913681::DEBUG::2013-11-20 03:10:42,076::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913681::DEBUG::2013-11-20 03:10:42,076::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913681::DEBUG::2013-11-20 03:10:42,077::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913681::DEBUG::2013-11-20 03:10:42,077::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913681::DEBUG::2013-11-20 03:10:42,078::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913681::DEBUG::2013-11-20 03:10:42,078::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913681::DEBUG::2013-11-20 03:10:42,079::task::495::TaskManager.Task::(_debug) Task 496be271-d34a-4515-9d3e-09649851cbc0: ref 0 aborting False >Thread-15913681::INFO::2013-11-20 03:10:42,079::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:10:42,124::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913683::DEBUG::2013-11-20 03:10:42,313::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4760db3e-d3a8-4045-9153-12c36ec16ba2',) {} >Thread-15913683::DEBUG::2013-11-20 03:10:42,315::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk stats not available >Thread-15913683::DEBUG::2013-11-20 03:10:42,316::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk latency not available >Thread-15913683::DEBUG::2013-11-20 03:10:42,317::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '49279', 'displayIp': '0', 'displayPort': u'5906', 'session': 'Unknown', 'displaySecurePort': u'5907', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet4': {'macAddr': u'00:1a:4a:a8:64:04', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet4'}}, 'vmId': '4760db3e-d3a8-4045-9153-12c36ec16ba2', 'monitorResponse': '0', 'cpuUser': '403.48', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '85899345920', 'writeLatency': '0', 'imageID': '6f97dfa5-ee35-4f37-8495-1e6b3136b92c', 'flushLatency': '0', 'truesize': '85899345920'}}, 'boot': 'cd', 'statsAge': '0.29', 'cpuIdle': '0.00', 'elapsedTime': '54', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913684::DEBUG::2013-11-20 03:10:42,322::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('0dc7e997-e884-4493-b701-790fb37b3d80',) {} >Thread-15913684::DEBUG::2013-11-20 03:10:42,323::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk stats not available >Thread-15913684::DEBUG::2013-11-20 03:10:42,323::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk latency not available >Thread-15913684::DEBUG::2013-11-20 03:10:42,324::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '49674', 'displayIp': '0', 'displayPort': u'5908', 'session': 'Unknown', 'displaySecurePort': u'5909', 'timeOffset': '3600', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:03', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'vmId': '0dc7e997-e884-4493-b701-790fb37b3d80', 'monitorResponse': '0', 'cpuUser': '142.45', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '85899345920', 'writeLatency': '0', 'imageID': 'b74dd33b-748d-48df-83f2-ebcc3523e117', 'flushLatency': '0', 'truesize': '85899345920'}}, 'boot': 'cd', 'statsAge': '3.30', 'cpuIdle': '0.00', 'elapsedTime': '50', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:10:42,806::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0491502 s, 20.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:10:44,817::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913685::INFO::2013-11-20 03:10:45,343::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913685::DEBUG::2013-11-20 03:10:45,344::task::495::TaskManager.Task::(_debug) Task 288edaed-2936-4af9-8d53-3672328b9302: moving from state init -> state preparing >Thread-15913685::DEBUG::2013-11-20 03:10:45,345::task::495::TaskManager.Task::(_debug) Task 288edaed-2936-4af9-8d53-3672328b9302: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000231981277466', 'lastCheck': 1384917044.0317581, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00153398513794', 'lastCheck': 1384917042.032937, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00118613243103', 'lastCheck': 1384917043.0324719, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000571012496948', 'lastCheck': 1384917044.0299499, 'valid': True, 'code': 0}} >Thread-15913685::DEBUG::2013-11-20 03:10:45,345::task::495::TaskManager.Task::(_debug) Task 288edaed-2936-4af9-8d53-3672328b9302: moving from state preparing -> state finished >Thread-15913685::DEBUG::2013-11-20 03:10:45,346::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913685::DEBUG::2013-11-20 03:10:45,347::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913685::DEBUG::2013-11-20 03:10:45,347::task::495::TaskManager.Task::(_debug) Task 288edaed-2936-4af9-8d53-3672328b9302: ref 0 aborting False >Thread-15913685::INFO::2013-11-20 03:10:45,348::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000231981277466', 'lastCheck': 1384917044.0317581, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00153398513794', 'lastCheck': 1384917042.032937, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00118613243103', 'lastCheck': 1384917043.0324719, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000571012496948', 'lastCheck': 1384917044.0299499, 'valid': True, 'code': 0}} >Thread-15913686::DEBUG::2013-11-20 03:10:45,362::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk stats not available >Thread-15913686::DEBUG::2013-11-20 03:10:45,363::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk latency not available >Thread-15913686::DEBUG::2013-11-20 03:10:45,363::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk stats not available >Thread-15913686::DEBUG::2013-11-20 03:10:45,364::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk latency not available >Dummy-15912651::DEBUG::2013-11-20 03:10:45,423::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.074653 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:10:47,434::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:10:47,842::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.058134 s, 17.6 MB/s\n'; <rc> = 0 >Thread-15913688::DEBUG::2013-11-20 03:10:48,887::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('4760db3e-d3a8-4045-9153-12c36ec16ba2',) {} >Thread-15913688::DEBUG::2013-11-20 03:10:48,888::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk stats not available >Thread-15913688::DEBUG::2013-11-20 03:10:48,888::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk latency not available >Thread-15913688::DEBUG::2013-11-20 03:10:48,889::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '49279', 'displayIp': '0', 'displayPort': u'5906', 'session': 'Unknown', 'displaySecurePort': u'5907', 'timeOffset': '-1', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet4': {'macAddr': u'00:1a:4a:a8:64:04', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet4'}}, 'vmId': '4760db3e-d3a8-4045-9153-12c36ec16ba2', 'monitorResponse': '0', 'cpuUser': '403.48', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '85899345920', 'writeLatency': '0', 'imageID': '6f97dfa5-ee35-4f37-8495-1e6b3136b92c', 'flushLatency': '0', 'truesize': '85899345920'}}, 'boot': 'cd', 'statsAge': '1.99', 'cpuIdle': '0.00', 'elapsedTime': '61', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913689::DEBUG::2013-11-20 03:10:48,893::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('0dc7e997-e884-4493-b701-790fb37b3d80',) {} >Thread-15913689::DEBUG::2013-11-20 03:10:48,893::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk stats not available >Thread-15913689::DEBUG::2013-11-20 03:10:48,894::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk latency not available >Thread-15913689::DEBUG::2013-11-20 03:10:48,895::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '49674', 'displayIp': '0', 'displayPort': u'5908', 'session': 'Unknown', 'displaySecurePort': u'5909', 'timeOffset': '3600', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:03', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'vmId': '0dc7e997-e884-4493-b701-790fb37b3d80', 'monitorResponse': '0', 'cpuUser': '142.45', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '85899345920', 'writeLatency': '0', 'imageID': 'b74dd33b-748d-48df-83f2-ebcc3523e117', 'flushLatency': '0', 'truesize': '85899345920'}}, 'boot': 'cd', 'statsAge': '2.00', 'cpuIdle': '0.00', 'elapsedTime': '56', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:10:49,851::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:10:50,289::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0457747 s, 22.4 MB/s\n'; <rc> = 0 >Thread-15913691::DEBUG::2013-11-20 03:10:52,236::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('0dc7e997-e884-4493-b701-790fb37b3d80',) {} >Thread-15913691::DEBUG::2013-11-20 03:10:52,237::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk stats not available >Thread-15913691::DEBUG::2013-11-20 03:10:52,237::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk latency not available >Thread-15913691::DEBUG::2013-11-20 03:10:52,238::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '49674', 'displayIp': '0', 'displayPort': u'5908', 'session': 'Unknown', 'displaySecurePort': u'5909', 'timeOffset': '3600', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet5': {'macAddr': u'00:1a:4a:a8:64:03', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet5'}}, 'vmId': '0dc7e997-e884-4493-b701-790fb37b3d80', 'monitorResponse': '0', 'cpuUser': '142.45', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '85899345920', 'writeLatency': '0', 'imageID': 'b74dd33b-748d-48df-83f2-ebcc3523e117', 'flushLatency': '0', 'truesize': '85899345920'}}, 'boot': 'cd', 'statsAge': '1.34', 'cpuIdle': '0.00', 'elapsedTime': '60', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:10:52,299::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913692::DEBUG::2013-11-20 03:10:52,828::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913692::INFO::2013-11-20 03:10:56,296::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >libvirtEventLoop::DEBUG::2013-11-20 03:10:56,321::libvirtvm::1401::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::event Started detail 0 opaque None >Thread-15913692::DEBUG::2013-11-20 03:10:56,339::task::495::TaskManager.Task::(_debug) Task 5f3bc5b9-8751-451d-bc6a-fdbff25653b8: moving from state init -> state preparing >Thread-15913692::DEBUG::2013-11-20 03:10:56,486::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913692::DEBUG::2013-11-20 03:10:56,492::task::495::TaskManager.Task::(_debug) Task 5f3bc5b9-8751-451d-bc6a-fdbff25653b8: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913692::DEBUG::2013-11-20 03:10:56,494::task::495::TaskManager.Task::(_debug) Task 5f3bc5b9-8751-451d-bc6a-fdbff25653b8: moving from state preparing -> state finished >Thread-15913692::DEBUG::2013-11-20 03:10:56,494::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913692::DEBUG::2013-11-20 03:10:56,495::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913692::DEBUG::2013-11-20 03:10:56,496::task::495::TaskManager.Task::(_debug) Task 5f3bc5b9-8751-451d-bc6a-fdbff25653b8: ref 0 aborting False >Thread-15913692::INFO::2013-11-20 03:10:56,496::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913694::DEBUG::2013-11-20 03:10:56,501::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913694::INFO::2013-11-20 03:10:56,503::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913694::DEBUG::2013-11-20 03:10:56,503::task::495::TaskManager.Task::(_debug) Task 037f9f2a-6d1e-453b-9634-b9e7d672e9c0: moving from state init -> state preparing >Thread-15913694::DEBUG::2013-11-20 03:10:56,504::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6aa110e3-d866-4843-9eea-07ed58e5ad10`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913694::DEBUG::2013-11-20 03:10:56,505::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913694::DEBUG::2013-11-20 03:10:56,506::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913694::DEBUG::2013-11-20 03:10:56,507::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6aa110e3-d866-4843-9eea-07ed58e5ad10`::Granted request >Thread-15913694::DEBUG::2013-11-20 03:10:56,509::task::495::TaskManager.Task::(_debug) Task 037f9f2a-6d1e-453b-9634-b9e7d672e9c0: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913694::DEBUG::2013-11-20 03:10:56,509::task::495::TaskManager.Task::(_debug) Task 037f9f2a-6d1e-453b-9634-b9e7d672e9c0: ref 1 aborting False >Thread-15913694::WARNING::2013-11-20 03:10:56,511::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913694::DEBUG::2013-11-20 03:10:56,512::task::495::TaskManager.Task::(_debug) Task 037f9f2a-6d1e-453b-9634-b9e7d672e9c0: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913694::DEBUG::2013-11-20 03:10:56,513::task::495::TaskManager.Task::(_debug) Task 037f9f2a-6d1e-453b-9634-b9e7d672e9c0: moving from state preparing -> state finished >Thread-15913694::DEBUG::2013-11-20 03:10:56,514::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913694::DEBUG::2013-11-20 03:10:56,514::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913694::DEBUG::2013-11-20 03:10:56,515::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913694::DEBUG::2013-11-20 03:10:56,516::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913694::DEBUG::2013-11-20 03:10:56,517::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913694::DEBUG::2013-11-20 03:10:56,517::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913694::DEBUG::2013-11-20 03:10:56,518::task::495::TaskManager.Task::(_debug) Task 037f9f2a-6d1e-453b-9634-b9e7d672e9c0: ref 0 aborting False >Thread-15913694::INFO::2013-11-20 03:10:56,518::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913635::DEBUG::2013-11-20 03:10:56,617::utils::307::vm.Vm::(start) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::Start statistics collection >Thread-15913695::DEBUG::2013-11-20 03:10:56,620::utils::336::vm.Vm::(run) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::Stats thread started >Thread-15913696::DEBUG::2013-11-20 03:10:56,622::guestIF::47::root::(_connect) Attempting connection to /var/lib/libvirt/qemu/channels/KLEE-Print01.com.redhat.rhevm.vdsm >Thread-15913696::DEBUG::2013-11-20 03:10:56,624::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /usr/share/vdsm/prepare-vmchannel /var/lib/libvirt/qemu/channels/KLEE-Print01.com.redhat.rhevm.vdsm' (cwd None) >Thread-15913635::WARNING::2013-11-20 03:10:56,821::libvirtvm::1124::vm.Vm::(_readPauseCode) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::_readPauseCode unsupported by libvirt vm >Thread-15913635::DEBUG::2013-11-20 03:10:56,824::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /usr/share/vdsm/get-vm-pid KLEE-Print01' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:10:57,052::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.205391 s, 5.0 MB/s\n'; <rc> = 0 >Thread-15913696::DEBUG::2013-11-20 03:10:57,055::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913696::DEBUG::2013-11-20 03:10:57,059::guestIF::53::root::(_connect) connected to /var/lib/libvirt/qemu/channels/KLEE-Print01.com.redhat.rhevm.vdsm >Thread-15913635::DEBUG::2013-11-20 03:10:57,214::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913635::DEBUG::2013-11-20 03:10:57,221::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /sbin/service ksmtuned retune' (cwd None) >Thread-15913695::INFO::2013-11-20 03:10:57,396::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:10:57,398::task::495::TaskManager.Task::(_debug) Task 51582c95-42b7-4322-9e4c-750f6a713988: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:10:57,400::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0f4b4cd8-2966-43a5-915e-ca3a4ed21f9a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:10:57,401::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:10:57,402::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:10:57,402::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0f4b4cd8-2966-43a5-915e-ca3a4ed21f9a`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:10:57,403::task::495::TaskManager.Task::(_debug) Task 51582c95-42b7-4322-9e4c-750f6a713988: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:10:57,404::task::495::TaskManager.Task::(_debug) Task 51582c95-42b7-4322-9e4c-750f6a713988: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:10:57,405::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15913695::DEBUG::2013-11-20 03:10:57,406::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15913635::DEBUG::2013-11-20 03:10:57,761::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913635::DEBUG::2013-11-20 03:10:57,763::vm::358::vm.Vm::(_startUnderlyingVm) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::_ongoingCreations released >Thread-15913695::DEBUG::2013-11-20 03:10:58,319::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913695::DEBUG::2013-11-20 03:10:58,490::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913695::DEBUG::2013-11-20 03:10:58,491::task::495::TaskManager.Task::(_debug) Task 51582c95-42b7-4322-9e4c-750f6a713988: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:10:58,492::task::495::TaskManager.Task::(_debug) Task 51582c95-42b7-4322-9e4c-750f6a713988: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:10:58,492::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:10:58,492::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:10:58,493::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:10:58,493::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:10:58,494::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:10:58,494::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:10:58,494::task::495::TaskManager.Task::(_debug) Task 51582c95-42b7-4322-9e4c-750f6a713988: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:10:58,495::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:10:59,073::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913698::DEBUG::2013-11-20 03:10:59,373::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('11b5c641-e4f7-487e-9758-c42cb014e2d2',) {} >Thread-15913698::DEBUG::2013-11-20 03:10:59,374::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::CPU stats not available >Thread-15913698::DEBUG::2013-11-20 03:10:59,375::libvirtvm::202::vm.Vm::(_getNetworkStats) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::Network stats not available >Thread-15913698::DEBUG::2013-11-20 03:10:59,375::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::Disk stats not available >Thread-15913698::DEBUG::2013-11-20 03:10:59,375::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::Disk latency not available >Thread-15913698::DEBUG::2013-11-20 03:10:59,376::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '50239', 'displayIp': '0', 'displayPort': u'5910', 'session': 'Unknown', 'displaySecurePort': u'5911', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet6': {'state': 'unknown', 'macAddr': u'00:1a:4a:a8:64:0d', 'speed': '1000', 'name': u'vnet6'}}, 'vmId': '11b5c641-e4f7-487e-9758-c42cb014e2d2', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '20401094656', 'writeLatency': '0', 'imageID': 'cc863eeb-33db-4227-b4eb-9f4347299cf3', 'flushLatency': '0', 'truesize': '20401094656'}}, 'boot': 'd', 'statsAge': '2.75', 'cpuIdle': '100.00', 'elapsedTime': '64', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '-1', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:10:59,491::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0488093 s, 21.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:11:01,505::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:11:01,957::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0559277 s, 18.3 MB/s\n'; <rc> = 0 >Thread-15913699::INFO::2013-11-20 03:11:02,496::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913699::DEBUG::2013-11-20 03:11:02,496::task::495::TaskManager.Task::(_debug) Task 8826498a-6dcd-44f8-b291-857dd03a9156: moving from state init -> state preparing >Thread-15913699::DEBUG::2013-11-20 03:11:02,497::task::495::TaskManager.Task::(_debug) Task 8826498a-6dcd-44f8-b291-857dd03a9156: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00118899345398', 'lastCheck': 1384917057.4003839, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000864028930664', 'lastCheck': 1384917052.053179, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00623917579651', 'lastCheck': 1384917056.493603, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00217604637146', 'lastCheck': 1384917057.3941691, 'valid': True, 'code': 0}} >Thread-15913699::DEBUG::2013-11-20 03:11:02,498::task::495::TaskManager.Task::(_debug) Task 8826498a-6dcd-44f8-b291-857dd03a9156: moving from state preparing -> state finished >Thread-15913699::DEBUG::2013-11-20 03:11:02,498::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913699::DEBUG::2013-11-20 03:11:02,499::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913699::DEBUG::2013-11-20 03:11:02,499::task::495::TaskManager.Task::(_debug) Task 8826498a-6dcd-44f8-b291-857dd03a9156: ref 0 aborting False >Thread-15913699::INFO::2013-11-20 03:11:02,500::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00118899345398', 'lastCheck': 1384917057.4003839, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000864028930664', 'lastCheck': 1384917052.053179, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00623917579651', 'lastCheck': 1384917056.493603, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00217604637146', 'lastCheck': 1384917057.3941691, 'valid': True, 'code': 0}} >Thread-15913700::DEBUG::2013-11-20 03:11:02,513::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::CPU stats not available >Thread-15913700::DEBUG::2013-11-20 03:11:02,514::libvirtvm::202::vm.Vm::(_getNetworkStats) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::Network stats not available >Thread-15913700::DEBUG::2013-11-20 03:11:02,514::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::Disk stats not available >Thread-15913700::DEBUG::2013-11-20 03:11:02,515::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::Disk latency not available >Thread-15913700::DEBUG::2013-11-20 03:11:02,516::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk stats not available >Thread-15913700::DEBUG::2013-11-20 03:11:02,517::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk latency not available >Thread-15913700::DEBUG::2013-11-20 03:11:02,518::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk stats not available >Thread-15913700::DEBUG::2013-11-20 03:11:02,518::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::Disk latency not available >Dummy-15912651::DEBUG::2013-11-20 03:11:03,968::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913546::INFO::2013-11-20 03:11:04,349::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:11:04,350::task::495::TaskManager.Task::(_debug) Task f1d94931-68f2-48af-b09d-e6c8a09de24c: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:11:04,351::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`214fbc03-2b6e-481b-80a8-117110ac59cd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:11:04,352::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:11:04,352::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:11:04,353::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`214fbc03-2b6e-481b-80a8-117110ac59cd`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:11:04,353::task::495::TaskManager.Task::(_debug) Task f1d94931-68f2-48af-b09d-e6c8a09de24c: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:11:04,354::task::495::TaskManager.Task::(_debug) Task f1d94931-68f2-48af-b09d-e6c8a09de24c: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:11:04,354::task::495::TaskManager.Task::(_debug) Task f1d94931-68f2-48af-b09d-e6c8a09de24c: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:11:04,355::task::495::TaskManager.Task::(_debug) Task f1d94931-68f2-48af-b09d-e6c8a09de24c: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:11:04,355::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:11:04,355::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:11:04,356::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:11:04,356::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:11:04,357::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:11:04,357::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:11:04,357::task::495::TaskManager.Task::(_debug) Task f1d94931-68f2-48af-b09d-e6c8a09de24c: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:11:04,358::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:11:04,358::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:11:04,359::task::495::TaskManager.Task::(_debug) Task d8916e6e-690b-4078-9fbc-d09f23a03048: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:11:04,360::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`6964572d-5ef5-4899-a909-96cc99098017`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:11:04,360::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:11:04,361::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:11:04,362::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`6964572d-5ef5-4899-a909-96cc99098017`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:11:04,363::task::495::TaskManager.Task::(_debug) Task d8916e6e-690b-4078-9fbc-d09f23a03048: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:11:04,363::task::495::TaskManager.Task::(_debug) Task d8916e6e-690b-4078-9fbc-d09f23a03048: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:11:04,364::task::495::TaskManager.Task::(_debug) Task d8916e6e-690b-4078-9fbc-d09f23a03048: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:11:04,365::task::495::TaskManager.Task::(_debug) Task d8916e6e-690b-4078-9fbc-d09f23a03048: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:11:04,366::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:11:04,366::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:11:04,366::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:11:04,367::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:11:04,367::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:11:04,368::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:11:04,368::task::495::TaskManager.Task::(_debug) Task d8916e6e-690b-4078-9fbc-d09f23a03048: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:11:04,368::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:11:04,369::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:11:04,369::task::495::TaskManager.Task::(_debug) Task 8186739e-33ef-4064-a2e8-3dbd4e195987: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:11:04,370::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`a383ab22-31af-479d-9bb2-a82d512b91f1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:11:04,370::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:11:04,371::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:11:04,371::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`a383ab22-31af-479d-9bb2-a82d512b91f1`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:11:04,372::task::495::TaskManager.Task::(_debug) Task 8186739e-33ef-4064-a2e8-3dbd4e195987: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:11:04,372::task::495::TaskManager.Task::(_debug) Task 8186739e-33ef-4064-a2e8-3dbd4e195987: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:11:04,372::task::495::TaskManager.Task::(_debug) Task 8186739e-33ef-4064-a2e8-3dbd4e195987: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:11:04,373::task::495::TaskManager.Task::(_debug) Task 8186739e-33ef-4064-a2e8-3dbd4e195987: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:11:04,373::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:11:04,373::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:11:04,374::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:11:04,374::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:11:04,374::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:11:04,375::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:11:04,375::task::495::TaskManager.Task::(_debug) Task 8186739e-33ef-4064-a2e8-3dbd4e195987: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:11:04,376::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:11:04,376::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:11:04,376::task::495::TaskManager.Task::(_debug) Task 9a3c9a41-4853-43dc-ac52-53d8feba494d: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:11:04,377::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`99f7cc3f-c998-4f3a-976d-ff1bd694bc8f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:11:04,377::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:11:04,378::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:11:04,378::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`99f7cc3f-c998-4f3a-976d-ff1bd694bc8f`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:11:04,379::task::495::TaskManager.Task::(_debug) Task 9a3c9a41-4853-43dc-ac52-53d8feba494d: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:11:04,379::task::495::TaskManager.Task::(_debug) Task 9a3c9a41-4853-43dc-ac52-53d8feba494d: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:11:04,379::task::495::TaskManager.Task::(_debug) Task 9a3c9a41-4853-43dc-ac52-53d8feba494d: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:11:04,380::task::495::TaskManager.Task::(_debug) Task 9a3c9a41-4853-43dc-ac52-53d8feba494d: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:11:04,380::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:11:04,380::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:11:04,381::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:11:04,381::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:11:04,382::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:11:04,382::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:11:04,382::task::495::TaskManager.Task::(_debug) Task 9a3c9a41-4853-43dc-ac52-53d8feba494d: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:11:04,383::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:11:04,503::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0630825 s, 16.2 MB/s\n'; <rc> = 0 >Thread-15913701::DEBUG::2013-11-20 03:11:05,805::clientIF::54::vds::(wrapper) [192.168.100.135]::call setVmTicket with ('4760db3e-d3a8-4045-9153-12c36ec16ba2', 'xTPe7syms//D', '120') {} >Thread-15913701::DEBUG::2013-11-20 03:11:05,826::clientIF::59::vds::(wrapper) return setVmTicket with {'status': {'message': 'Done', 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:11:06,512::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913703::DEBUG::2013-11-20 03:11:06,773::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913703::INFO::2013-11-20 03:11:06,791::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913703::DEBUG::2013-11-20 03:11:06,792::task::495::TaskManager.Task::(_debug) Task a97487c8-a13a-4d7e-a2b0-f368b1ae1207: moving from state init -> state preparing >Thread-15913703::DEBUG::2013-11-20 03:11:06,792::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913703::DEBUG::2013-11-20 03:11:06,793::task::495::TaskManager.Task::(_debug) Task a97487c8-a13a-4d7e-a2b0-f368b1ae1207: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913703::DEBUG::2013-11-20 03:11:06,793::task::495::TaskManager.Task::(_debug) Task a97487c8-a13a-4d7e-a2b0-f368b1ae1207: moving from state preparing -> state finished >Thread-15913703::DEBUG::2013-11-20 03:11:06,794::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913703::DEBUG::2013-11-20 03:11:06,794::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913703::DEBUG::2013-11-20 03:11:06,794::task::495::TaskManager.Task::(_debug) Task a97487c8-a13a-4d7e-a2b0-f368b1ae1207: ref 0 aborting False >Thread-15913703::INFO::2013-11-20 03:11:06,795::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913704::DEBUG::2013-11-20 03:11:06,798::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913704::INFO::2013-11-20 03:11:06,799::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913704::DEBUG::2013-11-20 03:11:06,799::task::495::TaskManager.Task::(_debug) Task 399289b3-46fe-414b-8dfe-b29a2f61c53a: moving from state init -> state preparing >Thread-15913704::DEBUG::2013-11-20 03:11:06,800::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`7e2e1a83-dbe4-4941-8240-465c980e6731`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913704::DEBUG::2013-11-20 03:11:06,801::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913704::DEBUG::2013-11-20 03:11:06,801::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913704::DEBUG::2013-11-20 03:11:06,802::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`7e2e1a83-dbe4-4941-8240-465c980e6731`::Granted request >Thread-15913704::DEBUG::2013-11-20 03:11:06,802::task::495::TaskManager.Task::(_debug) Task 399289b3-46fe-414b-8dfe-b29a2f61c53a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913704::DEBUG::2013-11-20 03:11:06,803::task::495::TaskManager.Task::(_debug) Task 399289b3-46fe-414b-8dfe-b29a2f61c53a: ref 1 aborting False >Thread-15913704::WARNING::2013-11-20 03:11:06,804::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913704::DEBUG::2013-11-20 03:11:06,805::task::495::TaskManager.Task::(_debug) Task 399289b3-46fe-414b-8dfe-b29a2f61c53a: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913704::DEBUG::2013-11-20 03:11:06,806::task::495::TaskManager.Task::(_debug) Task 399289b3-46fe-414b-8dfe-b29a2f61c53a: moving from state preparing -> state finished >Thread-15913704::DEBUG::2013-11-20 03:11:06,806::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913704::DEBUG::2013-11-20 03:11:06,807::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913704::DEBUG::2013-11-20 03:11:06,808::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913704::DEBUG::2013-11-20 03:11:06,808::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913704::DEBUG::2013-11-20 03:11:06,809::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913704::DEBUG::2013-11-20 03:11:06,809::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913704::DEBUG::2013-11-20 03:11:06,810::task::495::TaskManager.Task::(_debug) Task 399289b3-46fe-414b-8dfe-b29a2f61c53a: ref 0 aborting False >Thread-15913704::INFO::2013-11-20 03:11:06,810::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:11:07,042::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0594308 s, 17.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:11:09,052::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:11:09,459::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.05087 s, 20.1 MB/s\n'; <rc> = 0 >Thread-15913706::DEBUG::2013-11-20 03:11:10,375::clientIF::54::vds::(wrapper) [192.168.100.135]::call setVmTicket with ('4760db3e-d3a8-4045-9153-12c36ec16ba2', 'MGHgWv5aMM1B', '120') {} >Thread-15913706::DEBUG::2013-11-20 03:11:10,394::clientIF::59::vds::(wrapper) return setVmTicket with {'status': {'message': 'Done', 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:11:11,469::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:11:11,879::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0507813 s, 20.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:11:13,889::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913639::INFO::2013-11-20 03:11:14,373::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:11:14,376::task::495::TaskManager.Task::(_debug) Task 4e120776-0c2a-4686-af90-ce8725f10d72: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:11:14,378::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a8d3a554-e24a-45ed-b20e-07abaa693204`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:11:14,378::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:11:14,380::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:11:14,380::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a8d3a554-e24a-45ed-b20e-07abaa693204`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:11:14,385::task::495::TaskManager.Task::(_debug) Task 4e120776-0c2a-4686-af90-ce8725f10d72: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:11:14,385::task::495::TaskManager.Task::(_debug) Task 4e120776-0c2a-4686-af90-ce8725f10d72: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:11:14,386::task::495::TaskManager.Task::(_debug) Task 4e120776-0c2a-4686-af90-ce8725f10d72: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:11:14,387::task::495::TaskManager.Task::(_debug) Task 4e120776-0c2a-4686-af90-ce8725f10d72: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:11:14,387::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:11:14,388::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:11:14,389::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:11:14,390::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:11:14,390::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:11:14,391::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:11:14,392::task::495::TaskManager.Task::(_debug) Task 4e120776-0c2a-4686-af90-ce8725f10d72: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:11:14,393::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:11:14,468::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.134324 s, 7.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:11:16,483::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:11:16,955::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0616967 s, 16.6 MB/s\n'; <rc> = 0 >Thread-15913709::DEBUG::2013-11-20 03:11:17,363::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913709::INFO::2013-11-20 03:11:17,364::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913709::DEBUG::2013-11-20 03:11:17,366::task::495::TaskManager.Task::(_debug) Task bcab8380-ef37-4e39-b843-462a4afb828a: moving from state init -> state preparing >Thread-15913709::DEBUG::2013-11-20 03:11:17,371::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913709::DEBUG::2013-11-20 03:11:17,373::task::495::TaskManager.Task::(_debug) Task bcab8380-ef37-4e39-b843-462a4afb828a: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913709::DEBUG::2013-11-20 03:11:17,374::task::495::TaskManager.Task::(_debug) Task bcab8380-ef37-4e39-b843-462a4afb828a: moving from state preparing -> state finished >Thread-15913709::DEBUG::2013-11-20 03:11:17,374::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913709::DEBUG::2013-11-20 03:11:17,376::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913709::DEBUG::2013-11-20 03:11:17,377::task::495::TaskManager.Task::(_debug) Task bcab8380-ef37-4e39-b843-462a4afb828a: ref 0 aborting False >Thread-15913709::INFO::2013-11-20 03:11:17,377::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913710::DEBUG::2013-11-20 03:11:17,382::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913710::INFO::2013-11-20 03:11:17,383::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913710::DEBUG::2013-11-20 03:11:17,383::task::495::TaskManager.Task::(_debug) Task fbf44ba7-0bbc-45db-aaa9-e7004aaa784b: moving from state init -> state preparing >Thread-15913710::DEBUG::2013-11-20 03:11:17,384::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`35aa8c09-fbba-4086-af51-4726039ff196`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913710::DEBUG::2013-11-20 03:11:17,385::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913710::DEBUG::2013-11-20 03:11:17,385::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913710::DEBUG::2013-11-20 03:11:17,386::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`35aa8c09-fbba-4086-af51-4726039ff196`::Granted request >Thread-15913710::DEBUG::2013-11-20 03:11:17,387::task::495::TaskManager.Task::(_debug) Task fbf44ba7-0bbc-45db-aaa9-e7004aaa784b: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913710::DEBUG::2013-11-20 03:11:17,387::task::495::TaskManager.Task::(_debug) Task fbf44ba7-0bbc-45db-aaa9-e7004aaa784b: ref 1 aborting False >Thread-15913710::WARNING::2013-11-20 03:11:17,389::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913710::DEBUG::2013-11-20 03:11:17,390::task::495::TaskManager.Task::(_debug) Task fbf44ba7-0bbc-45db-aaa9-e7004aaa784b: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913710::DEBUG::2013-11-20 03:11:17,390::task::495::TaskManager.Task::(_debug) Task fbf44ba7-0bbc-45db-aaa9-e7004aaa784b: moving from state preparing -> state finished >Thread-15913710::DEBUG::2013-11-20 03:11:17,391::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913710::DEBUG::2013-11-20 03:11:17,391::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913710::DEBUG::2013-11-20 03:11:17,392::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913710::DEBUG::2013-11-20 03:11:17,392::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913710::DEBUG::2013-11-20 03:11:17,393::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913710::DEBUG::2013-11-20 03:11:17,393::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913710::DEBUG::2013-11-20 03:11:17,394::task::495::TaskManager.Task::(_debug) Task fbf44ba7-0bbc-45db-aaa9-e7004aaa784b: ref 0 aborting False >Thread-15913710::INFO::2013-11-20 03:11:17,394::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913711::INFO::2013-11-20 03:11:17,943::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913711::DEBUG::2013-11-20 03:11:17,943::task::495::TaskManager.Task::(_debug) Task de306af0-5571-4e65-9e3f-af06b3d6a528: moving from state init -> state preparing >Thread-15913711::DEBUG::2013-11-20 03:11:17,955::task::495::TaskManager.Task::(_debug) Task de306af0-5571-4e65-9e3f-af06b3d6a528: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000243902206421', 'lastCheck': 1384917077.441107, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00100493431091', 'lastCheck': 1384917075.4347601, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00127005577087', 'lastCheck': 1384917076.7083399, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000327110290527', 'lastCheck': 1384917077.420923, 'valid': True, 'code': 0}} >Thread-15913711::DEBUG::2013-11-20 03:11:17,956::task::495::TaskManager.Task::(_debug) Task de306af0-5571-4e65-9e3f-af06b3d6a528: moving from state preparing -> state finished >Thread-15913711::DEBUG::2013-11-20 03:11:17,956::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913711::DEBUG::2013-11-20 03:11:17,957::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913711::DEBUG::2013-11-20 03:11:17,957::task::495::TaskManager.Task::(_debug) Task de306af0-5571-4e65-9e3f-af06b3d6a528: ref 0 aborting False >Thread-15913711::INFO::2013-11-20 03:11:17,958::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000243902206421', 'lastCheck': 1384917077.441107, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00100493431091', 'lastCheck': 1384917075.4347601, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00127005577087', 'lastCheck': 1384917076.7083399, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000327110290527', 'lastCheck': 1384917077.420923, 'valid': True, 'code': 0}} >Thread-15913712::DEBUG::2013-11-20 03:11:17,970::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::Disk stats not available >Thread-15913712::DEBUG::2013-11-20 03:11:17,970::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::Disk latency not available >Thread-15913712::DEBUG::2013-11-20 03:11:17,972::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk stats not available >Thread-15913712::DEBUG::2013-11-20 03:11:17,972::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk latency not available >Dummy-15912651::DEBUG::2013-11-20 03:11:18,966::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:11:20,149::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0517913 s, 19.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:11:22,160::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:11:22,607::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0488065 s, 21.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:11:24,618::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:11:25,193::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0553224 s, 18.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:11:27,206::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:11:27,610::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.04684 s, 21.9 MB/s\n'; <rc> = 0 >Thread-15913716::DEBUG::2013-11-20 03:11:27,729::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913716::INFO::2013-11-20 03:11:27,730::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913716::DEBUG::2013-11-20 03:11:27,731::task::495::TaskManager.Task::(_debug) Task b59c2a19-0f5a-46d3-85b4-3fd6f47dbba0: moving from state init -> state preparing >Thread-15913716::DEBUG::2013-11-20 03:11:27,731::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913716::DEBUG::2013-11-20 03:11:27,732::task::495::TaskManager.Task::(_debug) Task b59c2a19-0f5a-46d3-85b4-3fd6f47dbba0: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913716::DEBUG::2013-11-20 03:11:27,732::task::495::TaskManager.Task::(_debug) Task b59c2a19-0f5a-46d3-85b4-3fd6f47dbba0: moving from state preparing -> state finished >Thread-15913716::DEBUG::2013-11-20 03:11:27,733::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913716::DEBUG::2013-11-20 03:11:27,733::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913716::DEBUG::2013-11-20 03:11:27,733::task::495::TaskManager.Task::(_debug) Task b59c2a19-0f5a-46d3-85b4-3fd6f47dbba0: ref 0 aborting False >Thread-15913716::INFO::2013-11-20 03:11:27,734::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913717::DEBUG::2013-11-20 03:11:27,737::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913717::INFO::2013-11-20 03:11:27,738::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913717::DEBUG::2013-11-20 03:11:27,738::task::495::TaskManager.Task::(_debug) Task 6a5eed91-6863-407c-b4ce-d075b4fbd54f: moving from state init -> state preparing >Thread-15913717::DEBUG::2013-11-20 03:11:27,739::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a1c529cf-8c3b-4359-a7ab-5e89350364cc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913717::DEBUG::2013-11-20 03:11:27,739::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913717::DEBUG::2013-11-20 03:11:27,740::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913717::DEBUG::2013-11-20 03:11:27,740::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a1c529cf-8c3b-4359-a7ab-5e89350364cc`::Granted request >Thread-15913717::DEBUG::2013-11-20 03:11:27,741::task::495::TaskManager.Task::(_debug) Task 6a5eed91-6863-407c-b4ce-d075b4fbd54f: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913717::DEBUG::2013-11-20 03:11:27,741::task::495::TaskManager.Task::(_debug) Task 6a5eed91-6863-407c-b4ce-d075b4fbd54f: ref 1 aborting False >Thread-15913717::WARNING::2013-11-20 03:11:27,743::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913717::DEBUG::2013-11-20 03:11:27,743::task::495::TaskManager.Task::(_debug) Task 6a5eed91-6863-407c-b4ce-d075b4fbd54f: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913717::DEBUG::2013-11-20 03:11:27,744::task::495::TaskManager.Task::(_debug) Task 6a5eed91-6863-407c-b4ce-d075b4fbd54f: moving from state preparing -> state finished >Thread-15913717::DEBUG::2013-11-20 03:11:27,744::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913717::DEBUG::2013-11-20 03:11:27,745::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913717::DEBUG::2013-11-20 03:11:27,745::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913717::DEBUG::2013-11-20 03:11:27,746::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913717::DEBUG::2013-11-20 03:11:27,746::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913717::DEBUG::2013-11-20 03:11:27,747::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913717::DEBUG::2013-11-20 03:11:27,747::task::495::TaskManager.Task::(_debug) Task 6a5eed91-6863-407c-b4ce-d075b4fbd54f: ref 0 aborting False >Thread-15913717::INFO::2013-11-20 03:11:27,748::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:11:29,620::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:11:30,807::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0495314 s, 20.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:11:32,820::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:11:33,293::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0534308 s, 19.2 MB/s\n'; <rc> = 0 >Thread-15913719::INFO::2013-11-20 03:11:33,540::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913719::DEBUG::2013-11-20 03:11:33,540::task::495::TaskManager.Task::(_debug) Task c877edfa-5789-4153-9420-efdf027fd953: moving from state init -> state preparing >Thread-15913719::DEBUG::2013-11-20 03:11:33,541::task::495::TaskManager.Task::(_debug) Task c877edfa-5789-4153-9420-efdf027fd953: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000301122665405', 'lastCheck': 1384917087.4535971, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0221340656281', 'lastCheck': 1384917085.4699149, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0141389369965', 'lastCheck': 1384917086.8147681, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000437021255493', 'lastCheck': 1384917087.4309959, 'valid': True, 'code': 0}} >Thread-15913719::DEBUG::2013-11-20 03:11:33,541::task::495::TaskManager.Task::(_debug) Task c877edfa-5789-4153-9420-efdf027fd953: moving from state preparing -> state finished >Thread-15913719::DEBUG::2013-11-20 03:11:33,542::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913719::DEBUG::2013-11-20 03:11:33,542::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913719::DEBUG::2013-11-20 03:11:33,543::task::495::TaskManager.Task::(_debug) Task c877edfa-5789-4153-9420-efdf027fd953: ref 0 aborting False >Thread-15913719::INFO::2013-11-20 03:11:33,543::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000301122665405', 'lastCheck': 1384917087.4535971, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0221340656281', 'lastCheck': 1384917085.4699149, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0141389369965', 'lastCheck': 1384917086.8147681, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000437021255493', 'lastCheck': 1384917087.4309959, 'valid': True, 'code': 0}} >Thread-15913720::DEBUG::2013-11-20 03:11:33,574::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::Disk stats not available >Thread-15913720::DEBUG::2013-11-20 03:11:33,575::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::Disk latency not available >Thread-15913720::DEBUG::2013-11-20 03:11:33,576::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk stats not available >Thread-15913720::DEBUG::2013-11-20 03:11:33,576::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::Disk latency not available >Thread-15913658::INFO::2013-11-20 03:11:34,386::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:11:34,387::task::495::TaskManager.Task::(_debug) Task 62c9006a-495c-4ecf-b921-00928a82cfca: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:11:34,388::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`1289aa57-26c6-490e-b2cd-c843bcf48dd5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:11:34,388::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:11:34,389::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:11:34,389::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`1289aa57-26c6-490e-b2cd-c843bcf48dd5`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:11:34,390::task::495::TaskManager.Task::(_debug) Task 62c9006a-495c-4ecf-b921-00928a82cfca: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:11:34,391::task::495::TaskManager.Task::(_debug) Task 62c9006a-495c-4ecf-b921-00928a82cfca: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:11:34,391::task::495::TaskManager.Task::(_debug) Task 62c9006a-495c-4ecf-b921-00928a82cfca: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:11:34,392::task::495::TaskManager.Task::(_debug) Task 62c9006a-495c-4ecf-b921-00928a82cfca: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:11:34,392::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:11:34,393::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:11:34,393::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:11:34,394::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:11:34,394::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:11:34,394::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:11:34,395::task::495::TaskManager.Task::(_debug) Task 62c9006a-495c-4ecf-b921-00928a82cfca: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:11:34,395::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:11:35,304::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:11:37,021::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0552596 s, 18.5 MB/s\n'; <rc> = 0 >Thread-42838::DEBUG::2013-11-20 03:11:37,496::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 03:11:38,091::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913722::DEBUG::2013-11-20 03:11:38,460::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913722::INFO::2013-11-20 03:11:38,461::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913722::DEBUG::2013-11-20 03:11:38,462::task::495::TaskManager.Task::(_debug) Task 4d243623-ee1d-4a8f-a547-2422b1e9df0e: moving from state init -> state preparing >Thread-15913722::DEBUG::2013-11-20 03:11:38,462::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913722::DEBUG::2013-11-20 03:11:38,463::task::495::TaskManager.Task::(_debug) Task 4d243623-ee1d-4a8f-a547-2422b1e9df0e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913722::DEBUG::2013-11-20 03:11:38,463::task::495::TaskManager.Task::(_debug) Task 4d243623-ee1d-4a8f-a547-2422b1e9df0e: moving from state preparing -> state finished >Thread-15913722::DEBUG::2013-11-20 03:11:38,463::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913722::DEBUG::2013-11-20 03:11:38,464::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913722::DEBUG::2013-11-20 03:11:38,464::task::495::TaskManager.Task::(_debug) Task 4d243623-ee1d-4a8f-a547-2422b1e9df0e: ref 0 aborting False >Thread-15913722::INFO::2013-11-20 03:11:38,465::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913723::DEBUG::2013-11-20 03:11:38,468::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913723::INFO::2013-11-20 03:11:38,469::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913723::DEBUG::2013-11-20 03:11:38,469::task::495::TaskManager.Task::(_debug) Task ced88d08-e93f-4e74-baf6-baa4d9fdf646: moving from state init -> state preparing >Thread-15913723::DEBUG::2013-11-20 03:11:38,470::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`74b5f494-7a2f-4d09-9430-0cd14cf3c2af`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913723::DEBUG::2013-11-20 03:11:38,471::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913723::DEBUG::2013-11-20 03:11:38,471::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913723::DEBUG::2013-11-20 03:11:38,472::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`74b5f494-7a2f-4d09-9430-0cd14cf3c2af`::Granted request >Thread-15913723::DEBUG::2013-11-20 03:11:38,472::task::495::TaskManager.Task::(_debug) Task ced88d08-e93f-4e74-baf6-baa4d9fdf646: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913723::DEBUG::2013-11-20 03:11:38,473::task::495::TaskManager.Task::(_debug) Task ced88d08-e93f-4e74-baf6-baa4d9fdf646: ref 1 aborting False >Thread-15913723::WARNING::2013-11-20 03:11:38,474::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913723::DEBUG::2013-11-20 03:11:38,475::task::495::TaskManager.Task::(_debug) Task ced88d08-e93f-4e74-baf6-baa4d9fdf646: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913723::DEBUG::2013-11-20 03:11:38,475::task::495::TaskManager.Task::(_debug) Task ced88d08-e93f-4e74-baf6-baa4d9fdf646: moving from state preparing -> state finished >Thread-15913723::DEBUG::2013-11-20 03:11:38,476::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913723::DEBUG::2013-11-20 03:11:38,476::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913723::DEBUG::2013-11-20 03:11:38,476::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913723::DEBUG::2013-11-20 03:11:38,477::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913723::DEBUG::2013-11-20 03:11:38,477::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913723::DEBUG::2013-11-20 03:11:38,477::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913723::DEBUG::2013-11-20 03:11:38,478::task::495::TaskManager.Task::(_debug) Task ced88d08-e93f-4e74-baf6-baa4d9fdf646: ref 0 aborting False >Thread-15913723::INFO::2013-11-20 03:11:38,478::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:11:39,035::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:11:39,478::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0515771 s, 19.9 MB/s\n'; <rc> = 0 >Thread-15912392::INFO::2013-11-20 03:11:40,933::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:11:40,934::task::495::TaskManager.Task::(_debug) Task dc3b2580-e178-4471-a816-08568b7383f2: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:11:40,935::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`4f78811a-4c15-436e-84b1-b214f22aa468`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:11:40,936::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:11:40,936::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:11:40,937::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`4f78811a-4c15-436e-84b1-b214f22aa468`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:11:40,938::task::495::TaskManager.Task::(_debug) Task dc3b2580-e178-4471-a816-08568b7383f2: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:11:40,938::task::495::TaskManager.Task::(_debug) Task dc3b2580-e178-4471-a816-08568b7383f2: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:11:40,939::task::495::TaskManager.Task::(_debug) Task dc3b2580-e178-4471-a816-08568b7383f2: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:11:40,939::task::495::TaskManager.Task::(_debug) Task dc3b2580-e178-4471-a816-08568b7383f2: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:11:40,940::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:11:40,940::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:11:40,941::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:11:40,941::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:11:40,942::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:11:40,942::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:11:40,943::task::495::TaskManager.Task::(_debug) Task dc3b2580-e178-4471-a816-08568b7383f2: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:11:40,943::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:11:41,489::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:11:42,060::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.104569 s, 9.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:11:44,071::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:11:44,505::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0586969 s, 17.4 MB/s\n'; <rc> = 0 >Thread-15913518::INFO::2013-11-20 03:11:45,276::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:11:45,277::task::495::TaskManager.Task::(_debug) Task de646adf-7e54-4a28-905a-6acd12e14fa2: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:11:45,278::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0c311331-81cb-4228-9385-a8f3c433e293`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:11:45,278::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:11:45,279::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:11:45,279::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0c311331-81cb-4228-9385-a8f3c433e293`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:11:45,280::task::495::TaskManager.Task::(_debug) Task de646adf-7e54-4a28-905a-6acd12e14fa2: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:11:45,280::task::495::TaskManager.Task::(_debug) Task de646adf-7e54-4a28-905a-6acd12e14fa2: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:11:45,281::task::495::TaskManager.Task::(_debug) Task de646adf-7e54-4a28-905a-6acd12e14fa2: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:11:45,281::task::495::TaskManager.Task::(_debug) Task de646adf-7e54-4a28-905a-6acd12e14fa2: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:11:45,281::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:11:45,282::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:11:45,282::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:11:45,282::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:11:45,283::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:11:45,283::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:11:45,283::task::495::TaskManager.Task::(_debug) Task de646adf-7e54-4a28-905a-6acd12e14fa2: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:11:45,284::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:11:45,284::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:11:45,285::task::495::TaskManager.Task::(_debug) Task 4a0c9659-5d78-4927-8eff-47da250d59f8: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:11:45,285::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c9a5f04c-76d2-4055-9d7d-bc69ba2fc549`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:11:45,286::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:11:45,286::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:11:45,286::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c9a5f04c-76d2-4055-9d7d-bc69ba2fc549`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:11:45,287::task::495::TaskManager.Task::(_debug) Task 4a0c9659-5d78-4927-8eff-47da250d59f8: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:11:45,287::task::495::TaskManager.Task::(_debug) Task 4a0c9659-5d78-4927-8eff-47da250d59f8: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:11:45,287::task::495::TaskManager.Task::(_debug) Task 4a0c9659-5d78-4927-8eff-47da250d59f8: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:11:45,288::task::495::TaskManager.Task::(_debug) Task 4a0c9659-5d78-4927-8eff-47da250d59f8: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:11:45,288::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:11:45,288::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:11:45,289::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:11:45,289::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:11:45,290::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:11:45,290::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:11:45,290::task::495::TaskManager.Task::(_debug) Task 4a0c9659-5d78-4927-8eff-47da250d59f8: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:11:45,291::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:11:46,518::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:11:46,941::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0507749 s, 20.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:11:48,951::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913727::DEBUG::2013-11-20 03:11:49,181::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913727::INFO::2013-11-20 03:11:49,183::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913727::DEBUG::2013-11-20 03:11:49,184::task::495::TaskManager.Task::(_debug) Task e6bddd01-f817-4763-8583-e716e9eff46f: moving from state init -> state preparing >Thread-15913727::DEBUG::2013-11-20 03:11:49,185::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913727::DEBUG::2013-11-20 03:11:49,185::task::495::TaskManager.Task::(_debug) Task e6bddd01-f817-4763-8583-e716e9eff46f: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913727::DEBUG::2013-11-20 03:11:49,185::task::495::TaskManager.Task::(_debug) Task e6bddd01-f817-4763-8583-e716e9eff46f: moving from state preparing -> state finished >Thread-15913727::DEBUG::2013-11-20 03:11:49,186::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913727::DEBUG::2013-11-20 03:11:49,186::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913727::DEBUG::2013-11-20 03:11:49,187::task::495::TaskManager.Task::(_debug) Task e6bddd01-f817-4763-8583-e716e9eff46f: ref 0 aborting False >Thread-15913727::INFO::2013-11-20 03:11:49,187::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913728::DEBUG::2013-11-20 03:11:49,191::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913728::INFO::2013-11-20 03:11:49,192::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913728::DEBUG::2013-11-20 03:11:49,192::task::495::TaskManager.Task::(_debug) Task b23d1a7d-d77d-4a09-a4f1-35a4b9bdf2f9: moving from state init -> state preparing >Thread-15913728::DEBUG::2013-11-20 03:11:49,193::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`309305a6-76c9-4c77-abdc-011d44e4b989`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913728::DEBUG::2013-11-20 03:11:49,193::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913728::DEBUG::2013-11-20 03:11:49,194::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913728::DEBUG::2013-11-20 03:11:49,194::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`309305a6-76c9-4c77-abdc-011d44e4b989`::Granted request >Thread-15913728::DEBUG::2013-11-20 03:11:49,195::task::495::TaskManager.Task::(_debug) Task b23d1a7d-d77d-4a09-a4f1-35a4b9bdf2f9: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913728::DEBUG::2013-11-20 03:11:49,195::task::495::TaskManager.Task::(_debug) Task b23d1a7d-d77d-4a09-a4f1-35a4b9bdf2f9: ref 1 aborting False >Thread-15913728::WARNING::2013-11-20 03:11:49,197::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913728::DEBUG::2013-11-20 03:11:49,197::task::495::TaskManager.Task::(_debug) Task b23d1a7d-d77d-4a09-a4f1-35a4b9bdf2f9: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913728::DEBUG::2013-11-20 03:11:49,198::task::495::TaskManager.Task::(_debug) Task b23d1a7d-d77d-4a09-a4f1-35a4b9bdf2f9: moving from state preparing -> state finished >Thread-15913728::DEBUG::2013-11-20 03:11:49,198::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913728::DEBUG::2013-11-20 03:11:49,199::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913728::DEBUG::2013-11-20 03:11:49,199::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913728::DEBUG::2013-11-20 03:11:49,200::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913728::DEBUG::2013-11-20 03:11:49,200::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913728::DEBUG::2013-11-20 03:11:49,200::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913728::DEBUG::2013-11-20 03:11:49,201::task::495::TaskManager.Task::(_debug) Task b23d1a7d-d77d-4a09-a4f1-35a4b9bdf2f9: ref 0 aborting False >Thread-15913728::INFO::2013-11-20 03:11:49,201::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:11:49,398::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0508222 s, 20.1 MB/s\n'; <rc> = 0 >Thread-15913729::INFO::2013-11-20 03:11:49,481::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913729::DEBUG::2013-11-20 03:11:49,481::task::495::TaskManager.Task::(_debug) Task de48a4fb-89cf-42a8-8906-5529fd2b8f93: moving from state init -> state preparing >Thread-15913729::DEBUG::2013-11-20 03:11:49,482::task::495::TaskManager.Task::(_debug) Task de48a4fb-89cf-42a8-8906-5529fd2b8f93: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000226974487305', 'lastCheck': 1384917108.2509661, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0129749774933', 'lastCheck': 1384917105.9833441, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0141789913177', 'lastCheck': 1384917107.0371189, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000257015228271', 'lastCheck': 1384917107.693994, 'valid': True, 'code': 0}} >Thread-15913729::DEBUG::2013-11-20 03:11:49,483::task::495::TaskManager.Task::(_debug) Task de48a4fb-89cf-42a8-8906-5529fd2b8f93: moving from state preparing -> state finished >Thread-15913729::DEBUG::2013-11-20 03:11:49,483::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913729::DEBUG::2013-11-20 03:11:49,483::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913729::DEBUG::2013-11-20 03:11:49,484::task::495::TaskManager.Task::(_debug) Task de48a4fb-89cf-42a8-8906-5529fd2b8f93: ref 0 aborting False >Thread-15913729::INFO::2013-11-20 03:11:49,484::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000226974487305', 'lastCheck': 1384917108.2509661, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0129749774933', 'lastCheck': 1384917105.9833441, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0141789913177', 'lastCheck': 1384917107.0371189, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000257015228271', 'lastCheck': 1384917107.693994, 'valid': True, 'code': 0}} >Thread-15913730::DEBUG::2013-11-20 03:11:49,542::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::Disk stats not available >Thread-15913730::DEBUG::2013-11-20 03:11:49,542::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::Disk latency not available >Dummy-15912651::DEBUG::2013-11-20 03:11:51,407::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:11:51,852::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0571939 s, 17.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:11:53,862::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:11:56,439::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0591032 s, 17.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:11:58,450::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:11:58,875::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0496416 s, 20.6 MB/s\n'; <rc> = 0 >Thread-15913734::DEBUG::2013-11-20 03:11:59,428::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913734::INFO::2013-11-20 03:11:59,435::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913734::DEBUG::2013-11-20 03:11:59,436::task::495::TaskManager.Task::(_debug) Task 3fe5a340-ae1c-455e-9951-fb74af25c357: moving from state init -> state preparing >Thread-15913734::DEBUG::2013-11-20 03:11:59,437::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913734::DEBUG::2013-11-20 03:11:59,438::task::495::TaskManager.Task::(_debug) Task 3fe5a340-ae1c-455e-9951-fb74af25c357: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913734::DEBUG::2013-11-20 03:11:59,439::task::495::TaskManager.Task::(_debug) Task 3fe5a340-ae1c-455e-9951-fb74af25c357: moving from state preparing -> state finished >Thread-15913734::DEBUG::2013-11-20 03:11:59,439::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913734::DEBUG::2013-11-20 03:11:59,440::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913734::DEBUG::2013-11-20 03:11:59,440::task::495::TaskManager.Task::(_debug) Task 3fe5a340-ae1c-455e-9951-fb74af25c357: ref 0 aborting False >Thread-15913734::INFO::2013-11-20 03:11:59,441::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913735::DEBUG::2013-11-20 03:11:59,449::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913735::INFO::2013-11-20 03:11:59,450::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913735::DEBUG::2013-11-20 03:11:59,450::task::495::TaskManager.Task::(_debug) Task a19ea7c0-5812-42e2-b952-e3b921222099: moving from state init -> state preparing >Thread-15913735::DEBUG::2013-11-20 03:11:59,451::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b09eacb6-1f08-4200-8f29-d7f123648d80`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913735::DEBUG::2013-11-20 03:11:59,451::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913735::DEBUG::2013-11-20 03:11:59,452::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913735::DEBUG::2013-11-20 03:11:59,453::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b09eacb6-1f08-4200-8f29-d7f123648d80`::Granted request >Thread-15913735::DEBUG::2013-11-20 03:11:59,453::task::495::TaskManager.Task::(_debug) Task a19ea7c0-5812-42e2-b952-e3b921222099: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913735::DEBUG::2013-11-20 03:11:59,454::task::495::TaskManager.Task::(_debug) Task a19ea7c0-5812-42e2-b952-e3b921222099: ref 1 aborting False >Thread-15913735::WARNING::2013-11-20 03:11:59,455::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913735::DEBUG::2013-11-20 03:11:59,456::task::495::TaskManager.Task::(_debug) Task a19ea7c0-5812-42e2-b952-e3b921222099: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913735::DEBUG::2013-11-20 03:11:59,457::task::495::TaskManager.Task::(_debug) Task a19ea7c0-5812-42e2-b952-e3b921222099: moving from state preparing -> state finished >Thread-15913735::DEBUG::2013-11-20 03:11:59,457::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913735::DEBUG::2013-11-20 03:11:59,458::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913735::DEBUG::2013-11-20 03:11:59,458::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913735::DEBUG::2013-11-20 03:11:59,459::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913735::DEBUG::2013-11-20 03:11:59,459::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913735::DEBUG::2013-11-20 03:11:59,459::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913735::DEBUG::2013-11-20 03:11:59,460::task::495::TaskManager.Task::(_debug) Task a19ea7c0-5812-42e2-b952-e3b921222099: ref 0 aborting False >Thread-15913735::INFO::2013-11-20 03:11:59,460::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:12:00,885::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:12:01,375::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0514781 s, 19.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:12:03,454::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:12:04,097::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0520837 s, 19.7 MB/s\n'; <rc> = 0 >Thread-15913546::INFO::2013-11-20 03:12:04,624::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:12:04,625::task::495::TaskManager.Task::(_debug) Task 5e7d9a65-1f1a-4d27-8d22-dc05ff85e01b: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:12:04,626::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`f0874edc-f9ac-4426-86e3-536a622ed873`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:12:04,626::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:12:04,627::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:12:04,627::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`f0874edc-f9ac-4426-86e3-536a622ed873`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:12:04,628::task::495::TaskManager.Task::(_debug) Task 5e7d9a65-1f1a-4d27-8d22-dc05ff85e01b: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:12:04,628::task::495::TaskManager.Task::(_debug) Task 5e7d9a65-1f1a-4d27-8d22-dc05ff85e01b: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:12:04,629::task::495::TaskManager.Task::(_debug) Task 5e7d9a65-1f1a-4d27-8d22-dc05ff85e01b: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:12:04,629::task::495::TaskManager.Task::(_debug) Task 5e7d9a65-1f1a-4d27-8d22-dc05ff85e01b: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:12:04,629::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:12:04,630::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:12:04,630::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:12:04,630::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:12:04,631::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:12:04,631::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:12:04,631::task::495::TaskManager.Task::(_debug) Task 5e7d9a65-1f1a-4d27-8d22-dc05ff85e01b: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:12:04,632::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:12:04,632::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:12:04,633::task::495::TaskManager.Task::(_debug) Task d01603eb-b077-42a0-98cf-a9982ef60397: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:12:04,633::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`30c5f505-9c6d-4d24-8f02-58c498ac9350`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:12:04,634::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:12:04,634::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:12:04,635::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`30c5f505-9c6d-4d24-8f02-58c498ac9350`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:12:04,636::task::495::TaskManager.Task::(_debug) Task d01603eb-b077-42a0-98cf-a9982ef60397: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:12:04,636::task::495::TaskManager.Task::(_debug) Task d01603eb-b077-42a0-98cf-a9982ef60397: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:12:04,636::task::495::TaskManager.Task::(_debug) Task d01603eb-b077-42a0-98cf-a9982ef60397: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:12:04,637::task::495::TaskManager.Task::(_debug) Task d01603eb-b077-42a0-98cf-a9982ef60397: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:12:04,637::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:12:04,637::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:12:04,638::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:12:04,638::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:12:04,638::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:12:04,639::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:12:04,639::task::495::TaskManager.Task::(_debug) Task d01603eb-b077-42a0-98cf-a9982ef60397: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:12:04,639::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:12:04,640::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:12:04,640::task::495::TaskManager.Task::(_debug) Task 2cf7f061-325d-47aa-be5b-fcc59f300052: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:12:04,641::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`bd97cb59-ac74-4f34-9397-01e240ffdbab`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:12:04,641::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:12:04,642::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:12:04,642::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`bd97cb59-ac74-4f34-9397-01e240ffdbab`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:12:04,643::task::495::TaskManager.Task::(_debug) Task 2cf7f061-325d-47aa-be5b-fcc59f300052: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:12:04,643::task::495::TaskManager.Task::(_debug) Task 2cf7f061-325d-47aa-be5b-fcc59f300052: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:12:04,643::task::495::TaskManager.Task::(_debug) Task 2cf7f061-325d-47aa-be5b-fcc59f300052: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:12:04,644::task::495::TaskManager.Task::(_debug) Task 2cf7f061-325d-47aa-be5b-fcc59f300052: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:12:04,644::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:12:04,644::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:12:04,645::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:12:04,645::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:12:04,645::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:12:04,646::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:12:04,646::task::495::TaskManager.Task::(_debug) Task 2cf7f061-325d-47aa-be5b-fcc59f300052: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:12:04,646::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:12:04,647::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:12:04,647::task::495::TaskManager.Task::(_debug) Task d5f15f2a-7469-43d8-98b9-12516c16220a: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:12:04,648::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`b298d285-764a-4364-9f71-1d869fa0ca03`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:12:04,648::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:12:04,648::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:12:04,649::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`b298d285-764a-4364-9f71-1d869fa0ca03`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:12:04,649::task::495::TaskManager.Task::(_debug) Task d5f15f2a-7469-43d8-98b9-12516c16220a: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:12:04,650::task::495::TaskManager.Task::(_debug) Task d5f15f2a-7469-43d8-98b9-12516c16220a: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:12:04,650::task::495::TaskManager.Task::(_debug) Task d5f15f2a-7469-43d8-98b9-12516c16220a: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:12:04,650::task::495::TaskManager.Task::(_debug) Task d5f15f2a-7469-43d8-98b9-12516c16220a: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:12:04,651::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:12:04,651::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:12:04,651::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:12:04,652::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:12:04,652::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:12:04,652::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:12:04,653::task::495::TaskManager.Task::(_debug) Task d5f15f2a-7469-43d8-98b9-12516c16220a: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:12:04,653::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913737::INFO::2013-11-20 03:12:05,403::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913737::DEBUG::2013-11-20 03:12:05,404::task::495::TaskManager.Task::(_debug) Task 54282b64-480d-487d-af1e-58d85d2dbe79: moving from state init -> state preparing >Thread-15913737::DEBUG::2013-11-20 03:12:05,404::task::495::TaskManager.Task::(_debug) Task 54282b64-480d-487d-af1e-58d85d2dbe79: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000262022018433', 'lastCheck': 1384917119.1702211, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00093412399292', 'lastCheck': 1384917116.187726, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000868797302246', 'lastCheck': 1384917117.187288, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000303030014038', 'lastCheck': 1384917118.1651831, 'valid': True, 'code': 0}} >Thread-15913737::DEBUG::2013-11-20 03:12:05,405::task::495::TaskManager.Task::(_debug) Task 54282b64-480d-487d-af1e-58d85d2dbe79: moving from state preparing -> state finished >Thread-15913737::DEBUG::2013-11-20 03:12:05,405::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913737::DEBUG::2013-11-20 03:12:05,406::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913737::DEBUG::2013-11-20 03:12:05,406::task::495::TaskManager.Task::(_debug) Task 54282b64-480d-487d-af1e-58d85d2dbe79: ref 0 aborting False >Thread-15913737::INFO::2013-11-20 03:12:05,406::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000262022018433', 'lastCheck': 1384917119.1702211, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00093412399292', 'lastCheck': 1384917116.187726, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000868797302246', 'lastCheck': 1384917117.187288, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000303030014038', 'lastCheck': 1384917118.1651831, 'valid': True, 'code': 0}} >Thread-15913738::DEBUG::2013-11-20 03:12:05,439::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::Disk stats not available >Thread-15913738::DEBUG::2013-11-20 03:12:05,439::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::Disk latency not available >Dummy-15912651::DEBUG::2013-11-20 03:12:06,107::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:12:06,554::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0495487 s, 20.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:12:08,564::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913740::DEBUG::2013-11-20 03:12:09,767::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913740::INFO::2013-11-20 03:12:09,768::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913740::DEBUG::2013-11-20 03:12:09,769::task::495::TaskManager.Task::(_debug) Task 897d8a7d-da9e-4a34-88c7-b71823e3afdc: moving from state init -> state preparing >Thread-15913740::DEBUG::2013-11-20 03:12:09,770::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913740::DEBUG::2013-11-20 03:12:09,770::task::495::TaskManager.Task::(_debug) Task 897d8a7d-da9e-4a34-88c7-b71823e3afdc: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913740::DEBUG::2013-11-20 03:12:09,771::task::495::TaskManager.Task::(_debug) Task 897d8a7d-da9e-4a34-88c7-b71823e3afdc: moving from state preparing -> state finished >Thread-15913740::DEBUG::2013-11-20 03:12:09,771::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913740::DEBUG::2013-11-20 03:12:09,771::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913740::DEBUG::2013-11-20 03:12:09,772::task::495::TaskManager.Task::(_debug) Task 897d8a7d-da9e-4a34-88c7-b71823e3afdc: ref 0 aborting False >Thread-15913740::INFO::2013-11-20 03:12:09,772::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913741::DEBUG::2013-11-20 03:12:09,777::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913741::INFO::2013-11-20 03:12:09,778::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913741::DEBUG::2013-11-20 03:12:09,779::task::495::TaskManager.Task::(_debug) Task bf123faf-fc5f-4e40-b6b7-f4152f51da98: moving from state init -> state preparing >Thread-15913741::DEBUG::2013-11-20 03:12:09,780::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a188a763-cbb5-4dec-b10e-3af5d6c918ff`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913741::DEBUG::2013-11-20 03:12:09,781::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913741::DEBUG::2013-11-20 03:12:09,782::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913741::DEBUG::2013-11-20 03:12:09,782::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a188a763-cbb5-4dec-b10e-3af5d6c918ff`::Granted request >Thread-15913741::DEBUG::2013-11-20 03:12:09,783::task::495::TaskManager.Task::(_debug) Task bf123faf-fc5f-4e40-b6b7-f4152f51da98: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913741::DEBUG::2013-11-20 03:12:09,784::task::495::TaskManager.Task::(_debug) Task bf123faf-fc5f-4e40-b6b7-f4152f51da98: ref 1 aborting False >Thread-15913741::WARNING::2013-11-20 03:12:09,786::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913741::DEBUG::2013-11-20 03:12:09,786::task::495::TaskManager.Task::(_debug) Task bf123faf-fc5f-4e40-b6b7-f4152f51da98: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913741::DEBUG::2013-11-20 03:12:09,787::task::495::TaskManager.Task::(_debug) Task bf123faf-fc5f-4e40-b6b7-f4152f51da98: moving from state preparing -> state finished >Thread-15913741::DEBUG::2013-11-20 03:12:09,788::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913741::DEBUG::2013-11-20 03:12:09,788::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913741::DEBUG::2013-11-20 03:12:09,789::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913741::DEBUG::2013-11-20 03:12:09,789::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913741::DEBUG::2013-11-20 03:12:09,790::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913741::DEBUG::2013-11-20 03:12:09,790::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913741::DEBUG::2013-11-20 03:12:09,791::task::495::TaskManager.Task::(_debug) Task bf123faf-fc5f-4e40-b6b7-f4152f51da98: ref 0 aborting False >Thread-15913741::INFO::2013-11-20 03:12:09,791::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:12:10,009::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0475086 s, 21.6 MB/s\n'; <rc> = 0 >Thread-15913695::INFO::2013-11-20 03:12:11,465::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:12:11,466::task::495::TaskManager.Task::(_debug) Task a700e5ad-5866-49e7-ab22-b69b50d91e77: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:12:11,467::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a394817a-d712-4ca0-b9a0-74b7dc587491`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:12:11,468::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:12:11,468::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:12:11,469::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a394817a-d712-4ca0-b9a0-74b7dc587491`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:12:11,469::task::495::TaskManager.Task::(_debug) Task a700e5ad-5866-49e7-ab22-b69b50d91e77: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:12:11,470::task::495::TaskManager.Task::(_debug) Task a700e5ad-5866-49e7-ab22-b69b50d91e77: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:12:11,471::task::495::TaskManager.Task::(_debug) Task a700e5ad-5866-49e7-ab22-b69b50d91e77: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:12:11,471::task::495::TaskManager.Task::(_debug) Task a700e5ad-5866-49e7-ab22-b69b50d91e77: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:12:11,472::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:12:11,472::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:12:11,473::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:12:11,473::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:12:11,474::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:12:11,474::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:12:11,475::task::495::TaskManager.Task::(_debug) Task a700e5ad-5866-49e7-ab22-b69b50d91e77: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:12:11,475::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:12:12,018::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:12:13,292::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0523678 s, 19.6 MB/s\n'; <rc> = 0 >Thread-15913639::INFO::2013-11-20 03:12:14,464::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:12:14,469::task::495::TaskManager.Task::(_debug) Task 89f4e729-5f9f-4dfa-95f1-0ae4cf113497: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:12:14,471::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`292551e2-ea4d-4ac8-aa7a-750e7ed0d344`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:12:14,472::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:12:14,473::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:12:14,473::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`292551e2-ea4d-4ac8-aa7a-750e7ed0d344`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:12:14,474::task::495::TaskManager.Task::(_debug) Task 89f4e729-5f9f-4dfa-95f1-0ae4cf113497: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:12:14,475::task::495::TaskManager.Task::(_debug) Task 89f4e729-5f9f-4dfa-95f1-0ae4cf113497: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:12:14,475::task::495::TaskManager.Task::(_debug) Task 89f4e729-5f9f-4dfa-95f1-0ae4cf113497: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:12:14,476::task::495::TaskManager.Task::(_debug) Task 89f4e729-5f9f-4dfa-95f1-0ae4cf113497: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:12:14,477::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:12:14,477::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:12:14,478::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:12:14,479::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:12:14,479::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:12:14,480::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:12:14,480::task::495::TaskManager.Task::(_debug) Task 89f4e729-5f9f-4dfa-95f1-0ae4cf113497: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:12:14,481::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:12:15,303::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:12:15,709::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0468457 s, 21.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:12:17,719::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:12:18,930::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.050402 s, 20.3 MB/s\n'; <rc> = 0 >Thread-15913745::DEBUG::2013-11-20 03:12:20,222::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913745::INFO::2013-11-20 03:12:20,223::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913745::DEBUG::2013-11-20 03:12:20,223::task::495::TaskManager.Task::(_debug) Task 4670e633-10c5-4781-bac4-65ebbc0be1d9: moving from state init -> state preparing >Thread-15913745::DEBUG::2013-11-20 03:12:20,224::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913745::DEBUG::2013-11-20 03:12:20,224::task::495::TaskManager.Task::(_debug) Task 4670e633-10c5-4781-bac4-65ebbc0be1d9: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913745::DEBUG::2013-11-20 03:12:20,225::task::495::TaskManager.Task::(_debug) Task 4670e633-10c5-4781-bac4-65ebbc0be1d9: moving from state preparing -> state finished >Thread-15913745::DEBUG::2013-11-20 03:12:20,225::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913745::DEBUG::2013-11-20 03:12:20,226::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913745::DEBUG::2013-11-20 03:12:20,226::task::495::TaskManager.Task::(_debug) Task 4670e633-10c5-4781-bac4-65ebbc0be1d9: ref 0 aborting False >Thread-15913745::INFO::2013-11-20 03:12:20,227::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913746::DEBUG::2013-11-20 03:12:20,231::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913746::INFO::2013-11-20 03:12:20,231::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913746::DEBUG::2013-11-20 03:12:20,232::task::495::TaskManager.Task::(_debug) Task 8905db49-258d-41e7-a400-b95a54a9a7da: moving from state init -> state preparing >Thread-15913746::DEBUG::2013-11-20 03:12:20,233::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9ca352ff-670d-4f79-974b-7d1d0c044fda`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913746::DEBUG::2013-11-20 03:12:20,233::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913746::DEBUG::2013-11-20 03:12:20,234::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913746::DEBUG::2013-11-20 03:12:20,234::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9ca352ff-670d-4f79-974b-7d1d0c044fda`::Granted request >Thread-15913746::DEBUG::2013-11-20 03:12:20,235::task::495::TaskManager.Task::(_debug) Task 8905db49-258d-41e7-a400-b95a54a9a7da: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913746::DEBUG::2013-11-20 03:12:20,236::task::495::TaskManager.Task::(_debug) Task 8905db49-258d-41e7-a400-b95a54a9a7da: ref 1 aborting False >Thread-15913746::WARNING::2013-11-20 03:12:20,237::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913746::DEBUG::2013-11-20 03:12:20,238::task::495::TaskManager.Task::(_debug) Task 8905db49-258d-41e7-a400-b95a54a9a7da: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913746::DEBUG::2013-11-20 03:12:20,239::task::495::TaskManager.Task::(_debug) Task 8905db49-258d-41e7-a400-b95a54a9a7da: moving from state preparing -> state finished >Thread-15913746::DEBUG::2013-11-20 03:12:20,239::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913746::DEBUG::2013-11-20 03:12:20,239::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913746::DEBUG::2013-11-20 03:12:20,240::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913746::DEBUG::2013-11-20 03:12:20,240::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913746::DEBUG::2013-11-20 03:12:20,240::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913746::DEBUG::2013-11-20 03:12:20,241::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913746::DEBUG::2013-11-20 03:12:20,241::task::495::TaskManager.Task::(_debug) Task 8905db49-258d-41e7-a400-b95a54a9a7da: ref 0 aborting False >Thread-15913746::INFO::2013-11-20 03:12:20,241::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913747::INFO::2013-11-20 03:12:20,916::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913747::DEBUG::2013-11-20 03:12:20,917::task::495::TaskManager.Task::(_debug) Task 721af1fb-84d5-453b-b6c1-0ff53d4d863c: moving from state init -> state preparing >Thread-15913747::DEBUG::2013-11-20 03:12:20,917::task::495::TaskManager.Task::(_debug) Task 721af1fb-84d5-453b-b6c1-0ff53d4d863c: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000274896621704', 'lastCheck': 1384917139.469301, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0367619991302', 'lastCheck': 1384917136.5124249, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0399279594421', 'lastCheck': 1384917137.5164299, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000701904296875', 'lastCheck': 1384917138.4712701, 'valid': True, 'code': 0}} >Thread-15913747::DEBUG::2013-11-20 03:12:20,918::task::495::TaskManager.Task::(_debug) Task 721af1fb-84d5-453b-b6c1-0ff53d4d863c: moving from state preparing -> state finished >Thread-15913747::DEBUG::2013-11-20 03:12:20,918::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913747::DEBUG::2013-11-20 03:12:20,918::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913747::DEBUG::2013-11-20 03:12:20,919::task::495::TaskManager.Task::(_debug) Task 721af1fb-84d5-453b-b6c1-0ff53d4d863c: ref 0 aborting False >Thread-15913747::INFO::2013-11-20 03:12:20,919::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000274896621704', 'lastCheck': 1384917139.469301, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0367619991302', 'lastCheck': 1384917136.5124249, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0399279594421', 'lastCheck': 1384917137.5164299, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000701904296875', 'lastCheck': 1384917138.4712701, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:12:20,939::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:12:21,389::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0468736 s, 21.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:12:23,399::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:12:24,417::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0547368 s, 18.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:12:26,428::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:12:26,966::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0467099 s, 21.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:12:28,976::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:12:29,384::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0499612 s, 20.5 MB/s\n'; <rc> = 0 >Thread-15913752::DEBUG::2013-11-20 03:12:30,793::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913752::INFO::2013-11-20 03:12:30,794::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913752::DEBUG::2013-11-20 03:12:30,795::task::495::TaskManager.Task::(_debug) Task 2555e5ca-d16d-4fe7-a036-bea06b403f30: moving from state init -> state preparing >Thread-15913752::DEBUG::2013-11-20 03:12:30,795::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913752::DEBUG::2013-11-20 03:12:30,796::task::495::TaskManager.Task::(_debug) Task 2555e5ca-d16d-4fe7-a036-bea06b403f30: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913752::DEBUG::2013-11-20 03:12:30,796::task::495::TaskManager.Task::(_debug) Task 2555e5ca-d16d-4fe7-a036-bea06b403f30: moving from state preparing -> state finished >Thread-15913752::DEBUG::2013-11-20 03:12:30,797::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913752::DEBUG::2013-11-20 03:12:30,797::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913752::DEBUG::2013-11-20 03:12:30,797::task::495::TaskManager.Task::(_debug) Task 2555e5ca-d16d-4fe7-a036-bea06b403f30: ref 0 aborting False >Thread-15913752::INFO::2013-11-20 03:12:30,798::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913753::DEBUG::2013-11-20 03:12:30,801::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913753::INFO::2013-11-20 03:12:30,802::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913753::DEBUG::2013-11-20 03:12:30,802::task::495::TaskManager.Task::(_debug) Task 9a81aeb6-a106-454d-954a-ffcce1f42894: moving from state init -> state preparing >Thread-15913753::DEBUG::2013-11-20 03:12:30,803::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`041b2a06-a692-4c95-bb3f-d495d9f1a966`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913753::DEBUG::2013-11-20 03:12:30,803::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913753::DEBUG::2013-11-20 03:12:30,804::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913753::DEBUG::2013-11-20 03:12:30,804::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`041b2a06-a692-4c95-bb3f-d495d9f1a966`::Granted request >Thread-15913753::DEBUG::2013-11-20 03:12:30,805::task::495::TaskManager.Task::(_debug) Task 9a81aeb6-a106-454d-954a-ffcce1f42894: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913753::DEBUG::2013-11-20 03:12:30,805::task::495::TaskManager.Task::(_debug) Task 9a81aeb6-a106-454d-954a-ffcce1f42894: ref 1 aborting False >Thread-15913753::WARNING::2013-11-20 03:12:30,807::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913753::DEBUG::2013-11-20 03:12:30,807::task::495::TaskManager.Task::(_debug) Task 9a81aeb6-a106-454d-954a-ffcce1f42894: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913753::DEBUG::2013-11-20 03:12:30,808::task::495::TaskManager.Task::(_debug) Task 9a81aeb6-a106-454d-954a-ffcce1f42894: moving from state preparing -> state finished >Thread-15913753::DEBUG::2013-11-20 03:12:30,808::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913753::DEBUG::2013-11-20 03:12:30,808::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913753::DEBUG::2013-11-20 03:12:30,809::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913753::DEBUG::2013-11-20 03:12:30,809::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913753::DEBUG::2013-11-20 03:12:30,810::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913753::DEBUG::2013-11-20 03:12:30,810::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913753::DEBUG::2013-11-20 03:12:30,810::task::495::TaskManager.Task::(_debug) Task 9a81aeb6-a106-454d-954a-ffcce1f42894: ref 0 aborting False >Thread-15913753::INFO::2013-11-20 03:12:30,811::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:12:31,394::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:12:32,388::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0495414 s, 20.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:12:34,398::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913658::INFO::2013-11-20 03:12:34,616::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:12:34,617::task::495::TaskManager.Task::(_debug) Task 070031ab-9dd8-45bf-9879-c40088fb7350: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:12:34,620::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`baed3696-781b-45ea-9b4d-c183ce176a29`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:12:34,621::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:12:34,622::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:12:34,622::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`baed3696-781b-45ea-9b4d-c183ce176a29`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:12:34,624::task::495::TaskManager.Task::(_debug) Task 070031ab-9dd8-45bf-9879-c40088fb7350: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:12:34,624::task::495::TaskManager.Task::(_debug) Task 070031ab-9dd8-45bf-9879-c40088fb7350: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:12:34,625::task::495::TaskManager.Task::(_debug) Task 070031ab-9dd8-45bf-9879-c40088fb7350: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:12:34,626::task::495::TaskManager.Task::(_debug) Task 070031ab-9dd8-45bf-9879-c40088fb7350: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:12:34,626::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:12:34,627::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:12:34,627::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:12:34,627::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:12:34,628::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:12:34,628::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:12:34,628::task::495::TaskManager.Task::(_debug) Task 070031ab-9dd8-45bf-9879-c40088fb7350: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:12:34,629::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:12:34,818::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0556724 s, 18.4 MB/s\n'; <rc> = 0 >Thread-15913755::INFO::2013-11-20 03:12:36,645::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913755::DEBUG::2013-11-20 03:12:36,646::task::495::TaskManager.Task::(_debug) Task 9bf4140f-2c3a-415a-8c84-d41d31cbb530: moving from state init -> state preparing >Thread-15913755::DEBUG::2013-11-20 03:12:36,646::task::495::TaskManager.Task::(_debug) Task 9bf4140f-2c3a-415a-8c84-d41d31cbb530: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000273942947388', 'lastCheck': 1384917149.617115, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000874042510986', 'lastCheck': 1384917156.6318271, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00113201141357', 'lastCheck': 1384917147.6141601, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000324010848999', 'lastCheck': 1384917148.6131041, 'valid': True, 'code': 0}} >Thread-15913755::DEBUG::2013-11-20 03:12:36,647::task::495::TaskManager.Task::(_debug) Task 9bf4140f-2c3a-415a-8c84-d41d31cbb530: moving from state preparing -> state finished >Thread-15913755::DEBUG::2013-11-20 03:12:36,647::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913755::DEBUG::2013-11-20 03:12:36,647::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913755::DEBUG::2013-11-20 03:12:36,648::task::495::TaskManager.Task::(_debug) Task 9bf4140f-2c3a-415a-8c84-d41d31cbb530: ref 0 aborting False >Thread-15913755::INFO::2013-11-20 03:12:36,648::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000273942947388', 'lastCheck': 1384917149.617115, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000874042510986', 'lastCheck': 1384917156.6318271, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00113201141357', 'lastCheck': 1384917147.6141601, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000324010848999', 'lastCheck': 1384917148.6131041, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:12:36,832::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:12:37,272::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0486889 s, 21.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:12:39,282::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:12:39,736::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0488889 s, 20.9 MB/s\n'; <rc> = 0 >Thread-15912392::INFO::2013-11-20 03:12:41,624::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:12:41,624::task::495::TaskManager.Task::(_debug) Task 62d91d3e-9d03-4b28-b836-806068189dfb: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:12:41,633::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`8e162c8d-cd48-46e5-a2e0-503828d432ea`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:12:41,634::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:12:41,634::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:12:41,635::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`8e162c8d-cd48-46e5-a2e0-503828d432ea`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:12:41,636::task::495::TaskManager.Task::(_debug) Task 62d91d3e-9d03-4b28-b836-806068189dfb: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:12:41,636::task::495::TaskManager.Task::(_debug) Task 62d91d3e-9d03-4b28-b836-806068189dfb: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:12:41,637::task::495::TaskManager.Task::(_debug) Task 62d91d3e-9d03-4b28-b836-806068189dfb: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:12:41,637::task::495::TaskManager.Task::(_debug) Task 62d91d3e-9d03-4b28-b836-806068189dfb: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:12:41,637::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:12:41,638::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:12:41,638::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:12:41,639::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:12:41,639::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:12:41,639::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:12:41,640::task::495::TaskManager.Task::(_debug) Task 62d91d3e-9d03-4b28-b836-806068189dfb: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:12:41,640::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:12:41,746::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913758::DEBUG::2013-11-20 03:12:42,208::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913758::INFO::2013-11-20 03:12:42,210::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913758::DEBUG::2013-11-20 03:12:42,210::task::495::TaskManager.Task::(_debug) Task bc363295-d772-467d-929a-300e5f0babe7: moving from state init -> state preparing >Thread-15913758::DEBUG::2013-11-20 03:12:42,211::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913758::DEBUG::2013-11-20 03:12:42,212::task::495::TaskManager.Task::(_debug) Task bc363295-d772-467d-929a-300e5f0babe7: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913758::DEBUG::2013-11-20 03:12:42,212::task::495::TaskManager.Task::(_debug) Task bc363295-d772-467d-929a-300e5f0babe7: moving from state preparing -> state finished >Thread-15913758::DEBUG::2013-11-20 03:12:42,213::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913758::DEBUG::2013-11-20 03:12:42,213::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913758::DEBUG::2013-11-20 03:12:42,214::task::495::TaskManager.Task::(_debug) Task bc363295-d772-467d-929a-300e5f0babe7: ref 0 aborting False >Thread-15913758::INFO::2013-11-20 03:12:42,214::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913759::DEBUG::2013-11-20 03:12:42,219::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913759::INFO::2013-11-20 03:12:42,220::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913759::DEBUG::2013-11-20 03:12:42,220::task::495::TaskManager.Task::(_debug) Task dd0f2ce1-8f46-4014-942c-6df8202780a2: moving from state init -> state preparing >Thread-15913759::DEBUG::2013-11-20 03:12:42,222::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`7e6ed0e1-3837-4172-b8d8-b47d56520ce3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913759::DEBUG::2013-11-20 03:12:42,222::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913759::DEBUG::2013-11-20 03:12:42,223::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913759::DEBUG::2013-11-20 03:12:42,223::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`7e6ed0e1-3837-4172-b8d8-b47d56520ce3`::Granted request >Thread-15913759::DEBUG::2013-11-20 03:12:42,224::task::495::TaskManager.Task::(_debug) Task dd0f2ce1-8f46-4014-942c-6df8202780a2: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913759::DEBUG::2013-11-20 03:12:42,225::task::495::TaskManager.Task::(_debug) Task dd0f2ce1-8f46-4014-942c-6df8202780a2: ref 1 aborting False >Thread-15913759::WARNING::2013-11-20 03:12:42,227::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913759::DEBUG::2013-11-20 03:12:42,228::task::495::TaskManager.Task::(_debug) Task dd0f2ce1-8f46-4014-942c-6df8202780a2: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913759::DEBUG::2013-11-20 03:12:42,228::task::495::TaskManager.Task::(_debug) Task dd0f2ce1-8f46-4014-942c-6df8202780a2: moving from state preparing -> state finished >Thread-15913759::DEBUG::2013-11-20 03:12:42,229::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913759::DEBUG::2013-11-20 03:12:42,229::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913759::DEBUG::2013-11-20 03:12:42,230::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913759::DEBUG::2013-11-20 03:12:42,230::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913759::DEBUG::2013-11-20 03:12:42,231::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913759::DEBUG::2013-11-20 03:12:42,231::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913759::DEBUG::2013-11-20 03:12:42,231::task::495::TaskManager.Task::(_debug) Task dd0f2ce1-8f46-4014-942c-6df8202780a2: ref 0 aborting False >Thread-15913759::INFO::2013-11-20 03:12:42,232::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:12:42,698::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0525957 s, 19.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:12:44,708::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:12:45,104::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0465989 s, 22.0 MB/s\n'; <rc> = 0 >Thread-15913518::INFO::2013-11-20 03:12:46,950::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:12:46,951::task::495::TaskManager.Task::(_debug) Task a2b937d7-3a70-4dad-a1a7-ed6f011db46b: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:12:46,953::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`9fb59804-8580-4dc6-b206-08d36a0f1852`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:12:46,953::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:12:46,954::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:12:46,954::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`9fb59804-8580-4dc6-b206-08d36a0f1852`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:12:46,955::task::495::TaskManager.Task::(_debug) Task a2b937d7-3a70-4dad-a1a7-ed6f011db46b: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:12:46,955::task::495::TaskManager.Task::(_debug) Task a2b937d7-3a70-4dad-a1a7-ed6f011db46b: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:12:46,956::task::495::TaskManager.Task::(_debug) Task a2b937d7-3a70-4dad-a1a7-ed6f011db46b: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:12:46,956::task::495::TaskManager.Task::(_debug) Task a2b937d7-3a70-4dad-a1a7-ed6f011db46b: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:12:46,957::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:12:46,957::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:12:46,957::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:12:46,958::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:12:46,958::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:12:46,958::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:12:46,959::task::495::TaskManager.Task::(_debug) Task a2b937d7-3a70-4dad-a1a7-ed6f011db46b: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:12:46,959::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:12:46,960::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:12:46,960::task::495::TaskManager.Task::(_debug) Task ddb7315f-8ff5-4479-88f3-26e73d97e260: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:12:46,961::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`bddc9ba8-93ed-4095-9ba1-d6ab5e50f8da`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:12:46,961::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:12:46,961::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:12:46,962::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`bddc9ba8-93ed-4095-9ba1-d6ab5e50f8da`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:12:46,962::task::495::TaskManager.Task::(_debug) Task ddb7315f-8ff5-4479-88f3-26e73d97e260: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:12:46,962::task::495::TaskManager.Task::(_debug) Task ddb7315f-8ff5-4479-88f3-26e73d97e260: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:12:46,963::task::495::TaskManager.Task::(_debug) Task ddb7315f-8ff5-4479-88f3-26e73d97e260: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:12:46,963::task::495::TaskManager.Task::(_debug) Task ddb7315f-8ff5-4479-88f3-26e73d97e260: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:12:46,963::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:12:46,964::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:12:46,964::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:12:46,965::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:12:46,965::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:12:46,965::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:12:46,965::task::495::TaskManager.Task::(_debug) Task ddb7315f-8ff5-4479-88f3-26e73d97e260: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:12:46,966::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:12:47,114::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:12:47,540::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0461336 s, 22.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:12:49,550::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:12:50,575::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0494876 s, 20.7 MB/s\n'; <rc> = 0 >Thread-15913763::INFO::2013-11-20 03:12:52,219::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913763::DEBUG::2013-11-20 03:12:52,220::task::495::TaskManager.Task::(_debug) Task 46a1c140-41fa-4775-a69d-bcc0bd1dcce5: moving from state init -> state preparing >Thread-15913763::DEBUG::2013-11-20 03:12:52,221::task::495::TaskManager.Task::(_debug) Task 46a1c140-41fa-4775-a69d-bcc0bd1dcce5: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000322818756104', 'lastCheck': 1384917169.7453351, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0147819519043', 'lastCheck': 1384917166.6587031, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0144040584564', 'lastCheck': 1384917167.6542349, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000259876251221', 'lastCheck': 1384917168.633007, 'valid': True, 'code': 0}} >Thread-15913763::DEBUG::2013-11-20 03:12:52,222::task::495::TaskManager.Task::(_debug) Task 46a1c140-41fa-4775-a69d-bcc0bd1dcce5: moving from state preparing -> state finished >Thread-15913763::DEBUG::2013-11-20 03:12:52,222::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913763::DEBUG::2013-11-20 03:12:52,223::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913763::DEBUG::2013-11-20 03:12:52,223::task::495::TaskManager.Task::(_debug) Task 46a1c140-41fa-4775-a69d-bcc0bd1dcce5: ref 0 aborting False >Thread-15913763::INFO::2013-11-20 03:12:52,224::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000322818756104', 'lastCheck': 1384917169.7453351, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0147819519043', 'lastCheck': 1384917166.6587031, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0144040584564', 'lastCheck': 1384917167.6542349, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000259876251221', 'lastCheck': 1384917168.633007, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:12:52,585::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913765::DEBUG::2013-11-20 03:12:52,759::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913765::INFO::2013-11-20 03:12:52,761::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913765::DEBUG::2013-11-20 03:12:52,761::task::495::TaskManager.Task::(_debug) Task 920db819-04ec-4741-ac72-fea44ad77228: moving from state init -> state preparing >Thread-15913765::DEBUG::2013-11-20 03:12:52,762::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913765::DEBUG::2013-11-20 03:12:52,763::task::495::TaskManager.Task::(_debug) Task 920db819-04ec-4741-ac72-fea44ad77228: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913765::DEBUG::2013-11-20 03:12:52,763::task::495::TaskManager.Task::(_debug) Task 920db819-04ec-4741-ac72-fea44ad77228: moving from state preparing -> state finished >Thread-15913765::DEBUG::2013-11-20 03:12:52,764::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913765::DEBUG::2013-11-20 03:12:52,764::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913765::DEBUG::2013-11-20 03:12:52,765::task::495::TaskManager.Task::(_debug) Task 920db819-04ec-4741-ac72-fea44ad77228: ref 0 aborting False >Thread-15913765::INFO::2013-11-20 03:12:52,766::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913766::DEBUG::2013-11-20 03:12:52,770::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913766::INFO::2013-11-20 03:12:52,771::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913766::DEBUG::2013-11-20 03:12:52,771::task::495::TaskManager.Task::(_debug) Task 615ed387-8db8-4a3c-8159-19e6cdd38c94: moving from state init -> state preparing >Thread-15913766::DEBUG::2013-11-20 03:12:52,772::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2ee3df7f-acab-436e-b633-ecb24821083c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913766::DEBUG::2013-11-20 03:12:52,773::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913766::DEBUG::2013-11-20 03:12:52,774::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913766::DEBUG::2013-11-20 03:12:52,774::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2ee3df7f-acab-436e-b633-ecb24821083c`::Granted request >Thread-15913766::DEBUG::2013-11-20 03:12:52,775::task::495::TaskManager.Task::(_debug) Task 615ed387-8db8-4a3c-8159-19e6cdd38c94: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913766::DEBUG::2013-11-20 03:12:52,775::task::495::TaskManager.Task::(_debug) Task 615ed387-8db8-4a3c-8159-19e6cdd38c94: ref 1 aborting False >Thread-15913766::WARNING::2013-11-20 03:12:52,777::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913766::DEBUG::2013-11-20 03:12:52,778::task::495::TaskManager.Task::(_debug) Task 615ed387-8db8-4a3c-8159-19e6cdd38c94: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913766::DEBUG::2013-11-20 03:12:52,778::task::495::TaskManager.Task::(_debug) Task 615ed387-8db8-4a3c-8159-19e6cdd38c94: moving from state preparing -> state finished >Thread-15913766::DEBUG::2013-11-20 03:12:52,778::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913766::DEBUG::2013-11-20 03:12:52,779::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913766::DEBUG::2013-11-20 03:12:52,779::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913766::DEBUG::2013-11-20 03:12:52,780::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913766::DEBUG::2013-11-20 03:12:52,780::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913766::DEBUG::2013-11-20 03:12:52,780::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913766::DEBUG::2013-11-20 03:12:52,781::task::495::TaskManager.Task::(_debug) Task 615ed387-8db8-4a3c-8159-19e6cdd38c94: ref 0 aborting False >Thread-15913766::INFO::2013-11-20 03:12:52,781::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:12:57,328::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0851201 s, 12.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:12:59,338::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:12:59,923::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0531298 s, 19.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:13:01,933::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:13:02,348::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0478763 s, 21.4 MB/s\n'; <rc> = 0 >Thread-15913770::DEBUG::2013-11-20 03:13:02,925::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913770::INFO::2013-11-20 03:13:02,926::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913770::DEBUG::2013-11-20 03:13:02,927::task::495::TaskManager.Task::(_debug) Task e397ad10-ea60-4450-9256-bb7fc888f585: moving from state init -> state preparing >Thread-15913770::DEBUG::2013-11-20 03:13:02,928::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913770::DEBUG::2013-11-20 03:13:02,928::task::495::TaskManager.Task::(_debug) Task e397ad10-ea60-4450-9256-bb7fc888f585: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913770::DEBUG::2013-11-20 03:13:02,928::task::495::TaskManager.Task::(_debug) Task e397ad10-ea60-4450-9256-bb7fc888f585: moving from state preparing -> state finished >Thread-15913770::DEBUG::2013-11-20 03:13:02,929::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913770::DEBUG::2013-11-20 03:13:02,929::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913770::DEBUG::2013-11-20 03:13:02,929::task::495::TaskManager.Task::(_debug) Task e397ad10-ea60-4450-9256-bb7fc888f585: ref 0 aborting False >Thread-15913770::INFO::2013-11-20 03:13:02,930::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913771::DEBUG::2013-11-20 03:13:02,933::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913771::INFO::2013-11-20 03:13:02,934::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913771::DEBUG::2013-11-20 03:13:02,934::task::495::TaskManager.Task::(_debug) Task 0f721744-c90d-462d-9d03-00d1646a45a4: moving from state init -> state preparing >Thread-15913771::DEBUG::2013-11-20 03:13:02,935::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`918637d0-bc65-4c6d-84db-60ca3e4c3131`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913771::DEBUG::2013-11-20 03:13:02,935::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913771::DEBUG::2013-11-20 03:13:02,936::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913771::DEBUG::2013-11-20 03:13:02,936::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`918637d0-bc65-4c6d-84db-60ca3e4c3131`::Granted request >Thread-15913771::DEBUG::2013-11-20 03:13:02,937::task::495::TaskManager.Task::(_debug) Task 0f721744-c90d-462d-9d03-00d1646a45a4: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913771::DEBUG::2013-11-20 03:13:02,937::task::495::TaskManager.Task::(_debug) Task 0f721744-c90d-462d-9d03-00d1646a45a4: ref 1 aborting False >Thread-15913771::WARNING::2013-11-20 03:13:02,939::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913771::DEBUG::2013-11-20 03:13:02,939::task::495::TaskManager.Task::(_debug) Task 0f721744-c90d-462d-9d03-00d1646a45a4: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913771::DEBUG::2013-11-20 03:13:02,940::task::495::TaskManager.Task::(_debug) Task 0f721744-c90d-462d-9d03-00d1646a45a4: moving from state preparing -> state finished >Thread-15913771::DEBUG::2013-11-20 03:13:02,940::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913771::DEBUG::2013-11-20 03:13:02,941::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913771::DEBUG::2013-11-20 03:13:02,941::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913771::DEBUG::2013-11-20 03:13:02,942::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913771::DEBUG::2013-11-20 03:13:02,942::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913771::DEBUG::2013-11-20 03:13:02,942::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913771::DEBUG::2013-11-20 03:13:02,943::task::495::TaskManager.Task::(_debug) Task 0f721744-c90d-462d-9d03-00d1646a45a4: ref 0 aborting False >Thread-15913771::INFO::2013-11-20 03:13:02,943::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:13:04,362::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:13:04,839::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0517386 s, 19.8 MB/s\n'; <rc> = 0 >Thread-15913546::INFO::2013-11-20 03:13:04,895::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:13:04,896::task::495::TaskManager.Task::(_debug) Task d30ecc33-eeaa-4cbf-9dfc-e36f21df7dc0: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:13:04,897::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`0c87e439-5549-4041-ba33-0cde170ff139`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:13:04,897::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:13:04,898::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:13:04,898::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`0c87e439-5549-4041-ba33-0cde170ff139`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:13:04,899::task::495::TaskManager.Task::(_debug) Task d30ecc33-eeaa-4cbf-9dfc-e36f21df7dc0: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:13:04,900::task::495::TaskManager.Task::(_debug) Task d30ecc33-eeaa-4cbf-9dfc-e36f21df7dc0: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:13:04,900::task::495::TaskManager.Task::(_debug) Task d30ecc33-eeaa-4cbf-9dfc-e36f21df7dc0: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:13:04,901::task::495::TaskManager.Task::(_debug) Task d30ecc33-eeaa-4cbf-9dfc-e36f21df7dc0: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:13:04,901::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:13:04,901::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:13:04,902::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:13:04,902::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:13:04,903::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:13:04,903::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:13:04,904::task::495::TaskManager.Task::(_debug) Task d30ecc33-eeaa-4cbf-9dfc-e36f21df7dc0: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:13:04,904::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:13:04,905::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:13:04,905::task::495::TaskManager.Task::(_debug) Task f34de3d5-752a-4d6d-a7eb-d8bcf1428028: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:13:04,906::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`761d8b19-055e-4e30-9ed2-921ce051b71b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:13:04,906::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:13:04,907::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:13:04,907::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`761d8b19-055e-4e30-9ed2-921ce051b71b`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:13:04,908::task::495::TaskManager.Task::(_debug) Task f34de3d5-752a-4d6d-a7eb-d8bcf1428028: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:13:04,908::task::495::TaskManager.Task::(_debug) Task f34de3d5-752a-4d6d-a7eb-d8bcf1428028: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:13:04,909::task::495::TaskManager.Task::(_debug) Task f34de3d5-752a-4d6d-a7eb-d8bcf1428028: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:13:04,909::task::495::TaskManager.Task::(_debug) Task f34de3d5-752a-4d6d-a7eb-d8bcf1428028: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:13:04,909::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:13:04,910::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:13:04,910::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:13:04,911::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:13:04,911::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:13:04,911::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:13:04,912::task::495::TaskManager.Task::(_debug) Task f34de3d5-752a-4d6d-a7eb-d8bcf1428028: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:13:04,912::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:13:04,913::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:13:04,913::task::495::TaskManager.Task::(_debug) Task 38f6a643-52af-488d-ae7f-68d03a92cfe9: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:13:04,914::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`9f18de12-8710-4f52-8df2-509f546b63c2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:13:04,914::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:13:04,915::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:13:04,915::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`9f18de12-8710-4f52-8df2-509f546b63c2`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:13:04,916::task::495::TaskManager.Task::(_debug) Task 38f6a643-52af-488d-ae7f-68d03a92cfe9: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:13:04,916::task::495::TaskManager.Task::(_debug) Task 38f6a643-52af-488d-ae7f-68d03a92cfe9: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:13:04,917::task::495::TaskManager.Task::(_debug) Task 38f6a643-52af-488d-ae7f-68d03a92cfe9: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:13:04,917::task::495::TaskManager.Task::(_debug) Task 38f6a643-52af-488d-ae7f-68d03a92cfe9: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:13:04,917::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:13:04,918::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:13:04,918::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:13:04,919::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:13:04,919::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:13:04,920::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:13:04,920::task::495::TaskManager.Task::(_debug) Task 38f6a643-52af-488d-ae7f-68d03a92cfe9: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:13:04,920::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:13:04,921::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:13:04,921::task::495::TaskManager.Task::(_debug) Task c6526e42-864d-48f0-a2a3-ddee4feddb93: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:13:04,922::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`51927a99-ee24-4dae-be2d-fc1aa39525db`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:13:04,922::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:13:04,923::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:13:04,923::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`51927a99-ee24-4dae-be2d-fc1aa39525db`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:13:04,924::task::495::TaskManager.Task::(_debug) Task c6526e42-864d-48f0-a2a3-ddee4feddb93: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:13:04,924::task::495::TaskManager.Task::(_debug) Task c6526e42-864d-48f0-a2a3-ddee4feddb93: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:13:04,925::task::495::TaskManager.Task::(_debug) Task c6526e42-864d-48f0-a2a3-ddee4feddb93: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:13:04,925::task::495::TaskManager.Task::(_debug) Task c6526e42-864d-48f0-a2a3-ddee4feddb93: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:13:04,926::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:13:04,926::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:13:04,926::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:13:04,927::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:13:04,927::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:13:04,928::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:13:04,928::task::495::TaskManager.Task::(_debug) Task c6526e42-864d-48f0-a2a3-ddee4feddb93: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:13:04,928::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:13:06,850::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:13:07,294::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.053438 s, 19.2 MB/s\n'; <rc> = 0 >Thread-15913773::INFO::2013-11-20 03:13:07,601::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913773::DEBUG::2013-11-20 03:13:07,602::task::495::TaskManager.Task::(_debug) Task 2a529376-580e-40d6-8a90-d192dcb30820: moving from state init -> state preparing >Thread-15913773::DEBUG::2013-11-20 03:13:07,602::task::495::TaskManager.Task::(_debug) Task 2a529376-580e-40d6-8a90-d192dcb30820: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000294923782349', 'lastCheck': 1384917179.7657461, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00983905792236', 'lastCheck': 1384917186.797807, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00885200500488', 'lastCheck': 1384917177.769592, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000303030014038', 'lastCheck': 1384917178.7576039, 'valid': True, 'code': 0}} >Thread-15913773::DEBUG::2013-11-20 03:13:07,603::task::495::TaskManager.Task::(_debug) Task 2a529376-580e-40d6-8a90-d192dcb30820: moving from state preparing -> state finished >Thread-15913773::DEBUG::2013-11-20 03:13:07,603::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913773::DEBUG::2013-11-20 03:13:07,604::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913773::DEBUG::2013-11-20 03:13:07,604::task::495::TaskManager.Task::(_debug) Task 2a529376-580e-40d6-8a90-d192dcb30820: ref 0 aborting False >Thread-15913773::INFO::2013-11-20 03:13:07,604::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000294923782349', 'lastCheck': 1384917179.7657461, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00983905792236', 'lastCheck': 1384917186.797807, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00885200500488', 'lastCheck': 1384917177.769592, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000303030014038', 'lastCheck': 1384917178.7576039, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:13:09,307::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:13:10,366::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0483934 s, 21.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:13:12,376::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:13:12,843::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0519392 s, 19.7 MB/s\n'; <rc> = 0 >Thread-15913776::DEBUG::2013-11-20 03:13:13,214::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913776::INFO::2013-11-20 03:13:13,215::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913776::DEBUG::2013-11-20 03:13:13,216::task::495::TaskManager.Task::(_debug) Task 69366f69-cb68-4c83-bc8a-9ba6453cd81a: moving from state init -> state preparing >Thread-15913776::DEBUG::2013-11-20 03:13:13,217::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913776::DEBUG::2013-11-20 03:13:13,217::task::495::TaskManager.Task::(_debug) Task 69366f69-cb68-4c83-bc8a-9ba6453cd81a: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913776::DEBUG::2013-11-20 03:13:13,217::task::495::TaskManager.Task::(_debug) Task 69366f69-cb68-4c83-bc8a-9ba6453cd81a: moving from state preparing -> state finished >Thread-15913776::DEBUG::2013-11-20 03:13:13,218::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913776::DEBUG::2013-11-20 03:13:13,218::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913776::DEBUG::2013-11-20 03:13:13,218::task::495::TaskManager.Task::(_debug) Task 69366f69-cb68-4c83-bc8a-9ba6453cd81a: ref 0 aborting False >Thread-15913776::INFO::2013-11-20 03:13:13,219::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913777::DEBUG::2013-11-20 03:13:13,223::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913777::INFO::2013-11-20 03:13:13,224::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913777::DEBUG::2013-11-20 03:13:13,225::task::495::TaskManager.Task::(_debug) Task 2772eca0-b92f-40c8-973d-ca47e0f997b2: moving from state init -> state preparing >Thread-15913777::DEBUG::2013-11-20 03:13:13,226::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c4760386-b97e-4464-bffc-1c6649e3820b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913777::DEBUG::2013-11-20 03:13:13,226::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913777::DEBUG::2013-11-20 03:13:13,227::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913777::DEBUG::2013-11-20 03:13:13,227::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c4760386-b97e-4464-bffc-1c6649e3820b`::Granted request >Thread-15913777::DEBUG::2013-11-20 03:13:13,228::task::495::TaskManager.Task::(_debug) Task 2772eca0-b92f-40c8-973d-ca47e0f997b2: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913777::DEBUG::2013-11-20 03:13:13,229::task::495::TaskManager.Task::(_debug) Task 2772eca0-b92f-40c8-973d-ca47e0f997b2: ref 1 aborting False >Thread-15913777::WARNING::2013-11-20 03:13:13,230::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913777::DEBUG::2013-11-20 03:13:13,231::task::495::TaskManager.Task::(_debug) Task 2772eca0-b92f-40c8-973d-ca47e0f997b2: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913777::DEBUG::2013-11-20 03:13:13,232::task::495::TaskManager.Task::(_debug) Task 2772eca0-b92f-40c8-973d-ca47e0f997b2: moving from state preparing -> state finished >Thread-15913777::DEBUG::2013-11-20 03:13:13,232::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913777::DEBUG::2013-11-20 03:13:13,233::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913777::DEBUG::2013-11-20 03:13:13,233::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913777::DEBUG::2013-11-20 03:13:13,234::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913777::DEBUG::2013-11-20 03:13:13,234::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913777::DEBUG::2013-11-20 03:13:13,235::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913777::DEBUG::2013-11-20 03:13:13,235::task::495::TaskManager.Task::(_debug) Task 2772eca0-b92f-40c8-973d-ca47e0f997b2: ref 0 aborting False >Thread-15913777::INFO::2013-11-20 03:13:13,236::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913639::INFO::2013-11-20 03:13:14,774::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:13:14,775::task::495::TaskManager.Task::(_debug) Task e4d6b235-b3de-401c-8130-fad9a0021226: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:13:14,780::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c5401be6-11aa-4591-83d0-9215d6a7a377`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:13:14,780::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:13:14,781::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:13:14,781::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c5401be6-11aa-4591-83d0-9215d6a7a377`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:13:14,782::task::495::TaskManager.Task::(_debug) Task e4d6b235-b3de-401c-8130-fad9a0021226: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:13:14,782::task::495::TaskManager.Task::(_debug) Task e4d6b235-b3de-401c-8130-fad9a0021226: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:13:14,783::task::495::TaskManager.Task::(_debug) Task e4d6b235-b3de-401c-8130-fad9a0021226: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:13:14,783::task::495::TaskManager.Task::(_debug) Task e4d6b235-b3de-401c-8130-fad9a0021226: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:13:14,783::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:13:14,784::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:13:14,784::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:13:14,785::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:13:14,785::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:13:14,785::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:13:14,786::task::495::TaskManager.Task::(_debug) Task e4d6b235-b3de-401c-8130-fad9a0021226: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:13:14,786::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:13:14,854::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:13:19,841::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0576206 s, 17.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:13:21,852::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:13:22,544::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0526362 s, 19.5 MB/s\n'; <rc> = 0 >Thread-15913781::INFO::2013-11-20 03:13:23,015::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913781::DEBUG::2013-11-20 03:13:23,016::task::495::TaskManager.Task::(_debug) Task d9fbf123-377b-4ae2-97c5-3ead329f4440: moving from state init -> state preparing >Thread-15913781::DEBUG::2013-11-20 03:13:23,016::task::495::TaskManager.Task::(_debug) Task d9fbf123-377b-4ae2-97c5-3ead329f4440: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000207185745239', 'lastCheck': 1384917200.5524969, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0117881298065', 'lastCheck': 1384917196.8216479, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0257561206818', 'lastCheck': 1384917199.6156311, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0146560668945', 'lastCheck': 1384917199.5775321, 'valid': True, 'code': 0}} >Thread-15913781::DEBUG::2013-11-20 03:13:23,017::task::495::TaskManager.Task::(_debug) Task d9fbf123-377b-4ae2-97c5-3ead329f4440: moving from state preparing -> state finished >Thread-15913781::DEBUG::2013-11-20 03:13:23,017::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913781::DEBUG::2013-11-20 03:13:23,018::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913781::DEBUG::2013-11-20 03:13:23,018::task::495::TaskManager.Task::(_debug) Task d9fbf123-377b-4ae2-97c5-3ead329f4440: ref 0 aborting False >Thread-15913781::INFO::2013-11-20 03:13:23,019::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000207185745239', 'lastCheck': 1384917200.5524969, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0117881298065', 'lastCheck': 1384917196.8216479, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0257561206818', 'lastCheck': 1384917199.6156311, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0146560668945', 'lastCheck': 1384917199.5775321, 'valid': True, 'code': 0}} >Thread-15913783::DEBUG::2013-11-20 03:13:23,388::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913783::INFO::2013-11-20 03:13:23,389::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913783::DEBUG::2013-11-20 03:13:23,389::task::495::TaskManager.Task::(_debug) Task 7a2fb530-1f02-4794-89f2-435c6ae9d74b: moving from state init -> state preparing >Thread-15913783::DEBUG::2013-11-20 03:13:23,389::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913783::DEBUG::2013-11-20 03:13:23,390::task::495::TaskManager.Task::(_debug) Task 7a2fb530-1f02-4794-89f2-435c6ae9d74b: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913783::DEBUG::2013-11-20 03:13:23,390::task::495::TaskManager.Task::(_debug) Task 7a2fb530-1f02-4794-89f2-435c6ae9d74b: moving from state preparing -> state finished >Thread-15913783::DEBUG::2013-11-20 03:13:23,391::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913783::DEBUG::2013-11-20 03:13:23,391::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913783::DEBUG::2013-11-20 03:13:23,391::task::495::TaskManager.Task::(_debug) Task 7a2fb530-1f02-4794-89f2-435c6ae9d74b: ref 0 aborting False >Thread-15913783::INFO::2013-11-20 03:13:23,392::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913784::DEBUG::2013-11-20 03:13:23,407::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913784::INFO::2013-11-20 03:13:23,407::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913784::DEBUG::2013-11-20 03:13:23,408::task::495::TaskManager.Task::(_debug) Task 5aa4f701-53dd-4862-b85d-3477def682c5: moving from state init -> state preparing >Thread-15913784::DEBUG::2013-11-20 03:13:23,409::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f2aa0492-fb8b-4b04-9b7b-9c116a509b2b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913784::DEBUG::2013-11-20 03:13:23,409::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913784::DEBUG::2013-11-20 03:13:23,409::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913784::DEBUG::2013-11-20 03:13:23,410::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f2aa0492-fb8b-4b04-9b7b-9c116a509b2b`::Granted request >Thread-15913784::DEBUG::2013-11-20 03:13:23,410::task::495::TaskManager.Task::(_debug) Task 5aa4f701-53dd-4862-b85d-3477def682c5: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913784::DEBUG::2013-11-20 03:13:23,411::task::495::TaskManager.Task::(_debug) Task 5aa4f701-53dd-4862-b85d-3477def682c5: ref 1 aborting False >Thread-15913784::WARNING::2013-11-20 03:13:23,412::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913784::DEBUG::2013-11-20 03:13:23,413::task::495::TaskManager.Task::(_debug) Task 5aa4f701-53dd-4862-b85d-3477def682c5: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913784::DEBUG::2013-11-20 03:13:23,413::task::495::TaskManager.Task::(_debug) Task 5aa4f701-53dd-4862-b85d-3477def682c5: moving from state preparing -> state finished >Thread-15913784::DEBUG::2013-11-20 03:13:23,414::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913784::DEBUG::2013-11-20 03:13:23,414::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913784::DEBUG::2013-11-20 03:13:23,414::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913784::DEBUG::2013-11-20 03:13:23,415::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913784::DEBUG::2013-11-20 03:13:23,415::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913784::DEBUG::2013-11-20 03:13:23,415::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913784::DEBUG::2013-11-20 03:13:23,416::task::495::TaskManager.Task::(_debug) Task 5aa4f701-53dd-4862-b85d-3477def682c5: ref 0 aborting False >Thread-15913784::INFO::2013-11-20 03:13:23,416::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:13:24,563::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:13:24,998::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.046167 s, 22.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:13:27,008::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:13:29,080::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0486069 s, 21.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:13:31,090::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:13:31,475::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0473744 s, 21.6 MB/s\n'; <rc> = 0 >Thread-15913695::INFO::2013-11-20 03:13:32,134::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:13:32,135::task::495::TaskManager.Task::(_debug) Task 52d43cd9-580d-47e8-8c6e-cb80a4a8b799: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:13:32,136::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`217afc28-e62c-433c-9a67-b16f4518c5ea`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:13:32,137::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:13:32,137::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:13:32,138::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`217afc28-e62c-433c-9a67-b16f4518c5ea`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:13:32,139::task::495::TaskManager.Task::(_debug) Task 52d43cd9-580d-47e8-8c6e-cb80a4a8b799: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:13:32,139::task::495::TaskManager.Task::(_debug) Task 52d43cd9-580d-47e8-8c6e-cb80a4a8b799: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:13:32,140::task::495::TaskManager.Task::(_debug) Task 52d43cd9-580d-47e8-8c6e-cb80a4a8b799: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:13:32,140::task::495::TaskManager.Task::(_debug) Task 52d43cd9-580d-47e8-8c6e-cb80a4a8b799: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:13:32,141::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:13:32,141::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:13:32,142::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:13:32,142::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:13:32,143::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:13:32,143::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:13:32,144::task::495::TaskManager.Task::(_debug) Task 52d43cd9-580d-47e8-8c6e-cb80a4a8b799: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:13:32,145::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:13:33,486::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913788::DEBUG::2013-11-20 03:13:33,666::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913788::INFO::2013-11-20 03:13:33,667::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913788::DEBUG::2013-11-20 03:13:33,668::task::495::TaskManager.Task::(_debug) Task e14101b3-3906-4bcf-9e21-fc206242a7df: moving from state init -> state preparing >Thread-15913788::DEBUG::2013-11-20 03:13:33,669::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913788::DEBUG::2013-11-20 03:13:33,670::task::495::TaskManager.Task::(_debug) Task e14101b3-3906-4bcf-9e21-fc206242a7df: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913788::DEBUG::2013-11-20 03:13:33,671::task::495::TaskManager.Task::(_debug) Task e14101b3-3906-4bcf-9e21-fc206242a7df: moving from state preparing -> state finished >Thread-15913788::DEBUG::2013-11-20 03:13:33,672::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913788::DEBUG::2013-11-20 03:13:33,672::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913788::DEBUG::2013-11-20 03:13:33,673::task::495::TaskManager.Task::(_debug) Task e14101b3-3906-4bcf-9e21-fc206242a7df: ref 0 aborting False >Thread-15913788::INFO::2013-11-20 03:13:33,674::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913789::DEBUG::2013-11-20 03:13:33,679::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913789::INFO::2013-11-20 03:13:33,679::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913789::DEBUG::2013-11-20 03:13:33,680::task::495::TaskManager.Task::(_debug) Task 1285469f-99a9-4441-9e8b-8b4764cf1ace: moving from state init -> state preparing >Thread-15913789::DEBUG::2013-11-20 03:13:33,681::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`184e79a9-b4e1-4934-824e-34d50c0f9c5b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913789::DEBUG::2013-11-20 03:13:33,682::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913789::DEBUG::2013-11-20 03:13:33,683::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913789::DEBUG::2013-11-20 03:13:33,683::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`184e79a9-b4e1-4934-824e-34d50c0f9c5b`::Granted request >Thread-15913789::DEBUG::2013-11-20 03:13:33,684::task::495::TaskManager.Task::(_debug) Task 1285469f-99a9-4441-9e8b-8b4764cf1ace: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913789::DEBUG::2013-11-20 03:13:33,684::task::495::TaskManager.Task::(_debug) Task 1285469f-99a9-4441-9e8b-8b4764cf1ace: ref 1 aborting False >Thread-15913789::WARNING::2013-11-20 03:13:33,686::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913789::DEBUG::2013-11-20 03:13:33,687::task::495::TaskManager.Task::(_debug) Task 1285469f-99a9-4441-9e8b-8b4764cf1ace: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913789::DEBUG::2013-11-20 03:13:33,687::task::495::TaskManager.Task::(_debug) Task 1285469f-99a9-4441-9e8b-8b4764cf1ace: moving from state preparing -> state finished >Thread-15913789::DEBUG::2013-11-20 03:13:33,688::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913789::DEBUG::2013-11-20 03:13:33,688::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913789::DEBUG::2013-11-20 03:13:33,689::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913789::DEBUG::2013-11-20 03:13:33,689::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913789::DEBUG::2013-11-20 03:13:33,689::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913789::DEBUG::2013-11-20 03:13:33,690::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913789::DEBUG::2013-11-20 03:13:33,690::task::495::TaskManager.Task::(_debug) Task 1285469f-99a9-4441-9e8b-8b4764cf1ace: ref 0 aborting False >Thread-15913789::INFO::2013-11-20 03:13:33,691::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:13:34,777::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0570995 s, 17.9 MB/s\n'; <rc> = 0 >Thread-15913658::INFO::2013-11-20 03:13:35,663::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:13:35,664::task::495::TaskManager.Task::(_debug) Task eddcf48c-281b-484e-934f-32e235ba8adf: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:13:35,665::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a4c77abb-60fe-4a6a-89b7-78eeff010b18`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:13:35,666::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:13:35,667::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:13:35,667::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a4c77abb-60fe-4a6a-89b7-78eeff010b18`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:13:35,668::task::495::TaskManager.Task::(_debug) Task eddcf48c-281b-484e-934f-32e235ba8adf: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:13:35,669::task::495::TaskManager.Task::(_debug) Task eddcf48c-281b-484e-934f-32e235ba8adf: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:13:35,669::task::495::TaskManager.Task::(_debug) Task eddcf48c-281b-484e-934f-32e235ba8adf: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:13:35,670::task::495::TaskManager.Task::(_debug) Task eddcf48c-281b-484e-934f-32e235ba8adf: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:13:35,670::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:13:35,670::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:13:35,671::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:13:35,671::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:13:35,672::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:13:35,672::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:13:35,673::task::495::TaskManager.Task::(_debug) Task eddcf48c-281b-484e-934f-32e235ba8adf: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:13:35,673::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:13:36,790::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:13:37,492::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0485095 s, 21.1 MB/s\n'; <rc> = 0 >Thread-15913791::INFO::2013-11-20 03:13:38,385::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913791::DEBUG::2013-11-20 03:13:38,386::task::495::TaskManager.Task::(_debug) Task 3ab41eaa-f699-4b3b-b56f-8eaa7fe1f5e0: moving from state init -> state preparing >Thread-15913791::DEBUG::2013-11-20 03:13:38,386::task::495::TaskManager.Task::(_debug) Task 3ab41eaa-f699-4b3b-b56f-8eaa7fe1f5e0: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000235080718994', 'lastCheck': 1384917210.5646851, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0111620426178', 'lastCheck': 1384917217.6777799, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00116491317749', 'lastCheck': 1384917209.770822, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000340938568115', 'lastCheck': 1384917209.7632101, 'valid': True, 'code': 0}} >Thread-15913791::DEBUG::2013-11-20 03:13:38,387::task::495::TaskManager.Task::(_debug) Task 3ab41eaa-f699-4b3b-b56f-8eaa7fe1f5e0: moving from state preparing -> state finished >Thread-15913791::DEBUG::2013-11-20 03:13:38,388::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913791::DEBUG::2013-11-20 03:13:38,388::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913791::DEBUG::2013-11-20 03:13:38,389::task::495::TaskManager.Task::(_debug) Task 3ab41eaa-f699-4b3b-b56f-8eaa7fe1f5e0: ref 0 aborting False >Thread-15913791::INFO::2013-11-20 03:13:38,389::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000235080718994', 'lastCheck': 1384917210.5646851, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0111620426178', 'lastCheck': 1384917217.6777799, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00116491317749', 'lastCheck': 1384917209.770822, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000340938568115', 'lastCheck': 1384917209.7632101, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:13:39,502::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:13:40,465::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0474033 s, 21.6 MB/s\n'; <rc> = 0 >Thread-15912392::INFO::2013-11-20 03:13:41,771::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:13:41,771::task::495::TaskManager.Task::(_debug) Task 155be91b-80c7-4cb6-b7f8-6172c712872e: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:13:41,773::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`49375163-b214-4183-953c-fc8cbfac1924`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:13:41,773::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:13:41,774::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:13:41,774::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`49375163-b214-4183-953c-fc8cbfac1924`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:13:41,775::task::495::TaskManager.Task::(_debug) Task 155be91b-80c7-4cb6-b7f8-6172c712872e: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:13:41,775::task::495::TaskManager.Task::(_debug) Task 155be91b-80c7-4cb6-b7f8-6172c712872e: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:13:41,776::task::495::TaskManager.Task::(_debug) Task 155be91b-80c7-4cb6-b7f8-6172c712872e: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:13:41,776::task::495::TaskManager.Task::(_debug) Task 155be91b-80c7-4cb6-b7f8-6172c712872e: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:13:41,776::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:13:41,777::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:13:41,777::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:13:41,777::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:13:41,778::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:13:41,778::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:13:41,778::task::495::TaskManager.Task::(_debug) Task 155be91b-80c7-4cb6-b7f8-6172c712872e: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:13:41,779::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:13:42,475::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:13:43,031::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0562993 s, 18.2 MB/s\n'; <rc> = 0 >Thread-15913794::DEBUG::2013-11-20 03:13:44,535::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913794::INFO::2013-11-20 03:13:44,536::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913794::DEBUG::2013-11-20 03:13:44,537::task::495::TaskManager.Task::(_debug) Task d8bc8eb0-0e09-4be6-b0b3-8064101134e3: moving from state init -> state preparing >Thread-15913794::DEBUG::2013-11-20 03:13:44,537::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913794::DEBUG::2013-11-20 03:13:44,538::task::495::TaskManager.Task::(_debug) Task d8bc8eb0-0e09-4be6-b0b3-8064101134e3: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913794::DEBUG::2013-11-20 03:13:44,538::task::495::TaskManager.Task::(_debug) Task d8bc8eb0-0e09-4be6-b0b3-8064101134e3: moving from state preparing -> state finished >Thread-15913794::DEBUG::2013-11-20 03:13:44,538::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913794::DEBUG::2013-11-20 03:13:44,539::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913794::DEBUG::2013-11-20 03:13:44,539::task::495::TaskManager.Task::(_debug) Task d8bc8eb0-0e09-4be6-b0b3-8064101134e3: ref 0 aborting False >Thread-15913794::INFO::2013-11-20 03:13:44,540::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913795::DEBUG::2013-11-20 03:13:44,543::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913795::INFO::2013-11-20 03:13:44,544::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913795::DEBUG::2013-11-20 03:13:44,545::task::495::TaskManager.Task::(_debug) Task 024aade7-19cb-47dc-aba1-0311dabeedc4: moving from state init -> state preparing >Thread-15913795::DEBUG::2013-11-20 03:13:44,546::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`413a1151-ff8a-4300-a669-68038e23bcb8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913795::DEBUG::2013-11-20 03:13:44,546::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913795::DEBUG::2013-11-20 03:13:44,547::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913795::DEBUG::2013-11-20 03:13:44,547::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`413a1151-ff8a-4300-a669-68038e23bcb8`::Granted request >Thread-15913795::DEBUG::2013-11-20 03:13:44,548::task::495::TaskManager.Task::(_debug) Task 024aade7-19cb-47dc-aba1-0311dabeedc4: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913795::DEBUG::2013-11-20 03:13:44,549::task::495::TaskManager.Task::(_debug) Task 024aade7-19cb-47dc-aba1-0311dabeedc4: ref 1 aborting False >Thread-15913795::WARNING::2013-11-20 03:13:44,550::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913795::DEBUG::2013-11-20 03:13:44,551::task::495::TaskManager.Task::(_debug) Task 024aade7-19cb-47dc-aba1-0311dabeedc4: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913795::DEBUG::2013-11-20 03:13:44,552::task::495::TaskManager.Task::(_debug) Task 024aade7-19cb-47dc-aba1-0311dabeedc4: moving from state preparing -> state finished >Thread-15913795::DEBUG::2013-11-20 03:13:44,552::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913795::DEBUG::2013-11-20 03:13:44,553::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913795::DEBUG::2013-11-20 03:13:44,553::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913795::DEBUG::2013-11-20 03:13:44,554::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913795::DEBUG::2013-11-20 03:13:44,554::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913795::DEBUG::2013-11-20 03:13:44,555::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913795::DEBUG::2013-11-20 03:13:44,555::task::495::TaskManager.Task::(_debug) Task 024aade7-19cb-47dc-aba1-0311dabeedc4: ref 0 aborting False >Thread-15913795::INFO::2013-11-20 03:13:44,556::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:13:45,041::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:13:45,955::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0486552 s, 21.0 MB/s\n'; <rc> = 0 >Thread-15913518::INFO::2013-11-20 03:13:47,881::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:13:47,881::task::495::TaskManager.Task::(_debug) Task 8c77bea9-a52c-44de-a067-de1e58bf0e6d: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:13:47,888::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`7122595f-94fe-45ef-8f6b-7a520c53ae32`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:13:47,888::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:13:47,888::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:13:47,889::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`7122595f-94fe-45ef-8f6b-7a520c53ae32`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:13:47,889::task::495::TaskManager.Task::(_debug) Task 8c77bea9-a52c-44de-a067-de1e58bf0e6d: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:13:47,890::task::495::TaskManager.Task::(_debug) Task 8c77bea9-a52c-44de-a067-de1e58bf0e6d: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:13:47,890::task::495::TaskManager.Task::(_debug) Task 8c77bea9-a52c-44de-a067-de1e58bf0e6d: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:13:47,891::task::495::TaskManager.Task::(_debug) Task 8c77bea9-a52c-44de-a067-de1e58bf0e6d: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:13:47,891::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:13:47,891::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:13:47,892::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:13:47,892::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:13:47,893::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:13:47,893::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:13:47,893::task::495::TaskManager.Task::(_debug) Task 8c77bea9-a52c-44de-a067-de1e58bf0e6d: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:13:47,894::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:13:47,894::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:13:47,895::task::495::TaskManager.Task::(_debug) Task e61e5a8e-6917-4903-a820-4b67fc9d024d: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:13:47,895::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`4979ac63-cc04-4905-b4fa-74f15499566f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:13:47,896::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:13:47,896::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:13:47,896::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`4979ac63-cc04-4905-b4fa-74f15499566f`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:13:47,897::task::495::TaskManager.Task::(_debug) Task e61e5a8e-6917-4903-a820-4b67fc9d024d: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:13:47,897::task::495::TaskManager.Task::(_debug) Task e61e5a8e-6917-4903-a820-4b67fc9d024d: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:13:47,898::task::495::TaskManager.Task::(_debug) Task e61e5a8e-6917-4903-a820-4b67fc9d024d: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:13:47,898::task::495::TaskManager.Task::(_debug) Task e61e5a8e-6917-4903-a820-4b67fc9d024d: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:13:47,899::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:13:47,899::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:13:47,900::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:13:47,901::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:13:47,901::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:13:47,902::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:13:47,902::task::495::TaskManager.Task::(_debug) Task e61e5a8e-6917-4903-a820-4b67fc9d024d: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:13:47,903::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:13:47,965::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:13:48,377::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0531688 s, 19.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:13:50,387::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:13:50,824::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0530377 s, 19.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:13:52,902::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:13:53,370::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0715108 s, 14.3 MB/s\n'; <rc> = 0 >Thread-15913799::INFO::2013-11-20 03:13:54,465::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913799::DEBUG::2013-11-20 03:13:54,466::task::495::TaskManager.Task::(_debug) Task a465fb92-26f1-4bad-890f-62d1cd053ee5: moving from state init -> state preparing >Thread-15913799::DEBUG::2013-11-20 03:13:54,467::task::495::TaskManager.Task::(_debug) Task a465fb92-26f1-4bad-890f-62d1cd053ee5: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000393867492676', 'lastCheck': 1384917230.6884651, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00132703781128', 'lastCheck': 1384917227.6896369, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00111508369446', 'lastCheck': 1384917229.8180089, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000381946563721', 'lastCheck': 1384917229.780268, 'valid': True, 'code': 0}} >Thread-15913799::DEBUG::2013-11-20 03:13:54,467::task::495::TaskManager.Task::(_debug) Task a465fb92-26f1-4bad-890f-62d1cd053ee5: moving from state preparing -> state finished >Thread-15913799::DEBUG::2013-11-20 03:13:54,468::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913799::DEBUG::2013-11-20 03:13:54,468::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913799::DEBUG::2013-11-20 03:13:54,469::task::495::TaskManager.Task::(_debug) Task a465fb92-26f1-4bad-890f-62d1cd053ee5: ref 0 aborting False >Thread-15913799::INFO::2013-11-20 03:13:54,469::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000393867492676', 'lastCheck': 1384917230.6884651, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00132703781128', 'lastCheck': 1384917227.6896369, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00111508369446', 'lastCheck': 1384917229.8180089, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000381946563721', 'lastCheck': 1384917229.780268, 'valid': True, 'code': 0}} >Thread-15913801::DEBUG::2013-11-20 03:13:54,808::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913801::INFO::2013-11-20 03:13:54,809::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913801::DEBUG::2013-11-20 03:13:54,809::task::495::TaskManager.Task::(_debug) Task bab0768d-a485-4945-aa5d-363368fb102b: moving from state init -> state preparing >Thread-15913801::DEBUG::2013-11-20 03:13:54,810::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913801::DEBUG::2013-11-20 03:13:54,810::task::495::TaskManager.Task::(_debug) Task bab0768d-a485-4945-aa5d-363368fb102b: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913801::DEBUG::2013-11-20 03:13:54,811::task::495::TaskManager.Task::(_debug) Task bab0768d-a485-4945-aa5d-363368fb102b: moving from state preparing -> state finished >Thread-15913801::DEBUG::2013-11-20 03:13:54,811::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913801::DEBUG::2013-11-20 03:13:54,812::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913801::DEBUG::2013-11-20 03:13:54,812::task::495::TaskManager.Task::(_debug) Task bab0768d-a485-4945-aa5d-363368fb102b: ref 0 aborting False >Thread-15913801::INFO::2013-11-20 03:13:54,812::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913802::DEBUG::2013-11-20 03:13:54,816::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913802::INFO::2013-11-20 03:13:54,816::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913802::DEBUG::2013-11-20 03:13:54,816::task::495::TaskManager.Task::(_debug) Task 72964ec5-dc52-4a7e-93cc-0741e4746979: moving from state init -> state preparing >Thread-15913802::DEBUG::2013-11-20 03:13:54,817::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`654af5d4-e971-4744-825d-a2ed6585c2f5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913802::DEBUG::2013-11-20 03:13:54,818::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913802::DEBUG::2013-11-20 03:13:54,818::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913802::DEBUG::2013-11-20 03:13:54,818::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`654af5d4-e971-4744-825d-a2ed6585c2f5`::Granted request >Thread-15913802::DEBUG::2013-11-20 03:13:54,819::task::495::TaskManager.Task::(_debug) Task 72964ec5-dc52-4a7e-93cc-0741e4746979: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913802::DEBUG::2013-11-20 03:13:54,819::task::495::TaskManager.Task::(_debug) Task 72964ec5-dc52-4a7e-93cc-0741e4746979: ref 1 aborting False >Thread-15913802::WARNING::2013-11-20 03:13:54,820::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913802::DEBUG::2013-11-20 03:13:54,821::task::495::TaskManager.Task::(_debug) Task 72964ec5-dc52-4a7e-93cc-0741e4746979: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913802::DEBUG::2013-11-20 03:13:54,822::task::495::TaskManager.Task::(_debug) Task 72964ec5-dc52-4a7e-93cc-0741e4746979: moving from state preparing -> state finished >Thread-15913802::DEBUG::2013-11-20 03:13:54,822::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913802::DEBUG::2013-11-20 03:13:54,822::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913802::DEBUG::2013-11-20 03:13:54,823::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913802::DEBUG::2013-11-20 03:13:54,823::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913802::DEBUG::2013-11-20 03:13:54,823::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913802::DEBUG::2013-11-20 03:13:54,824::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913802::DEBUG::2013-11-20 03:13:54,824::task::495::TaskManager.Task::(_debug) Task 72964ec5-dc52-4a7e-93cc-0741e4746979: ref 0 aborting False >Thread-15913802::INFO::2013-11-20 03:13:54,825::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:13:55,381::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:13:55,816::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0556619 s, 18.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:13:57,829::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:13:58,462::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0522219 s, 19.6 MB/s\n'; <rc> = 0 >Thread-15913696::DEBUG::2013-11-20 03:13:59,908::guestIF::97::vm.Vm::(_handleMessage) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::u'None' >Dummy-15912651::DEBUG::2013-11-20 03:14:00,473::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:14:00,935::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.055671 s, 18.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:14:02,946::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:14:04,828::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0516887 s, 19.8 MB/s\n'; <rc> = 0 >Thread-15913806::DEBUG::2013-11-20 03:14:05,133::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913806::INFO::2013-11-20 03:14:05,134::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913806::DEBUG::2013-11-20 03:14:05,134::task::495::TaskManager.Task::(_debug) Task d93ea4e1-9b52-4648-9778-c42aa47314bc: moving from state init -> state preparing >Thread-15913806::DEBUG::2013-11-20 03:14:05,135::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913806::DEBUG::2013-11-20 03:14:05,136::task::495::TaskManager.Task::(_debug) Task d93ea4e1-9b52-4648-9778-c42aa47314bc: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913806::DEBUG::2013-11-20 03:14:05,136::task::495::TaskManager.Task::(_debug) Task d93ea4e1-9b52-4648-9778-c42aa47314bc: moving from state preparing -> state finished >Thread-15913806::DEBUG::2013-11-20 03:14:05,136::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913806::DEBUG::2013-11-20 03:14:05,137::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913806::DEBUG::2013-11-20 03:14:05,137::task::495::TaskManager.Task::(_debug) Task d93ea4e1-9b52-4648-9778-c42aa47314bc: ref 0 aborting False >Thread-15913806::INFO::2013-11-20 03:14:05,137::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913807::DEBUG::2013-11-20 03:14:05,141::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913807::INFO::2013-11-20 03:14:05,142::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913807::DEBUG::2013-11-20 03:14:05,142::task::495::TaskManager.Task::(_debug) Task f50dbecd-e8d7-423f-a5e5-00a716422d8e: moving from state init -> state preparing >Thread-15913807::DEBUG::2013-11-20 03:14:05,143::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ba661cfe-948e-453e-b814-e30202c5614f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913807::DEBUG::2013-11-20 03:14:05,143::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913807::DEBUG::2013-11-20 03:14:05,144::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913807::DEBUG::2013-11-20 03:14:05,144::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ba661cfe-948e-453e-b814-e30202c5614f`::Granted request >Thread-15913807::DEBUG::2013-11-20 03:14:05,145::task::495::TaskManager.Task::(_debug) Task f50dbecd-e8d7-423f-a5e5-00a716422d8e: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913807::DEBUG::2013-11-20 03:14:05,145::task::495::TaskManager.Task::(_debug) Task f50dbecd-e8d7-423f-a5e5-00a716422d8e: ref 1 aborting False >Thread-15913807::WARNING::2013-11-20 03:14:05,146::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913807::DEBUG::2013-11-20 03:14:05,147::task::495::TaskManager.Task::(_debug) Task f50dbecd-e8d7-423f-a5e5-00a716422d8e: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913807::DEBUG::2013-11-20 03:14:05,148::task::495::TaskManager.Task::(_debug) Task f50dbecd-e8d7-423f-a5e5-00a716422d8e: moving from state preparing -> state finished >Thread-15913807::DEBUG::2013-11-20 03:14:05,148::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913807::DEBUG::2013-11-20 03:14:05,148::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913807::DEBUG::2013-11-20 03:14:05,149::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913807::DEBUG::2013-11-20 03:14:05,149::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913807::DEBUG::2013-11-20 03:14:05,150::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913807::DEBUG::2013-11-20 03:14:05,150::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913807::DEBUG::2013-11-20 03:14:05,151::task::495::TaskManager.Task::(_debug) Task f50dbecd-e8d7-423f-a5e5-00a716422d8e: ref 0 aborting False >Thread-15913807::INFO::2013-11-20 03:14:05,151::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913546::INFO::2013-11-20 03:14:05,691::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:14:05,691::task::495::TaskManager.Task::(_debug) Task 4b3580b1-43aa-4f75-8536-14b61a3afa0f: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:14:05,692::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`b085fd9e-dabb-4866-aa89-f5f6f74cb881`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:14:05,693::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:14:05,693::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:14:05,694::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`b085fd9e-dabb-4866-aa89-f5f6f74cb881`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:14:05,695::task::495::TaskManager.Task::(_debug) Task 4b3580b1-43aa-4f75-8536-14b61a3afa0f: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:14:05,695::task::495::TaskManager.Task::(_debug) Task 4b3580b1-43aa-4f75-8536-14b61a3afa0f: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:14:05,696::task::495::TaskManager.Task::(_debug) Task 4b3580b1-43aa-4f75-8536-14b61a3afa0f: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:14:05,696::task::495::TaskManager.Task::(_debug) Task 4b3580b1-43aa-4f75-8536-14b61a3afa0f: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:14:05,697::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:14:05,697::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:14:05,698::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:14:05,699::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:14:05,699::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:14:05,700::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:14:05,700::task::495::TaskManager.Task::(_debug) Task 4b3580b1-43aa-4f75-8536-14b61a3afa0f: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:14:05,701::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:14:05,701::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:14:05,702::task::495::TaskManager.Task::(_debug) Task 1597042c-363b-4af6-9f51-cf53e82da1e3: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:14:05,703::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fb1bedfc-3893-4c29-b17d-b80650dab9aa`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:14:05,703::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:14:05,704::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:14:05,704::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fb1bedfc-3893-4c29-b17d-b80650dab9aa`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:14:05,706::task::495::TaskManager.Task::(_debug) Task 1597042c-363b-4af6-9f51-cf53e82da1e3: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:14:05,706::task::495::TaskManager.Task::(_debug) Task 1597042c-363b-4af6-9f51-cf53e82da1e3: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:14:05,706::task::495::TaskManager.Task::(_debug) Task 1597042c-363b-4af6-9f51-cf53e82da1e3: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:14:05,707::task::495::TaskManager.Task::(_debug) Task 1597042c-363b-4af6-9f51-cf53e82da1e3: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:14:05,707::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:14:05,708::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:14:05,708::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:14:05,708::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:14:05,709::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:14:05,709::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:14:05,709::task::495::TaskManager.Task::(_debug) Task 1597042c-363b-4af6-9f51-cf53e82da1e3: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:14:05,710::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:14:05,710::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:14:05,710::task::495::TaskManager.Task::(_debug) Task 8989dcf1-06d3-40fb-8b74-8442db72b253: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:14:05,711::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`b51a726c-7544-46f7-b36b-b592b9cc7c13`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:14:05,711::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:14:05,712::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:14:05,712::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`b51a726c-7544-46f7-b36b-b592b9cc7c13`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:14:05,713::task::495::TaskManager.Task::(_debug) Task 8989dcf1-06d3-40fb-8b74-8442db72b253: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:14:05,713::task::495::TaskManager.Task::(_debug) Task 8989dcf1-06d3-40fb-8b74-8442db72b253: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:14:05,713::task::495::TaskManager.Task::(_debug) Task 8989dcf1-06d3-40fb-8b74-8442db72b253: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:14:05,714::task::495::TaskManager.Task::(_debug) Task 8989dcf1-06d3-40fb-8b74-8442db72b253: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:14:05,714::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:14:05,714::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:14:05,715::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:14:05,715::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:14:05,716::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:14:05,716::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:14:05,717::task::495::TaskManager.Task::(_debug) Task 8989dcf1-06d3-40fb-8b74-8442db72b253: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:14:05,717::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:14:05,718::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:14:05,718::task::495::TaskManager.Task::(_debug) Task 0846367e-31ed-4c9c-bef2-5e7c4f8333fe: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:14:05,719::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`ee622640-d9e4-488f-9df0-0f48e9a0e0e4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:14:05,719::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:14:05,719::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:14:05,720::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`ee622640-d9e4-488f-9df0-0f48e9a0e0e4`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:14:05,721::task::495::TaskManager.Task::(_debug) Task 0846367e-31ed-4c9c-bef2-5e7c4f8333fe: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:14:05,721::task::495::TaskManager.Task::(_debug) Task 0846367e-31ed-4c9c-bef2-5e7c4f8333fe: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:14:05,721::task::495::TaskManager.Task::(_debug) Task 0846367e-31ed-4c9c-bef2-5e7c4f8333fe: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:14:05,722::task::495::TaskManager.Task::(_debug) Task 0846367e-31ed-4c9c-bef2-5e7c4f8333fe: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:14:05,722::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:14:05,722::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:14:05,723::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:14:05,723::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:14:05,724::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:14:05,724::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:14:05,724::task::495::TaskManager.Task::(_debug) Task 0846367e-31ed-4c9c-bef2-5e7c4f8333fe: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:14:05,725::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:14:06,838::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:14:07,232::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0490772 s, 20.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:14:09,240::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-42840::DEBUG::2013-11-20 03:14:09,800::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:14:09,998::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.219772 s, 4.7 MB/s\n'; <rc> = 0 >Thread-15913809::INFO::2013-11-20 03:14:10,086::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913809::DEBUG::2013-11-20 03:14:10,087::task::495::TaskManager.Task::(_debug) Task e3264ad1-e5be-42a5-a109-b74918579b81: moving from state init -> state preparing >Thread-15913809::DEBUG::2013-11-20 03:14:10,087::task::495::TaskManager.Task::(_debug) Task e3264ad1-e5be-42a5-a109-b74918579b81: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000308990478516', 'lastCheck': 1384917240.706538, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0117788314819', 'lastCheck': 1384917247.739619, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00097393989563', 'lastCheck': 1384917250.0140569, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000396013259888', 'lastCheck': 1384917239.790266, 'valid': True, 'code': 0}} >Thread-15913809::DEBUG::2013-11-20 03:14:10,088::task::495::TaskManager.Task::(_debug) Task e3264ad1-e5be-42a5-a109-b74918579b81: moving from state preparing -> state finished >Thread-15913809::DEBUG::2013-11-20 03:14:10,089::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913809::DEBUG::2013-11-20 03:14:10,089::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913809::DEBUG::2013-11-20 03:14:10,089::task::495::TaskManager.Task::(_debug) Task e3264ad1-e5be-42a5-a109-b74918579b81: ref 0 aborting False >Thread-15913809::INFO::2013-11-20 03:14:10,090::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000308990478516', 'lastCheck': 1384917240.706538, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0117788314819', 'lastCheck': 1384917247.739619, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00097393989563', 'lastCheck': 1384917250.0140569, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000396013259888', 'lastCheck': 1384917239.790266, 'valid': True, 'code': 0}} >Thread-42840::DEBUG::2013-11-20 03:14:10,504::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:14:12,010::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:14:12,426::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0501577 s, 20.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:14:14,435::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:14:14,818::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0486028 s, 21.1 MB/s\n'; <rc> = 0 >Thread-15913812::DEBUG::2013-11-20 03:14:15,355::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913812::INFO::2013-11-20 03:14:15,356::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913812::DEBUG::2013-11-20 03:14:15,356::task::495::TaskManager.Task::(_debug) Task cc417ad1-50a6-4d7c-af33-d446b47efec6: moving from state init -> state preparing >Thread-15913812::DEBUG::2013-11-20 03:14:15,357::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913812::DEBUG::2013-11-20 03:14:15,357::task::495::TaskManager.Task::(_debug) Task cc417ad1-50a6-4d7c-af33-d446b47efec6: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913812::DEBUG::2013-11-20 03:14:15,358::task::495::TaskManager.Task::(_debug) Task cc417ad1-50a6-4d7c-af33-d446b47efec6: moving from state preparing -> state finished >Thread-15913812::DEBUG::2013-11-20 03:14:15,358::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913812::DEBUG::2013-11-20 03:14:15,358::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913812::DEBUG::2013-11-20 03:14:15,359::task::495::TaskManager.Task::(_debug) Task cc417ad1-50a6-4d7c-af33-d446b47efec6: ref 0 aborting False >Thread-15913812::INFO::2013-11-20 03:14:15,359::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913813::DEBUG::2013-11-20 03:14:15,362::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913813::INFO::2013-11-20 03:14:15,363::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913813::DEBUG::2013-11-20 03:14:15,363::task::495::TaskManager.Task::(_debug) Task 6de57773-62b0-4ac8-9fc5-09dcffe81dbd: moving from state init -> state preparing >Thread-15913813::DEBUG::2013-11-20 03:14:15,364::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`0863bcbc-5291-4b87-9fdf-00e55184f014`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913813::DEBUG::2013-11-20 03:14:15,364::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913813::DEBUG::2013-11-20 03:14:15,364::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913813::DEBUG::2013-11-20 03:14:15,365::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`0863bcbc-5291-4b87-9fdf-00e55184f014`::Granted request >Thread-15913813::DEBUG::2013-11-20 03:14:15,365::task::495::TaskManager.Task::(_debug) Task 6de57773-62b0-4ac8-9fc5-09dcffe81dbd: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913813::DEBUG::2013-11-20 03:14:15,366::task::495::TaskManager.Task::(_debug) Task 6de57773-62b0-4ac8-9fc5-09dcffe81dbd: ref 1 aborting False >Thread-15913813::WARNING::2013-11-20 03:14:15,367::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913813::DEBUG::2013-11-20 03:14:15,368::task::495::TaskManager.Task::(_debug) Task 6de57773-62b0-4ac8-9fc5-09dcffe81dbd: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913813::DEBUG::2013-11-20 03:14:15,368::task::495::TaskManager.Task::(_debug) Task 6de57773-62b0-4ac8-9fc5-09dcffe81dbd: moving from state preparing -> state finished >Thread-15913813::DEBUG::2013-11-20 03:14:15,368::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913813::DEBUG::2013-11-20 03:14:15,369::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913813::DEBUG::2013-11-20 03:14:15,369::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913813::DEBUG::2013-11-20 03:14:15,370::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913813::DEBUG::2013-11-20 03:14:15,370::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913813::DEBUG::2013-11-20 03:14:15,370::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913813::DEBUG::2013-11-20 03:14:15,370::task::495::TaskManager.Task::(_debug) Task 6de57773-62b0-4ac8-9fc5-09dcffe81dbd: ref 0 aborting False >Thread-15913813::INFO::2013-11-20 03:14:15,371::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913639::INFO::2013-11-20 03:14:15,705::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:14:15,705::task::495::TaskManager.Task::(_debug) Task fcccf6bc-2717-41ba-aec6-37f81e7d8334: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:14:15,706::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`9e1cbe14-5883-42e2-8cc7-549356370fe0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:14:15,707::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:14:15,707::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:14:15,708::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`9e1cbe14-5883-42e2-8cc7-549356370fe0`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:14:15,708::task::495::TaskManager.Task::(_debug) Task fcccf6bc-2717-41ba-aec6-37f81e7d8334: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:14:15,709::task::495::TaskManager.Task::(_debug) Task fcccf6bc-2717-41ba-aec6-37f81e7d8334: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:14:15,709::task::495::TaskManager.Task::(_debug) Task fcccf6bc-2717-41ba-aec6-37f81e7d8334: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:14:15,710::task::495::TaskManager.Task::(_debug) Task fcccf6bc-2717-41ba-aec6-37f81e7d8334: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:14:15,710::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:14:15,711::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:14:15,711::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:14:15,712::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:14:15,712::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:14:15,713::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:14:15,713::task::495::TaskManager.Task::(_debug) Task fcccf6bc-2717-41ba-aec6-37f81e7d8334: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:14:15,713::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:14:16,828::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:14:17,241::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0545015 s, 18.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:14:19,251::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:14:20,123::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0663359 s, 15.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:14:22,135::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:14:22,541::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0461147 s, 22.2 MB/s\n'; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-20 03:14:23,427::vm::525::vm.Vm::(_rtcUpdate) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::new rtc offset 2 >libvirtEventLoop::DEBUG::2013-11-20 03:14:23,428::vm::525::vm.Vm::(_rtcUpdate) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::new rtc offset 2 >libvirtEventLoop::DEBUG::2013-11-20 03:14:23,428::vm::525::vm.Vm::(_rtcUpdate) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::new rtc offset 3602 >libvirtEventLoop::DEBUG::2013-11-20 03:14:23,429::vm::525::vm.Vm::(_rtcUpdate) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::new rtc offset 3602 >libvirtEventLoop::DEBUG::2013-11-20 03:14:23,430::vm::525::vm.Vm::(_rtcUpdate) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::new rtc offset 3602 >libvirtEventLoop::DEBUG::2013-11-20 03:14:23,430::vm::525::vm.Vm::(_rtcUpdate) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::new rtc offset 3602 >libvirtEventLoop::DEBUG::2013-11-20 03:14:23,431::vm::525::vm.Vm::(_rtcUpdate) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::new rtc offset 3602 >libvirtEventLoop::DEBUG::2013-11-20 03:14:23,432::vm::525::vm.Vm::(_rtcUpdate) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::new rtc offset 3602 >libvirtEventLoop::DEBUG::2013-11-20 03:14:23,432::vm::525::vm.Vm::(_rtcUpdate) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::new rtc offset 3602 >libvirtEventLoop::DEBUG::2013-11-20 03:14:23,433::vm::525::vm.Vm::(_rtcUpdate) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::new rtc offset 3602 >libvirtEventLoop::DEBUG::2013-11-20 03:14:23,434::vm::525::vm.Vm::(_rtcUpdate) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::new rtc offset 3602 >libvirtEventLoop::DEBUG::2013-11-20 03:14:23,434::vm::525::vm.Vm::(_rtcUpdate) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::new rtc offset 3602 >libvirtEventLoop::DEBUG::2013-11-20 03:14:23,435::vm::525::vm.Vm::(_rtcUpdate) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::new rtc offset 3602 >libvirtEventLoop::DEBUG::2013-11-20 03:14:23,436::vm::525::vm.Vm::(_rtcUpdate) vmId=`11b5c641-e4f7-487e-9758-c42cb014e2d2`::new rtc offset 3602 >Dummy-15912651::DEBUG::2013-11-20 03:14:24,551::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:14:24,979::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0453107 s, 22.6 MB/s\n'; <rc> = 0 >Thread-15913817::INFO::2013-11-20 03:14:25,618::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913817::DEBUG::2013-11-20 03:14:25,621::task::495::TaskManager.Task::(_debug) Task 7d8c0a12-e4bf-47cd-b087-93c5ef5846ab: moving from state init -> state preparing >Thread-15913817::DEBUG::2013-11-20 03:14:25,624::task::495::TaskManager.Task::(_debug) Task 7d8c0a12-e4bf-47cd-b087-93c5ef5846ab: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000268936157227', 'lastCheck': 1384917260.732682, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0204908847809', 'lastCheck': 1384917257.771805, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0150320529938', 'lastCheck': 1384917260.1991401, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000453948974609', 'lastCheck': 1384917260.599427, 'valid': True, 'code': 0}} >Thread-15913817::DEBUG::2013-11-20 03:14:25,627::task::495::TaskManager.Task::(_debug) Task 7d8c0a12-e4bf-47cd-b087-93c5ef5846ab: moving from state preparing -> state finished >Thread-15913817::DEBUG::2013-11-20 03:14:25,628::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913817::DEBUG::2013-11-20 03:14:25,629::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913817::DEBUG::2013-11-20 03:14:25,629::task::495::TaskManager.Task::(_debug) Task 7d8c0a12-e4bf-47cd-b087-93c5ef5846ab: ref 0 aborting False >Thread-15913817::INFO::2013-11-20 03:14:25,630::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000268936157227', 'lastCheck': 1384917260.732682, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0204908847809', 'lastCheck': 1384917257.771805, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0150320529938', 'lastCheck': 1384917260.1991401, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000453948974609', 'lastCheck': 1384917260.599427, 'valid': True, 'code': 0}} >Thread-15913819::DEBUG::2013-11-20 03:14:25,664::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913819::INFO::2013-11-20 03:14:25,665::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913819::DEBUG::2013-11-20 03:14:25,666::task::495::TaskManager.Task::(_debug) Task 57490b90-2a6a-4cfb-a693-105611c54b71: moving from state init -> state preparing >Thread-15913819::DEBUG::2013-11-20 03:14:25,666::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913819::DEBUG::2013-11-20 03:14:25,666::task::495::TaskManager.Task::(_debug) Task 57490b90-2a6a-4cfb-a693-105611c54b71: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913819::DEBUG::2013-11-20 03:14:25,667::task::495::TaskManager.Task::(_debug) Task 57490b90-2a6a-4cfb-a693-105611c54b71: moving from state preparing -> state finished >Thread-15913819::DEBUG::2013-11-20 03:14:25,667::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913819::DEBUG::2013-11-20 03:14:25,672::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913819::DEBUG::2013-11-20 03:14:25,673::task::495::TaskManager.Task::(_debug) Task 57490b90-2a6a-4cfb-a693-105611c54b71: ref 0 aborting False >Thread-15913819::INFO::2013-11-20 03:14:25,673::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913820::DEBUG::2013-11-20 03:14:25,680::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913820::INFO::2013-11-20 03:14:25,680::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913820::DEBUG::2013-11-20 03:14:25,686::task::495::TaskManager.Task::(_debug) Task eaf89485-21d7-41d8-b398-a71bb2a36204: moving from state init -> state preparing >Thread-15913820::DEBUG::2013-11-20 03:14:25,692::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`8efec2b2-e5ed-49d2-98d4-1e9744f3f81b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913820::DEBUG::2013-11-20 03:14:25,693::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913820::DEBUG::2013-11-20 03:14:25,693::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913820::DEBUG::2013-11-20 03:14:25,694::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`8efec2b2-e5ed-49d2-98d4-1e9744f3f81b`::Granted request >Thread-15913820::DEBUG::2013-11-20 03:14:25,694::task::495::TaskManager.Task::(_debug) Task eaf89485-21d7-41d8-b398-a71bb2a36204: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913820::DEBUG::2013-11-20 03:14:25,695::task::495::TaskManager.Task::(_debug) Task eaf89485-21d7-41d8-b398-a71bb2a36204: ref 1 aborting False >Thread-15913820::WARNING::2013-11-20 03:14:25,696::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913820::DEBUG::2013-11-20 03:14:25,705::task::495::TaskManager.Task::(_debug) Task eaf89485-21d7-41d8-b398-a71bb2a36204: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913820::DEBUG::2013-11-20 03:14:25,716::task::495::TaskManager.Task::(_debug) Task eaf89485-21d7-41d8-b398-a71bb2a36204: moving from state preparing -> state finished >Thread-15913820::DEBUG::2013-11-20 03:14:25,717::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913820::DEBUG::2013-11-20 03:14:25,717::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913820::DEBUG::2013-11-20 03:14:25,718::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913820::DEBUG::2013-11-20 03:14:25,719::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913820::DEBUG::2013-11-20 03:14:25,719::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913820::DEBUG::2013-11-20 03:14:25,720::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913820::DEBUG::2013-11-20 03:14:25,720::task::495::TaskManager.Task::(_debug) Task eaf89485-21d7-41d8-b398-a71bb2a36204: ref 0 aborting False >Thread-15913820::INFO::2013-11-20 03:14:25,721::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:14:26,988::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:14:27,370::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0481656 s, 21.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:14:29,380::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:14:29,771::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.048415 s, 21.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:14:31,781::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:14:32,352::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0497303 s, 20.6 MB/s\n'; <rc> = 0 >Thread-15913695::INFO::2013-11-20 03:14:33,570::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:14:33,571::task::495::TaskManager.Task::(_debug) Task f3a081c1-9011-44d3-8bce-6a7fe31e094b: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:14:33,572::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`52730c5d-6d62-402a-94c8-716692a34ae1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:14:33,572::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:14:33,573::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:14:33,573::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`52730c5d-6d62-402a-94c8-716692a34ae1`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:14:33,574::task::495::TaskManager.Task::(_debug) Task f3a081c1-9011-44d3-8bce-6a7fe31e094b: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:14:33,574::task::495::TaskManager.Task::(_debug) Task f3a081c1-9011-44d3-8bce-6a7fe31e094b: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:14:33,575::task::495::TaskManager.Task::(_debug) Task f3a081c1-9011-44d3-8bce-6a7fe31e094b: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:14:33,575::task::495::TaskManager.Task::(_debug) Task f3a081c1-9011-44d3-8bce-6a7fe31e094b: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:14:33,576::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:14:33,576::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:14:33,576::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:14:33,577::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:14:33,577::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:14:33,577::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:14:33,578::task::495::TaskManager.Task::(_debug) Task f3a081c1-9011-44d3-8bce-6a7fe31e094b: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:14:33,578::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:14:34,363::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:14:34,961::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.050368 s, 20.3 MB/s\n'; <rc> = 0 >Thread-15913658::INFO::2013-11-20 03:14:35,723::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:14:35,723::task::495::TaskManager.Task::(_debug) Task 14d850f7-4923-4ddd-a3ae-b018c7640949: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:14:35,724::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`84789643-678c-4109-9400-70de1573c493`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:14:35,725::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:14:35,725::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:14:35,726::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`84789643-678c-4109-9400-70de1573c493`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:14:35,731::task::495::TaskManager.Task::(_debug) Task 14d850f7-4923-4ddd-a3ae-b018c7640949: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:14:35,732::task::495::TaskManager.Task::(_debug) Task 14d850f7-4923-4ddd-a3ae-b018c7640949: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:14:35,732::task::495::TaskManager.Task::(_debug) Task 14d850f7-4923-4ddd-a3ae-b018c7640949: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:14:35,733::task::495::TaskManager.Task::(_debug) Task 14d850f7-4923-4ddd-a3ae-b018c7640949: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:14:35,733::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:14:35,733::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:14:35,734::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:14:35,734::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:14:35,734::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:14:35,735::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:14:35,735::task::495::TaskManager.Task::(_debug) Task 14d850f7-4923-4ddd-a3ae-b018c7640949: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:14:35,735::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913824::DEBUG::2013-11-20 03:14:36,278::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913824::INFO::2013-11-20 03:14:36,285::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913824::DEBUG::2013-11-20 03:14:36,285::task::495::TaskManager.Task::(_debug) Task 3d53d7ec-79f5-44c5-8687-6610326f0202: moving from state init -> state preparing >Thread-15913824::DEBUG::2013-11-20 03:14:36,286::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913824::DEBUG::2013-11-20 03:14:36,287::task::495::TaskManager.Task::(_debug) Task 3d53d7ec-79f5-44c5-8687-6610326f0202: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913824::DEBUG::2013-11-20 03:14:36,287::task::495::TaskManager.Task::(_debug) Task 3d53d7ec-79f5-44c5-8687-6610326f0202: moving from state preparing -> state finished >Thread-15913824::DEBUG::2013-11-20 03:14:36,288::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913824::DEBUG::2013-11-20 03:14:36,289::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913824::DEBUG::2013-11-20 03:14:36,289::task::495::TaskManager.Task::(_debug) Task 3d53d7ec-79f5-44c5-8687-6610326f0202: ref 0 aborting False >Thread-15913824::INFO::2013-11-20 03:14:36,290::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913825::DEBUG::2013-11-20 03:14:36,300::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913825::INFO::2013-11-20 03:14:36,301::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913825::DEBUG::2013-11-20 03:14:36,301::task::495::TaskManager.Task::(_debug) Task 7d42a2fa-70ec-4df4-8f6f-abed957a4bb4: moving from state init -> state preparing >Thread-15913825::DEBUG::2013-11-20 03:14:36,307::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f57c9ead-b869-4961-97c3-a21c5f7a090e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913825::DEBUG::2013-11-20 03:14:36,312::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913825::DEBUG::2013-11-20 03:14:36,313::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913825::DEBUG::2013-11-20 03:14:36,313::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f57c9ead-b869-4961-97c3-a21c5f7a090e`::Granted request >Thread-15913825::DEBUG::2013-11-20 03:14:36,314::task::495::TaskManager.Task::(_debug) Task 7d42a2fa-70ec-4df4-8f6f-abed957a4bb4: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913825::DEBUG::2013-11-20 03:14:36,314::task::495::TaskManager.Task::(_debug) Task 7d42a2fa-70ec-4df4-8f6f-abed957a4bb4: ref 1 aborting False >Thread-15913825::WARNING::2013-11-20 03:14:36,316::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913825::DEBUG::2013-11-20 03:14:36,321::task::495::TaskManager.Task::(_debug) Task 7d42a2fa-70ec-4df4-8f6f-abed957a4bb4: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913825::DEBUG::2013-11-20 03:14:36,329::task::495::TaskManager.Task::(_debug) Task 7d42a2fa-70ec-4df4-8f6f-abed957a4bb4: moving from state preparing -> state finished >Thread-15913825::DEBUG::2013-11-20 03:14:36,329::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913825::DEBUG::2013-11-20 03:14:36,330::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913825::DEBUG::2013-11-20 03:14:36,332::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913825::DEBUG::2013-11-20 03:14:36,333::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913825::DEBUG::2013-11-20 03:14:36,333::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913825::DEBUG::2013-11-20 03:14:36,333::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913825::DEBUG::2013-11-20 03:14:36,334::task::495::TaskManager.Task::(_debug) Task 7d42a2fa-70ec-4df4-8f6f-abed957a4bb4: ref 0 aborting False >Thread-15913825::INFO::2013-11-20 03:14:36,335::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:14:36,971::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:14:38,325::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0507918 s, 20.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:14:40,335::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:14:40,785::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0535432 s, 19.1 MB/s\n'; <rc> = 0 >Thread-15913827::INFO::2013-11-20 03:14:41,351::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913827::DEBUG::2013-11-20 03:14:41,352::task::495::TaskManager.Task::(_debug) Task d6cbc2a8-d250-464c-aa23-e29948c27796: moving from state init -> state preparing >Thread-15913827::DEBUG::2013-11-20 03:14:41,353::task::495::TaskManager.Task::(_debug) Task d6cbc2a8-d250-464c-aa23-e29948c27796: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000553131103516', 'lastCheck': 1384917280.7583539, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00115084648132', 'lastCheck': 1384917278.0801699, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00112104415894', 'lastCheck': 1384917280.4999521, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000334978103638', 'lastCheck': 1384917280.732224, 'valid': True, 'code': 0}} >Thread-15913827::DEBUG::2013-11-20 03:14:41,364::task::495::TaskManager.Task::(_debug) Task d6cbc2a8-d250-464c-aa23-e29948c27796: moving from state preparing -> state finished >Thread-15913827::DEBUG::2013-11-20 03:14:41,364::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913827::DEBUG::2013-11-20 03:14:41,365::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913827::DEBUG::2013-11-20 03:14:41,365::task::495::TaskManager.Task::(_debug) Task d6cbc2a8-d250-464c-aa23-e29948c27796: ref 0 aborting False >Thread-15913827::INFO::2013-11-20 03:14:41,366::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000553131103516', 'lastCheck': 1384917280.7583539, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00115084648132', 'lastCheck': 1384917278.0801699, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00112104415894', 'lastCheck': 1384917280.4999521, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000334978103638', 'lastCheck': 1384917280.732224, 'valid': True, 'code': 0}} >Thread-15912392::INFO::2013-11-20 03:14:42,163::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:14:42,163::task::495::TaskManager.Task::(_debug) Task 93492eac-b502-4840-8fbb-8c96b70a34af: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:14:42,165::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ebbcede3-bcce-4a79-a978-e555f851d035`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:14:42,165::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:14:42,166::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:14:42,166::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ebbcede3-bcce-4a79-a978-e555f851d035`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:14:42,167::task::495::TaskManager.Task::(_debug) Task 93492eac-b502-4840-8fbb-8c96b70a34af: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:14:42,167::task::495::TaskManager.Task::(_debug) Task 93492eac-b502-4840-8fbb-8c96b70a34af: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:14:42,168::task::495::TaskManager.Task::(_debug) Task 93492eac-b502-4840-8fbb-8c96b70a34af: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:14:42,168::task::495::TaskManager.Task::(_debug) Task 93492eac-b502-4840-8fbb-8c96b70a34af: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:14:42,169::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:14:42,169::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:14:42,170::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:14:42,170::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:14:42,171::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:14:42,171::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:14:42,172::task::495::TaskManager.Task::(_debug) Task 93492eac-b502-4840-8fbb-8c96b70a34af: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:14:42,172::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:14:42,796::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:14:43,219::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.065694 s, 15.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:14:45,303::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:14:45,743::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0561768 s, 18.2 MB/s\n'; <rc> = 0 >Thread-15913640::DEBUG::2013-11-20 03:14:46,172::guestIF::97::vm.Vm::(_handleMessage) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::u'None' >Thread-15913830::DEBUG::2013-11-20 03:14:46,762::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913830::INFO::2013-11-20 03:14:46,763::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913830::DEBUG::2013-11-20 03:14:46,764::task::495::TaskManager.Task::(_debug) Task ebf145aa-169b-4515-9083-feaf54cd5130: moving from state init -> state preparing >Thread-15913830::DEBUG::2013-11-20 03:14:46,765::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913830::DEBUG::2013-11-20 03:14:46,765::task::495::TaskManager.Task::(_debug) Task ebf145aa-169b-4515-9083-feaf54cd5130: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913830::DEBUG::2013-11-20 03:14:46,766::task::495::TaskManager.Task::(_debug) Task ebf145aa-169b-4515-9083-feaf54cd5130: moving from state preparing -> state finished >Thread-15913830::DEBUG::2013-11-20 03:14:46,766::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913830::DEBUG::2013-11-20 03:14:46,767::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913830::DEBUG::2013-11-20 03:14:46,767::task::495::TaskManager.Task::(_debug) Task ebf145aa-169b-4515-9083-feaf54cd5130: ref 0 aborting False >Thread-15913830::INFO::2013-11-20 03:14:46,767::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913831::DEBUG::2013-11-20 03:14:46,788::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913831::INFO::2013-11-20 03:14:46,789::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913831::DEBUG::2013-11-20 03:14:46,812::task::495::TaskManager.Task::(_debug) Task dc3fb59b-4ab9-43d8-80f9-8acf46d10bc7: moving from state init -> state preparing >Thread-15913831::DEBUG::2013-11-20 03:14:46,837::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`94f8706b-e624-41ca-b137-a1f95dc02242`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913831::DEBUG::2013-11-20 03:14:46,845::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913831::DEBUG::2013-11-20 03:14:46,846::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913831::DEBUG::2013-11-20 03:14:46,846::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`94f8706b-e624-41ca-b137-a1f95dc02242`::Granted request >Thread-15913831::DEBUG::2013-11-20 03:14:46,852::task::495::TaskManager.Task::(_debug) Task dc3fb59b-4ab9-43d8-80f9-8acf46d10bc7: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913831::DEBUG::2013-11-20 03:14:46,853::task::495::TaskManager.Task::(_debug) Task dc3fb59b-4ab9-43d8-80f9-8acf46d10bc7: ref 1 aborting False >Thread-15913831::WARNING::2013-11-20 03:14:46,854::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913831::DEBUG::2013-11-20 03:14:46,862::task::495::TaskManager.Task::(_debug) Task dc3fb59b-4ab9-43d8-80f9-8acf46d10bc7: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913831::DEBUG::2013-11-20 03:14:46,869::task::495::TaskManager.Task::(_debug) Task dc3fb59b-4ab9-43d8-80f9-8acf46d10bc7: moving from state preparing -> state finished >Thread-15913831::DEBUG::2013-11-20 03:14:46,869::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913831::DEBUG::2013-11-20 03:14:46,870::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913831::DEBUG::2013-11-20 03:14:46,870::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913831::DEBUG::2013-11-20 03:14:46,875::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913831::DEBUG::2013-11-20 03:14:46,876::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913831::DEBUG::2013-11-20 03:14:46,876::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913831::DEBUG::2013-11-20 03:14:46,876::task::495::TaskManager.Task::(_debug) Task dc3fb59b-4ab9-43d8-80f9-8acf46d10bc7: ref 0 aborting False >Thread-15913831::INFO::2013-11-20 03:14:46,877::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:14:47,754::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913518::INFO::2013-11-20 03:14:48,677::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:14:48,678::task::495::TaskManager.Task::(_debug) Task 654e3280-9a1f-4bce-ab21-c416b05e244a: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:14:48,680::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ce19f176-a270-4824-a6ec-7020c2201f88`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:14:48,680::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:14:48,681::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:14:48,682::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ce19f176-a270-4824-a6ec-7020c2201f88`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:14:48,683::task::495::TaskManager.Task::(_debug) Task 654e3280-9a1f-4bce-ab21-c416b05e244a: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:14:48,683::task::495::TaskManager.Task::(_debug) Task 654e3280-9a1f-4bce-ab21-c416b05e244a: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:14:48,684::task::495::TaskManager.Task::(_debug) Task 654e3280-9a1f-4bce-ab21-c416b05e244a: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:14:48,685::task::495::TaskManager.Task::(_debug) Task 654e3280-9a1f-4bce-ab21-c416b05e244a: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:14:48,685::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:14:48,686::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:14:48,686::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:14:48,687::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:14:48,687::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:14:48,692::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:14:48,693::task::495::TaskManager.Task::(_debug) Task 654e3280-9a1f-4bce-ab21-c416b05e244a: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:14:48,693::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:14:48,694::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:14:48,694::task::495::TaskManager.Task::(_debug) Task 979351d0-7dfb-4ab4-9619-7d68822dbd4f: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:14:48,695::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`af42a3de-db8c-438b-a4c0-45c253af6233`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:14:48,696::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:14:48,696::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:14:48,697::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`af42a3de-db8c-438b-a4c0-45c253af6233`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:14:48,698::task::495::TaskManager.Task::(_debug) Task 979351d0-7dfb-4ab4-9619-7d68822dbd4f: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:14:48,698::task::495::TaskManager.Task::(_debug) Task 979351d0-7dfb-4ab4-9619-7d68822dbd4f: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:14:48,699::task::495::TaskManager.Task::(_debug) Task 979351d0-7dfb-4ab4-9619-7d68822dbd4f: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:14:48,699::task::495::TaskManager.Task::(_debug) Task 979351d0-7dfb-4ab4-9619-7d68822dbd4f: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:14:48,700::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:14:48,703::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:14:48,704::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:14:48,704::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:14:48,705::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:14:48,705::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:14:48,705::task::495::TaskManager.Task::(_debug) Task 979351d0-7dfb-4ab4-9619-7d68822dbd4f: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:14:48,706::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:14:49,430::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0497402 s, 20.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:14:51,440::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:14:53,490::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0519181 s, 19.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:14:55,500::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:14:55,931::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0459256 s, 22.3 MB/s\n'; <rc> = 0 >Thread-15913835::DEBUG::2013-11-20 03:14:56,876::clientIF::54::vds::(wrapper) [192.168.100.135]::call setVmTicket with ('4760db3e-d3a8-4045-9153-12c36ec16ba2', 'ntakNoXQbUqB', '120') {} >Thread-15913835::DEBUG::2013-11-20 03:14:56,921::clientIF::59::vds::(wrapper) return setVmTicket with {'status': {'message': 'Done', 'code': 0}} >Thread-15913836::INFO::2013-11-20 03:14:57,049::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913836::DEBUG::2013-11-20 03:14:57,050::task::495::TaskManager.Task::(_debug) Task 51d0408a-bf84-4c1d-9987-f2a1a64d3120: moving from state init -> state preparing >Thread-15913836::DEBUG::2013-11-20 03:14:57,051::task::495::TaskManager.Task::(_debug) Task 51d0408a-bf84-4c1d-9987-f2a1a64d3120: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00022292137146', 'lastCheck': 1384917290.946553, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.032075881958', 'lastCheck': 1384917288.127284, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00100708007812', 'lastCheck': 1384917290.513449, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000344038009644', 'lastCheck': 1384917290.7412441, 'valid': True, 'code': 0}} >Thread-15913836::DEBUG::2013-11-20 03:14:57,063::task::495::TaskManager.Task::(_debug) Task 51d0408a-bf84-4c1d-9987-f2a1a64d3120: moving from state preparing -> state finished >Thread-15913836::DEBUG::2013-11-20 03:14:57,064::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913836::DEBUG::2013-11-20 03:14:57,065::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913836::DEBUG::2013-11-20 03:14:57,066::task::495::TaskManager.Task::(_debug) Task 51d0408a-bf84-4c1d-9987-f2a1a64d3120: ref 0 aborting False >Thread-15913836::INFO::2013-11-20 03:14:57,067::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00022292137146', 'lastCheck': 1384917290.946553, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.032075881958', 'lastCheck': 1384917288.127284, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00100708007812', 'lastCheck': 1384917290.513449, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000344038009644', 'lastCheck': 1384917290.7412441, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:14:57,941::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913838::DEBUG::2013-11-20 03:14:58,186::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913838::INFO::2013-11-20 03:14:58,188::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913838::DEBUG::2013-11-20 03:14:58,189::task::495::TaskManager.Task::(_debug) Task ffe79e12-cad8-4852-b5ae-30776134e0e6: moving from state init -> state preparing >Thread-15913838::DEBUG::2013-11-20 03:14:58,190::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913838::DEBUG::2013-11-20 03:14:58,191::task::495::TaskManager.Task::(_debug) Task ffe79e12-cad8-4852-b5ae-30776134e0e6: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913838::DEBUG::2013-11-20 03:14:58,192::task::495::TaskManager.Task::(_debug) Task ffe79e12-cad8-4852-b5ae-30776134e0e6: moving from state preparing -> state finished >Thread-15913838::DEBUG::2013-11-20 03:14:58,192::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913838::DEBUG::2013-11-20 03:14:58,193::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913838::DEBUG::2013-11-20 03:14:58,194::task::495::TaskManager.Task::(_debug) Task ffe79e12-cad8-4852-b5ae-30776134e0e6: ref 0 aborting False >Thread-15913838::INFO::2013-11-20 03:14:58,194::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913839::DEBUG::2013-11-20 03:14:58,214::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913839::INFO::2013-11-20 03:14:58,215::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913839::DEBUG::2013-11-20 03:14:58,216::task::495::TaskManager.Task::(_debug) Task 099456f1-72e0-4d0e-b99f-499ce1a73db3: moving from state init -> state preparing >Thread-15913839::DEBUG::2013-11-20 03:14:58,219::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5ad1422d-e598-4d75-a96f-f9d2317b51c1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913839::DEBUG::2013-11-20 03:14:58,220::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913839::DEBUG::2013-11-20 03:14:58,221::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913839::DEBUG::2013-11-20 03:14:58,222::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5ad1422d-e598-4d75-a96f-f9d2317b51c1`::Granted request >Thread-15913839::DEBUG::2013-11-20 03:14:58,223::task::495::TaskManager.Task::(_debug) Task 099456f1-72e0-4d0e-b99f-499ce1a73db3: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913839::DEBUG::2013-11-20 03:14:58,224::task::495::TaskManager.Task::(_debug) Task 099456f1-72e0-4d0e-b99f-499ce1a73db3: ref 1 aborting False >Thread-15913839::WARNING::2013-11-20 03:14:58,226::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913839::DEBUG::2013-11-20 03:14:58,234::task::495::TaskManager.Task::(_debug) Task 099456f1-72e0-4d0e-b99f-499ce1a73db3: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913839::DEBUG::2013-11-20 03:14:58,240::task::495::TaskManager.Task::(_debug) Task 099456f1-72e0-4d0e-b99f-499ce1a73db3: moving from state preparing -> state finished >Thread-15913839::DEBUG::2013-11-20 03:14:58,241::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913839::DEBUG::2013-11-20 03:14:58,241::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913839::DEBUG::2013-11-20 03:14:58,242::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913839::DEBUG::2013-11-20 03:14:58,243::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913839::DEBUG::2013-11-20 03:14:58,244::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913839::DEBUG::2013-11-20 03:14:58,244::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913839::DEBUG::2013-11-20 03:14:58,245::task::495::TaskManager.Task::(_debug) Task 099456f1-72e0-4d0e-b99f-499ce1a73db3: ref 0 aborting False >Thread-15913839::INFO::2013-11-20 03:14:58,251::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:14:58,440::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0796807 s, 12.9 MB/s\n'; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-20 03:14:58,585::vm::525::vm.Vm::(_rtcUpdate) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::new rtc offset 1 >libvirtEventLoop::DEBUG::2013-11-20 03:14:58,587::vm::525::vm.Vm::(_rtcUpdate) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::new rtc offset 1 >libvirtEventLoop::DEBUG::2013-11-20 03:14:58,588::vm::525::vm.Vm::(_rtcUpdate) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::new rtc offset 3601 >libvirtEventLoop::DEBUG::2013-11-20 03:14:58,588::vm::525::vm.Vm::(_rtcUpdate) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::new rtc offset 3601 >libvirtEventLoop::DEBUG::2013-11-20 03:14:58,589::vm::525::vm.Vm::(_rtcUpdate) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::new rtc offset 3601 >libvirtEventLoop::DEBUG::2013-11-20 03:14:58,591::vm::525::vm.Vm::(_rtcUpdate) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::new rtc offset 3601 >libvirtEventLoop::DEBUG::2013-11-20 03:14:58,591::vm::525::vm.Vm::(_rtcUpdate) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::new rtc offset 3601 >libvirtEventLoop::DEBUG::2013-11-20 03:14:58,617::vm::525::vm.Vm::(_rtcUpdate) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::new rtc offset 3601 >libvirtEventLoop::DEBUG::2013-11-20 03:14:58,618::vm::525::vm.Vm::(_rtcUpdate) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::new rtc offset 3601 >libvirtEventLoop::DEBUG::2013-11-20 03:14:58,619::vm::525::vm.Vm::(_rtcUpdate) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::new rtc offset 3601 >libvirtEventLoop::DEBUG::2013-11-20 03:14:58,619::vm::525::vm.Vm::(_rtcUpdate) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::new rtc offset 3601 >libvirtEventLoop::DEBUG::2013-11-20 03:14:58,620::vm::525::vm.Vm::(_rtcUpdate) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::new rtc offset 3601 >libvirtEventLoop::DEBUG::2013-11-20 03:14:58,620::vm::525::vm.Vm::(_rtcUpdate) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::new rtc offset 3601 >libvirtEventLoop::DEBUG::2013-11-20 03:14:58,621::vm::525::vm.Vm::(_rtcUpdate) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::new rtc offset 3601 >Dummy-15912651::DEBUG::2013-11-20 03:15:00,454::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >libvirtEventLoop::DEBUG::2013-11-20 03:15:00,700::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::graphics event phase 0 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >libvirtEventLoop::DEBUG::2013-11-20 03:15:00,702::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::graphics event phase 1 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >libvirtEventLoop::DEBUG::2013-11-20 03:15:00,702::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::graphics event phase 0 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >libvirtEventLoop::DEBUG::2013-11-20 03:15:00,704::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::graphics event phase 1 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >libvirtEventLoop::DEBUG::2013-11-20 03:15:00,773::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::graphics event phase 0 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >libvirtEventLoop::DEBUG::2013-11-20 03:15:00,774::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::graphics event phase 1 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >libvirtEventLoop::DEBUG::2013-11-20 03:15:00,818::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::graphics event phase 0 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >libvirtEventLoop::DEBUG::2013-11-20 03:15:00,819::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::graphics event phase 1 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >libvirtEventLoop::DEBUG::2013-11-20 03:15:00,837::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::graphics event phase 0 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >libvirtEventLoop::DEBUG::2013-11-20 03:15:00,839::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::graphics event phase 1 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >Dummy-15912651::DEBUG::2013-11-20 03:15:00,900::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0620971 s, 16.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:15:02,910::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:15:03,453::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0441118 s, 23.2 MB/s\n'; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-20 03:15:05,210::libvirtconnection::43::vm.Vm::(__eventCallback) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::graphics event phase 2 localAddr {'node': '192.168.100.133', 'service': '', 'family': 0} remoteAddr {'node': '192.168.50.51', 'service': '', 'family': 0}authScheme spice subject [] >libvirtEventLoop::DEBUG::2013-11-20 03:15:05,211::guestIF::154::vm.Vm::(desktopLock) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::desktopLock called >Dummy-15912651::DEBUG::2013-11-20 03:15:05,464::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:15:06,011::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0727686 s, 14.1 MB/s\n'; <rc> = 0 >Thread-15913546::INFO::2013-11-20 03:15:06,155::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:15:06,156::task::495::TaskManager.Task::(_debug) Task 31abe6fb-024c-4b03-9915-bbbff0bf6c75: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:15:06,159::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`6d85e528-0e5f-4574-a0dc-6893b088c7ae`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:15:06,161::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:15:06,163::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:15:06,163::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`6d85e528-0e5f-4574-a0dc-6893b088c7ae`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:15:06,164::task::495::TaskManager.Task::(_debug) Task 31abe6fb-024c-4b03-9915-bbbff0bf6c75: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:15:06,164::task::495::TaskManager.Task::(_debug) Task 31abe6fb-024c-4b03-9915-bbbff0bf6c75: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:15:06,165::task::495::TaskManager.Task::(_debug) Task 31abe6fb-024c-4b03-9915-bbbff0bf6c75: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:15:06,165::task::495::TaskManager.Task::(_debug) Task 31abe6fb-024c-4b03-9915-bbbff0bf6c75: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:15:06,166::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:15:06,166::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:15:06,166::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:15:06,167::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:15:06,167::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:15:06,167::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:15:06,168::task::495::TaskManager.Task::(_debug) Task 31abe6fb-024c-4b03-9915-bbbff0bf6c75: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:15:06,168::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:15:06,169::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:15:06,169::task::495::TaskManager.Task::(_debug) Task 89a529dc-5346-4668-a920-aea771942ef3: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:15:06,170::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`6029dfc1-ea2a-4ffe-820c-1ffb76623bbe`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:15:06,170::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:15:06,171::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:15:06,171::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`6029dfc1-ea2a-4ffe-820c-1ffb76623bbe`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:15:06,171::task::495::TaskManager.Task::(_debug) Task 89a529dc-5346-4668-a920-aea771942ef3: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:15:06,172::task::495::TaskManager.Task::(_debug) Task 89a529dc-5346-4668-a920-aea771942ef3: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:15:06,172::task::495::TaskManager.Task::(_debug) Task 89a529dc-5346-4668-a920-aea771942ef3: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:15:06,172::task::495::TaskManager.Task::(_debug) Task 89a529dc-5346-4668-a920-aea771942ef3: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:15:06,173::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:15:06,173::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:15:06,173::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:15:06,174::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:15:06,174::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:15:06,174::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:15:06,175::task::495::TaskManager.Task::(_debug) Task 89a529dc-5346-4668-a920-aea771942ef3: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:15:06,175::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:15:06,176::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:15:06,176::task::495::TaskManager.Task::(_debug) Task e1231f7a-5fef-4648-ba2e-ba084afecb44: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:15:06,176::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`0fd9d3cc-a8a3-4524-8ab3-9db10bda95cc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:15:06,177::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:15:06,177::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:15:06,177::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`0fd9d3cc-a8a3-4524-8ab3-9db10bda95cc`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:15:06,178::task::495::TaskManager.Task::(_debug) Task e1231f7a-5fef-4648-ba2e-ba084afecb44: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:15:06,178::task::495::TaskManager.Task::(_debug) Task e1231f7a-5fef-4648-ba2e-ba084afecb44: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:15:06,179::task::495::TaskManager.Task::(_debug) Task e1231f7a-5fef-4648-ba2e-ba084afecb44: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:15:06,179::task::495::TaskManager.Task::(_debug) Task e1231f7a-5fef-4648-ba2e-ba084afecb44: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:15:06,180::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:15:06,180::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:15:06,181::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:15:06,181::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:15:06,182::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:15:06,182::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:15:06,183::task::495::TaskManager.Task::(_debug) Task e1231f7a-5fef-4648-ba2e-ba084afecb44: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:15:06,184::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:15:06,185::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:15:06,185::task::495::TaskManager.Task::(_debug) Task 5996c661-df24-4516-8aeb-f5289d7f5ac5: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:15:06,187::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`e9cb2e59-1b0a-42d5-9612-1f92a5e9980e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:15:06,187::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:15:06,188::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:15:06,189::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`e9cb2e59-1b0a-42d5-9612-1f92a5e9980e`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:15:06,189::task::495::TaskManager.Task::(_debug) Task 5996c661-df24-4516-8aeb-f5289d7f5ac5: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:15:06,190::task::495::TaskManager.Task::(_debug) Task 5996c661-df24-4516-8aeb-f5289d7f5ac5: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:15:06,190::task::495::TaskManager.Task::(_debug) Task 5996c661-df24-4516-8aeb-f5289d7f5ac5: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:15:06,191::task::495::TaskManager.Task::(_debug) Task 5996c661-df24-4516-8aeb-f5289d7f5ac5: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:15:06,191::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:15:06,191::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:15:06,192::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:15:06,193::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:15:06,193::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:15:06,194::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:15:06,194::task::495::TaskManager.Task::(_debug) Task 5996c661-df24-4516-8aeb-f5289d7f5ac5: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:15:06,194::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:15:08,026::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913840::DEBUG::2013-11-20 03:15:08,457::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913840::INFO::2013-11-20 03:15:08,459::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913840::DEBUG::2013-11-20 03:15:08,460::task::495::TaskManager.Task::(_debug) Task 68fb5e28-c9fe-47c8-9267-2d9cbff09a9d: moving from state init -> state preparing >Thread-15913840::DEBUG::2013-11-20 03:15:08,462::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913840::DEBUG::2013-11-20 03:15:08,463::task::495::TaskManager.Task::(_debug) Task 68fb5e28-c9fe-47c8-9267-2d9cbff09a9d: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913840::DEBUG::2013-11-20 03:15:08,464::task::495::TaskManager.Task::(_debug) Task 68fb5e28-c9fe-47c8-9267-2d9cbff09a9d: moving from state preparing -> state finished >Thread-15913840::DEBUG::2013-11-20 03:15:08,466::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913840::DEBUG::2013-11-20 03:15:08,467::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913840::DEBUG::2013-11-20 03:15:08,468::task::495::TaskManager.Task::(_debug) Task 68fb5e28-c9fe-47c8-9267-2d9cbff09a9d: ref 0 aborting False >Thread-15913840::INFO::2013-11-20 03:15:08,470::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913841::DEBUG::2013-11-20 03:15:08,485::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913841::INFO::2013-11-20 03:15:08,486::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913841::DEBUG::2013-11-20 03:15:08,486::task::495::TaskManager.Task::(_debug) Task 367f0a68-07cd-4211-9e4d-c69eacafd03b: moving from state init -> state preparing >Thread-15913841::DEBUG::2013-11-20 03:15:08,488::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b3a1ad0e-25e9-4e60-acae-13897727a434`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913841::DEBUG::2013-11-20 03:15:08,488::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913841::DEBUG::2013-11-20 03:15:08,489::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913841::DEBUG::2013-11-20 03:15:08,490::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b3a1ad0e-25e9-4e60-acae-13897727a434`::Granted request >Thread-15913841::DEBUG::2013-11-20 03:15:08,499::task::495::TaskManager.Task::(_debug) Task 367f0a68-07cd-4211-9e4d-c69eacafd03b: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913841::DEBUG::2013-11-20 03:15:08,500::task::495::TaskManager.Task::(_debug) Task 367f0a68-07cd-4211-9e4d-c69eacafd03b: ref 1 aborting False >Thread-15913841::WARNING::2013-11-20 03:15:08,503::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913841::DEBUG::2013-11-20 03:15:08,508::task::495::TaskManager.Task::(_debug) Task 367f0a68-07cd-4211-9e4d-c69eacafd03b: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913841::DEBUG::2013-11-20 03:15:08,515::task::495::TaskManager.Task::(_debug) Task 367f0a68-07cd-4211-9e4d-c69eacafd03b: moving from state preparing -> state finished >Thread-15913841::DEBUG::2013-11-20 03:15:08,516::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913841::DEBUG::2013-11-20 03:15:08,517::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913841::DEBUG::2013-11-20 03:15:08,521::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913841::DEBUG::2013-11-20 03:15:08,522::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913841::DEBUG::2013-11-20 03:15:08,522::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913841::DEBUG::2013-11-20 03:15:08,523::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913841::DEBUG::2013-11-20 03:15:08,524::task::495::TaskManager.Task::(_debug) Task 367f0a68-07cd-4211-9e4d-c69eacafd03b: ref 0 aborting False >Thread-15913841::INFO::2013-11-20 03:15:08,524::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:15:08,637::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.131657 s, 7.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:15:10,652::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:15:11,199::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0779735 s, 13.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:15:13,214::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:15:13,709::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.073577 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:15:15,723::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913639::INFO::2013-11-20 03:15:16,251::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:15:16,253::task::495::TaskManager.Task::(_debug) Task 3a8c377d-402b-411a-adfb-5658f6d7602f: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:15:16,258::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`84f8f04c-df4f-400e-bf89-7b9eb1ffcaf2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:15:16,259::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:15:16,263::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:15:16,264::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`84f8f04c-df4f-400e-bf89-7b9eb1ffcaf2`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:15:16,269::task::495::TaskManager.Task::(_debug) Task 3a8c377d-402b-411a-adfb-5658f6d7602f: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:15:16,270::task::495::TaskManager.Task::(_debug) Task 3a8c377d-402b-411a-adfb-5658f6d7602f: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:15:16,271::task::495::TaskManager.Task::(_debug) Task 3a8c377d-402b-411a-adfb-5658f6d7602f: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:15:16,272::task::495::TaskManager.Task::(_debug) Task 3a8c377d-402b-411a-adfb-5658f6d7602f: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:15:16,274::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:15:16,281::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:15:16,284::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:15:16,285::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:15:16,287::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:15:16,288::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:15:16,289::task::495::TaskManager.Task::(_debug) Task 3a8c377d-402b-411a-adfb-5658f6d7602f: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:15:16,290::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:15:16,295::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0723194 s, 14.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:15:18,312::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913845::DEBUG::2013-11-20 03:15:18,818::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913845::INFO::2013-11-20 03:15:18,820::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913845::DEBUG::2013-11-20 03:15:18,821::task::495::TaskManager.Task::(_debug) Task 9504abf1-cbfd-472e-a612-b7cd4a03e62d: moving from state init -> state preparing >Thread-15913845::DEBUG::2013-11-20 03:15:18,821::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913845::DEBUG::2013-11-20 03:15:18,822::task::495::TaskManager.Task::(_debug) Task 9504abf1-cbfd-472e-a612-b7cd4a03e62d: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913845::DEBUG::2013-11-20 03:15:18,823::task::495::TaskManager.Task::(_debug) Task 9504abf1-cbfd-472e-a612-b7cd4a03e62d: moving from state preparing -> state finished >Thread-15913845::DEBUG::2013-11-20 03:15:18,823::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913845::DEBUG::2013-11-20 03:15:18,824::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913845::DEBUG::2013-11-20 03:15:18,824::task::495::TaskManager.Task::(_debug) Task 9504abf1-cbfd-472e-a612-b7cd4a03e62d: ref 0 aborting False >Thread-15913845::INFO::2013-11-20 03:15:18,825::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913846::DEBUG::2013-11-20 03:15:18,836::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913846::INFO::2013-11-20 03:15:18,837::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913846::DEBUG::2013-11-20 03:15:18,838::task::495::TaskManager.Task::(_debug) Task 86e736cd-cc2d-4461-a90e-fe65509a393c: moving from state init -> state preparing >Thread-15913846::DEBUG::2013-11-20 03:15:18,839::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e646df18-f3cc-4c48-86d1-b4b3a8bd091f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913846::DEBUG::2013-11-20 03:15:18,840::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913846::DEBUG::2013-11-20 03:15:18,841::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913846::DEBUG::2013-11-20 03:15:18,841::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e646df18-f3cc-4c48-86d1-b4b3a8bd091f`::Granted request >Thread-15913846::DEBUG::2013-11-20 03:15:18,846::task::495::TaskManager.Task::(_debug) Task 86e736cd-cc2d-4461-a90e-fe65509a393c: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913846::DEBUG::2013-11-20 03:15:18,848::task::495::TaskManager.Task::(_debug) Task 86e736cd-cc2d-4461-a90e-fe65509a393c: ref 1 aborting False >Thread-15913846::WARNING::2013-11-20 03:15:18,850::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913846::DEBUG::2013-11-20 03:15:18,854::task::495::TaskManager.Task::(_debug) Task 86e736cd-cc2d-4461-a90e-fe65509a393c: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913846::DEBUG::2013-11-20 03:15:18,855::task::495::TaskManager.Task::(_debug) Task 86e736cd-cc2d-4461-a90e-fe65509a393c: moving from state preparing -> state finished >Thread-15913846::DEBUG::2013-11-20 03:15:18,855::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913846::DEBUG::2013-11-20 03:15:18,856::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913846::DEBUG::2013-11-20 03:15:18,859::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913846::DEBUG::2013-11-20 03:15:18,861::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913846::DEBUG::2013-11-20 03:15:18,862::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913846::DEBUG::2013-11-20 03:15:18,863::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913846::DEBUG::2013-11-20 03:15:18,865::task::495::TaskManager.Task::(_debug) Task 86e736cd-cc2d-4461-a90e-fe65509a393c: ref 0 aborting False >Thread-15913846::INFO::2013-11-20 03:15:18,866::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:15:18,916::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.111414 s, 9.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:15:20,946::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:15:21,457::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0518454 s, 19.8 MB/s\n'; <rc> = 0 >Thread-15913848::INFO::2013-11-20 03:15:23,129::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913848::DEBUG::2013-11-20 03:15:23,130::task::495::TaskManager.Task::(_debug) Task a43b1a99-c660-4bc4-8bfb-8e8c5eb053a7: moving from state init -> state preparing >Thread-15913848::DEBUG::2013-11-20 03:15:23,131::task::495::TaskManager.Task::(_debug) Task a43b1a99-c660-4bc4-8bfb-8e8c5eb053a7: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000326871871948', 'lastCheck': 1384917321.5482509, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00178408622742', 'lastCheck': 1384917318.5490601, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00157403945923', 'lastCheck': 1384917321.1813419, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00189995765686', 'lastCheck': 1384917321.1732731, 'valid': True, 'code': 0}} >Thread-15913848::DEBUG::2013-11-20 03:15:23,138::task::495::TaskManager.Task::(_debug) Task a43b1a99-c660-4bc4-8bfb-8e8c5eb053a7: moving from state preparing -> state finished >Thread-15913848::DEBUG::2013-11-20 03:15:23,147::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913848::DEBUG::2013-11-20 03:15:23,148::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913848::DEBUG::2013-11-20 03:15:23,148::task::495::TaskManager.Task::(_debug) Task a43b1a99-c660-4bc4-8bfb-8e8c5eb053a7: ref 0 aborting False >Thread-15913848::INFO::2013-11-20 03:15:23,149::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000326871871948', 'lastCheck': 1384917321.5482509, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00178408622742', 'lastCheck': 1384917318.5490601, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00157403945923', 'lastCheck': 1384917321.1813419, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00189995765686', 'lastCheck': 1384917321.1732731, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:15:23,472::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:15:24,017::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.096318 s, 10.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:15:26,031::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:15:26,516::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0543675 s, 18.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:15:28,531::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:15:29,120::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0792786 s, 12.9 MB/s\n'; <rc> = 0 >Thread-15913851::DEBUG::2013-11-20 03:15:29,270::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913851::INFO::2013-11-20 03:15:29,272::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913851::DEBUG::2013-11-20 03:15:29,273::task::495::TaskManager.Task::(_debug) Task 16ffa110-8397-4723-87ab-1aa98091ed33: moving from state init -> state preparing >Thread-15913851::DEBUG::2013-11-20 03:15:29,275::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913851::DEBUG::2013-11-20 03:15:29,275::task::495::TaskManager.Task::(_debug) Task 16ffa110-8397-4723-87ab-1aa98091ed33: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913851::DEBUG::2013-11-20 03:15:29,276::task::495::TaskManager.Task::(_debug) Task 16ffa110-8397-4723-87ab-1aa98091ed33: moving from state preparing -> state finished >Thread-15913851::DEBUG::2013-11-20 03:15:29,277::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913851::DEBUG::2013-11-20 03:15:29,278::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913851::DEBUG::2013-11-20 03:15:29,278::task::495::TaskManager.Task::(_debug) Task 16ffa110-8397-4723-87ab-1aa98091ed33: ref 0 aborting False >Thread-15913851::INFO::2013-11-20 03:15:29,279::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913852::DEBUG::2013-11-20 03:15:29,285::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913852::INFO::2013-11-20 03:15:29,286::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913852::DEBUG::2013-11-20 03:15:29,287::task::495::TaskManager.Task::(_debug) Task f12af0f5-f87e-4140-a6b9-1b43c7d95f84: moving from state init -> state preparing >Thread-15913852::DEBUG::2013-11-20 03:15:29,288::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`32dcd40c-bfb7-4002-a120-eceea4af3514`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913852::DEBUG::2013-11-20 03:15:29,289::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913852::DEBUG::2013-11-20 03:15:29,290::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913852::DEBUG::2013-11-20 03:15:29,291::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`32dcd40c-bfb7-4002-a120-eceea4af3514`::Granted request >Thread-15913852::DEBUG::2013-11-20 03:15:29,300::task::495::TaskManager.Task::(_debug) Task f12af0f5-f87e-4140-a6b9-1b43c7d95f84: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913852::DEBUG::2013-11-20 03:15:29,301::task::495::TaskManager.Task::(_debug) Task f12af0f5-f87e-4140-a6b9-1b43c7d95f84: ref 1 aborting False >Thread-15913852::WARNING::2013-11-20 03:15:29,304::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913852::DEBUG::2013-11-20 03:15:29,305::task::495::TaskManager.Task::(_debug) Task f12af0f5-f87e-4140-a6b9-1b43c7d95f84: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913852::DEBUG::2013-11-20 03:15:29,306::task::495::TaskManager.Task::(_debug) Task f12af0f5-f87e-4140-a6b9-1b43c7d95f84: moving from state preparing -> state finished >Thread-15913852::DEBUG::2013-11-20 03:15:29,307::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913852::DEBUG::2013-11-20 03:15:29,308::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913852::DEBUG::2013-11-20 03:15:29,309::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913852::DEBUG::2013-11-20 03:15:29,309::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913852::DEBUG::2013-11-20 03:15:29,310::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913852::DEBUG::2013-11-20 03:15:29,311::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913852::DEBUG::2013-11-20 03:15:29,311::task::495::TaskManager.Task::(_debug) Task f12af0f5-f87e-4140-a6b9-1b43c7d95f84: ref 0 aborting False >Thread-15913852::INFO::2013-11-20 03:15:29,312::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:15:31,135::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:15:31,657::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0536244 s, 19.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:15:33,672::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913695::INFO::2013-11-20 03:15:34,032::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:15:34,033::task::495::TaskManager.Task::(_debug) Task 0b41141d-8c6d-4bad-b0d1-77780ec3647f: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:15:34,034::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`bfe08a4d-464a-4c48-b217-87711be8e4b2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:15:34,035::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:15:34,036::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:15:34,036::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`bfe08a4d-464a-4c48-b217-87711be8e4b2`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:15:34,037::task::495::TaskManager.Task::(_debug) Task 0b41141d-8c6d-4bad-b0d1-77780ec3647f: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:15:34,038::task::495::TaskManager.Task::(_debug) Task 0b41141d-8c6d-4bad-b0d1-77780ec3647f: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:15:34,038::task::495::TaskManager.Task::(_debug) Task 0b41141d-8c6d-4bad-b0d1-77780ec3647f: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:15:34,039::task::495::TaskManager.Task::(_debug) Task 0b41141d-8c6d-4bad-b0d1-77780ec3647f: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:15:34,039::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:15:34,040::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:15:34,040::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:15:34,041::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:15:34,041::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:15:34,042::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:15:34,042::task::495::TaskManager.Task::(_debug) Task 0b41141d-8c6d-4bad-b0d1-77780ec3647f: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:15:34,043::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:15:34,240::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0741407 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15913659::DEBUG::2013-11-20 03:15:34,963::guestIF::97::vm.Vm::(_handleMessage) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::u'None' >Dummy-15912651::DEBUG::2013-11-20 03:15:36,256::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913658::INFO::2013-11-20 03:15:36,488::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:15:36,491::task::495::TaskManager.Task::(_debug) Task b5c0537a-f6ba-4f43-a757-434a7ea3d151: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:15:36,495::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`86f0b649-205b-4504-a929-b5d67596edf5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:15:36,495::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:15:36,497::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:15:36,498::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`86f0b649-205b-4504-a929-b5d67596edf5`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:15:36,499::task::495::TaskManager.Task::(_debug) Task b5c0537a-f6ba-4f43-a757-434a7ea3d151: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:15:36,499::task::495::TaskManager.Task::(_debug) Task b5c0537a-f6ba-4f43-a757-434a7ea3d151: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:15:36,500::task::495::TaskManager.Task::(_debug) Task b5c0537a-f6ba-4f43-a757-434a7ea3d151: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:15:36,501::task::495::TaskManager.Task::(_debug) Task b5c0537a-f6ba-4f43-a757-434a7ea3d151: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:15:36,501::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:15:36,502::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:15:36,503::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:15:36,503::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:15:36,504::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:15:36,504::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:15:36,505::task::495::TaskManager.Task::(_debug) Task b5c0537a-f6ba-4f43-a757-434a7ea3d151: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:15:36,505::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:15:36,845::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0782756 s, 13.1 MB/s\n'; <rc> = 0 >Thread-15913856::INFO::2013-11-20 03:15:38,810::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913856::DEBUG::2013-11-20 03:15:38,811::task::495::TaskManager.Task::(_debug) Task a47b3566-05f5-491b-b772-4b6792476895: moving from state init -> state preparing >Thread-15913856::DEBUG::2013-11-20 03:15:38,812::task::495::TaskManager.Task::(_debug) Task a47b3566-05f5-491b-b772-4b6792476895: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000524997711182', 'lastCheck': 1384917331.771553, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150299072266', 'lastCheck': 1384917328.775408, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0134840011597', 'lastCheck': 1384917331.3896489, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00231218338013', 'lastCheck': 1384917331.3697641, 'valid': True, 'code': 0}} >Thread-15913856::DEBUG::2013-11-20 03:15:38,813::task::495::TaskManager.Task::(_debug) Task a47b3566-05f5-491b-b772-4b6792476895: moving from state preparing -> state finished >Thread-15913856::DEBUG::2013-11-20 03:15:38,814::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913856::DEBUG::2013-11-20 03:15:38,814::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913856::DEBUG::2013-11-20 03:15:38,815::task::495::TaskManager.Task::(_debug) Task a47b3566-05f5-491b-b772-4b6792476895: ref 0 aborting False >Thread-15913856::INFO::2013-11-20 03:15:38,816::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000524997711182', 'lastCheck': 1384917331.771553, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150299072266', 'lastCheck': 1384917328.775408, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0134840011597', 'lastCheck': 1384917331.3896489, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00231218338013', 'lastCheck': 1384917331.3697641, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:15:38,856::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:15:39,437::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0832579 s, 12.3 MB/s\n'; <rc> = 0 >Thread-15913858::DEBUG::2013-11-20 03:15:39,698::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913858::INFO::2013-11-20 03:15:39,700::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913858::DEBUG::2013-11-20 03:15:39,700::task::495::TaskManager.Task::(_debug) Task 6a7c176d-55fa-45ca-a23c-57b3e6bc8299: moving from state init -> state preparing >Thread-15913858::DEBUG::2013-11-20 03:15:39,701::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913858::DEBUG::2013-11-20 03:15:39,702::task::495::TaskManager.Task::(_debug) Task 6a7c176d-55fa-45ca-a23c-57b3e6bc8299: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913858::DEBUG::2013-11-20 03:15:39,703::task::495::TaskManager.Task::(_debug) Task 6a7c176d-55fa-45ca-a23c-57b3e6bc8299: moving from state preparing -> state finished >Thread-15913858::DEBUG::2013-11-20 03:15:39,704::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913858::DEBUG::2013-11-20 03:15:39,704::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913858::DEBUG::2013-11-20 03:15:39,705::task::495::TaskManager.Task::(_debug) Task 6a7c176d-55fa-45ca-a23c-57b3e6bc8299: ref 0 aborting False >Thread-15913858::INFO::2013-11-20 03:15:39,705::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913859::DEBUG::2013-11-20 03:15:39,717::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913859::INFO::2013-11-20 03:15:39,718::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913859::DEBUG::2013-11-20 03:15:39,719::task::495::TaskManager.Task::(_debug) Task 3ed1efee-2ef0-4feb-a332-554fde45dd25: moving from state init -> state preparing >Thread-15913859::DEBUG::2013-11-20 03:15:39,721::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`0eba2c31-5043-4a9c-a47e-05845c82f590`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913859::DEBUG::2013-11-20 03:15:39,722::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913859::DEBUG::2013-11-20 03:15:39,723::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913859::DEBUG::2013-11-20 03:15:39,724::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`0eba2c31-5043-4a9c-a47e-05845c82f590`::Granted request >Thread-15913859::DEBUG::2013-11-20 03:15:39,726::task::495::TaskManager.Task::(_debug) Task 3ed1efee-2ef0-4feb-a332-554fde45dd25: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913859::DEBUG::2013-11-20 03:15:39,726::task::495::TaskManager.Task::(_debug) Task 3ed1efee-2ef0-4feb-a332-554fde45dd25: ref 1 aborting False >Thread-15913859::WARNING::2013-11-20 03:15:39,728::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913859::DEBUG::2013-11-20 03:15:39,729::task::495::TaskManager.Task::(_debug) Task 3ed1efee-2ef0-4feb-a332-554fde45dd25: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913859::DEBUG::2013-11-20 03:15:39,730::task::495::TaskManager.Task::(_debug) Task 3ed1efee-2ef0-4feb-a332-554fde45dd25: moving from state preparing -> state finished >Thread-15913859::DEBUG::2013-11-20 03:15:39,731::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913859::DEBUG::2013-11-20 03:15:39,731::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913859::DEBUG::2013-11-20 03:15:39,732::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913859::DEBUG::2013-11-20 03:15:39,732::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913859::DEBUG::2013-11-20 03:15:39,733::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913859::DEBUG::2013-11-20 03:15:39,733::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913859::DEBUG::2013-11-20 03:15:39,734::task::495::TaskManager.Task::(_debug) Task 3ed1efee-2ef0-4feb-a332-554fde45dd25: ref 0 aborting False >Thread-15913859::INFO::2013-11-20 03:15:39,735::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:15:41,452::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:15:42,027::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.077363 s, 13.2 MB/s\n'; <rc> = 0 >Thread-15912392::INFO::2013-11-20 03:15:42,490::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:15:42,491::task::495::TaskManager.Task::(_debug) Task 7df535df-a13d-450a-add3-cc69a9d3b02a: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:15:42,492::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d83c17b4-61ff-4353-a8db-e3a7abaea5d8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:15:42,492::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:15:42,493::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:15:42,493::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d83c17b4-61ff-4353-a8db-e3a7abaea5d8`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:15:42,494::task::495::TaskManager.Task::(_debug) Task 7df535df-a13d-450a-add3-cc69a9d3b02a: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:15:42,494::task::495::TaskManager.Task::(_debug) Task 7df535df-a13d-450a-add3-cc69a9d3b02a: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:15:42,495::task::495::TaskManager.Task::(_debug) Task 7df535df-a13d-450a-add3-cc69a9d3b02a: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:15:42,495::task::495::TaskManager.Task::(_debug) Task 7df535df-a13d-450a-add3-cc69a9d3b02a: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:15:42,495::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:15:42,496::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:15:42,496::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:15:42,496::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:15:42,497::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:15:42,497::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:15:42,498::task::495::TaskManager.Task::(_debug) Task 7df535df-a13d-450a-add3-cc69a9d3b02a: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:15:42,498::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:15:44,041::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:15:44,499::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.059759 s, 17.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:15:46,512::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:15:47,029::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0836849 s, 12.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:15:49,040::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:15:49,563::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0564909 s, 18.1 MB/s\n'; <rc> = 0 >Thread-15913518::INFO::2013-11-20 03:15:49,770::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:15:49,771::task::495::TaskManager.Task::(_debug) Task 5e4089d4-c208-4ed4-b683-d853f0e3fd89: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:15:49,773::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e2755a76-7349-4465-8e8d-338d6a1b504a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:15:49,774::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:15:49,775::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:15:49,776::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e2755a76-7349-4465-8e8d-338d6a1b504a`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:15:49,777::task::495::TaskManager.Task::(_debug) Task 5e4089d4-c208-4ed4-b683-d853f0e3fd89: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:15:49,778::task::495::TaskManager.Task::(_debug) Task 5e4089d4-c208-4ed4-b683-d853f0e3fd89: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:15:49,779::task::495::TaskManager.Task::(_debug) Task 5e4089d4-c208-4ed4-b683-d853f0e3fd89: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:15:49,780::task::495::TaskManager.Task::(_debug) Task 5e4089d4-c208-4ed4-b683-d853f0e3fd89: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:15:49,781::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:15:49,781::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:15:49,782::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:15:49,783::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:15:49,783::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:15:49,784::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:15:49,785::task::495::TaskManager.Task::(_debug) Task 5e4089d4-c208-4ed4-b683-d853f0e3fd89: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:15:49,785::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:15:49,786::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:15:49,787::task::495::TaskManager.Task::(_debug) Task eca4fdad-d323-4fbe-a202-54cd3c181600: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:15:49,788::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b038db89-8825-40fe-8e30-509739ca9b6e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:15:49,789::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:15:49,789::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:15:49,790::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b038db89-8825-40fe-8e30-509739ca9b6e`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:15:49,791::task::495::TaskManager.Task::(_debug) Task eca4fdad-d323-4fbe-a202-54cd3c181600: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:15:49,792::task::495::TaskManager.Task::(_debug) Task eca4fdad-d323-4fbe-a202-54cd3c181600: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:15:49,792::task::495::TaskManager.Task::(_debug) Task eca4fdad-d323-4fbe-a202-54cd3c181600: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:15:49,793::task::495::TaskManager.Task::(_debug) Task eca4fdad-d323-4fbe-a202-54cd3c181600: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:15:49,794::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:15:49,794::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:15:49,795::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:15:49,796::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:15:49,796::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:15:49,797::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:15:49,798::task::495::TaskManager.Task::(_debug) Task eca4fdad-d323-4fbe-a202-54cd3c181600: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:15:49,798::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913863::DEBUG::2013-11-20 03:15:49,985::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913863::INFO::2013-11-20 03:15:49,994::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913863::DEBUG::2013-11-20 03:15:49,999::task::495::TaskManager.Task::(_debug) Task 9642c63d-ae0d-4083-980f-2d471600f076: moving from state init -> state preparing >Thread-15913863::DEBUG::2013-11-20 03:15:50,001::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913863::DEBUG::2013-11-20 03:15:50,002::task::495::TaskManager.Task::(_debug) Task 9642c63d-ae0d-4083-980f-2d471600f076: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913863::DEBUG::2013-11-20 03:15:50,003::task::495::TaskManager.Task::(_debug) Task 9642c63d-ae0d-4083-980f-2d471600f076: moving from state preparing -> state finished >Thread-15913863::DEBUG::2013-11-20 03:15:50,004::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913863::DEBUG::2013-11-20 03:15:50,004::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913863::DEBUG::2013-11-20 03:15:50,005::task::495::TaskManager.Task::(_debug) Task 9642c63d-ae0d-4083-980f-2d471600f076: ref 0 aborting False >Thread-15913863::INFO::2013-11-20 03:15:50,005::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913864::DEBUG::2013-11-20 03:15:50,010::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913864::INFO::2013-11-20 03:15:50,011::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913864::DEBUG::2013-11-20 03:15:50,011::task::495::TaskManager.Task::(_debug) Task bc984a27-c13c-4d8e-bb14-1c042818a49d: moving from state init -> state preparing >Thread-15913864::DEBUG::2013-11-20 03:15:50,012::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a9b58190-1d20-40a2-b7e1-212f579644b8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913864::DEBUG::2013-11-20 03:15:50,013::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913864::DEBUG::2013-11-20 03:15:50,014::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913864::DEBUG::2013-11-20 03:15:50,014::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a9b58190-1d20-40a2-b7e1-212f579644b8`::Granted request >Thread-15913864::DEBUG::2013-11-20 03:15:50,015::task::495::TaskManager.Task::(_debug) Task bc984a27-c13c-4d8e-bb14-1c042818a49d: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913864::DEBUG::2013-11-20 03:15:50,016::task::495::TaskManager.Task::(_debug) Task bc984a27-c13c-4d8e-bb14-1c042818a49d: ref 1 aborting False >Thread-15913864::WARNING::2013-11-20 03:15:50,018::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913864::DEBUG::2013-11-20 03:15:50,020::task::495::TaskManager.Task::(_debug) Task bc984a27-c13c-4d8e-bb14-1c042818a49d: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913864::DEBUG::2013-11-20 03:15:50,021::task::495::TaskManager.Task::(_debug) Task bc984a27-c13c-4d8e-bb14-1c042818a49d: moving from state preparing -> state finished >Thread-15913864::DEBUG::2013-11-20 03:15:50,021::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913864::DEBUG::2013-11-20 03:15:50,022::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913864::DEBUG::2013-11-20 03:15:50,022::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913864::DEBUG::2013-11-20 03:15:50,023::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913864::DEBUG::2013-11-20 03:15:50,024::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913864::DEBUG::2013-11-20 03:15:50,024::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913864::DEBUG::2013-11-20 03:15:50,025::task::495::TaskManager.Task::(_debug) Task bc984a27-c13c-4d8e-bb14-1c042818a49d: ref 0 aborting False >Thread-15913864::INFO::2013-11-20 03:15:50,025::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:15:51,578::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:15:52,131::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0691709 s, 14.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:15:54,149::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:15:54,651::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0610361 s, 16.8 MB/s\n'; <rc> = 0 >Thread-15913866::INFO::2013-11-20 03:15:54,941::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913866::DEBUG::2013-11-20 03:15:54,942::task::495::TaskManager.Task::(_debug) Task c188476a-c608-4ad9-a9b8-432e760e55a0: moving from state init -> state preparing >Thread-15913866::DEBUG::2013-11-20 03:15:54,943::task::495::TaskManager.Task::(_debug) Task c188476a-c608-4ad9-a9b8-432e760e55a0: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000339984893799', 'lastCheck': 1384917352.2572141, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00194907188416', 'lastCheck': 1384917349.262284, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0119290351868', 'lastCheck': 1384917351.8535111, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00456094741821', 'lastCheck': 1384917351.8383951, 'valid': True, 'code': 0}} >Thread-15913866::DEBUG::2013-11-20 03:15:54,945::task::495::TaskManager.Task::(_debug) Task c188476a-c608-4ad9-a9b8-432e760e55a0: moving from state preparing -> state finished >Thread-15913866::DEBUG::2013-11-20 03:15:54,946::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913866::DEBUG::2013-11-20 03:15:54,947::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913866::DEBUG::2013-11-20 03:15:54,948::task::495::TaskManager.Task::(_debug) Task c188476a-c608-4ad9-a9b8-432e760e55a0: ref 0 aborting False >Thread-15913866::INFO::2013-11-20 03:15:54,949::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000339984893799', 'lastCheck': 1384917352.2572141, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00194907188416', 'lastCheck': 1384917349.262284, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0119290351868', 'lastCheck': 1384917351.8535111, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00456094741821', 'lastCheck': 1384917351.8383951, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:15:56,661::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:15:57,158::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.073785 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:15:59,175::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:15:59,675::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0761059 s, 13.5 MB/s\n'; <rc> = 0 >Thread-15913869::DEBUG::2013-11-20 03:16:00,320::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913869::INFO::2013-11-20 03:16:00,321::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913869::DEBUG::2013-11-20 03:16:00,322::task::495::TaskManager.Task::(_debug) Task 9ea3a740-ebfb-47c0-a42d-13d29df3c2bd: moving from state init -> state preparing >Thread-15913869::DEBUG::2013-11-20 03:16:00,323::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913869::DEBUG::2013-11-20 03:16:00,324::task::495::TaskManager.Task::(_debug) Task 9ea3a740-ebfb-47c0-a42d-13d29df3c2bd: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913869::DEBUG::2013-11-20 03:16:00,324::task::495::TaskManager.Task::(_debug) Task 9ea3a740-ebfb-47c0-a42d-13d29df3c2bd: moving from state preparing -> state finished >Thread-15913869::DEBUG::2013-11-20 03:16:00,325::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913869::DEBUG::2013-11-20 03:16:00,326::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913869::DEBUG::2013-11-20 03:16:00,326::task::495::TaskManager.Task::(_debug) Task 9ea3a740-ebfb-47c0-a42d-13d29df3c2bd: ref 0 aborting False >Thread-15913869::INFO::2013-11-20 03:16:00,327::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913870::DEBUG::2013-11-20 03:16:00,332::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913870::INFO::2013-11-20 03:16:00,334::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913870::DEBUG::2013-11-20 03:16:00,335::task::495::TaskManager.Task::(_debug) Task f4a2f341-2913-4b95-a05b-e9463f2c5dc9: moving from state init -> state preparing >Thread-15913870::DEBUG::2013-11-20 03:16:00,337::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`716c5780-a27d-40fc-bb11-5719bde85fd8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913870::DEBUG::2013-11-20 03:16:00,337::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913870::DEBUG::2013-11-20 03:16:00,339::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913870::DEBUG::2013-11-20 03:16:00,340::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`716c5780-a27d-40fc-bb11-5719bde85fd8`::Granted request >Thread-15913870::DEBUG::2013-11-20 03:16:00,341::task::495::TaskManager.Task::(_debug) Task f4a2f341-2913-4b95-a05b-e9463f2c5dc9: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913870::DEBUG::2013-11-20 03:16:00,342::task::495::TaskManager.Task::(_debug) Task f4a2f341-2913-4b95-a05b-e9463f2c5dc9: ref 1 aborting False >Thread-15913870::WARNING::2013-11-20 03:16:00,345::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913870::DEBUG::2013-11-20 03:16:00,346::task::495::TaskManager.Task::(_debug) Task f4a2f341-2913-4b95-a05b-e9463f2c5dc9: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913870::DEBUG::2013-11-20 03:16:00,347::task::495::TaskManager.Task::(_debug) Task f4a2f341-2913-4b95-a05b-e9463f2c5dc9: moving from state preparing -> state finished >Thread-15913870::DEBUG::2013-11-20 03:16:00,347::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913870::DEBUG::2013-11-20 03:16:00,348::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913870::DEBUG::2013-11-20 03:16:00,348::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913870::DEBUG::2013-11-20 03:16:00,349::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913870::DEBUG::2013-11-20 03:16:00,349::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913870::DEBUG::2013-11-20 03:16:00,350::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913870::DEBUG::2013-11-20 03:16:00,350::task::495::TaskManager.Task::(_debug) Task f4a2f341-2913-4b95-a05b-e9463f2c5dc9: ref 0 aborting False >Thread-15913870::INFO::2013-11-20 03:16:00,351::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:16:01,692::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:16:02,246::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0909309 s, 11.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:16:04,262::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:16:04,752::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0763729 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:16:06,766::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913546::INFO::2013-11-20 03:16:07,019::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:16:07,022::task::495::TaskManager.Task::(_debug) Task e4fb38c5-7bed-4a6f-8b4c-b21c9a90ffd7: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:16:07,033::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`edb3a3cc-bafa-46af-aa3b-a812438cfcfb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:16:07,034::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:16:07,035::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:16:07,036::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`edb3a3cc-bafa-46af-aa3b-a812438cfcfb`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:16:07,037::task::495::TaskManager.Task::(_debug) Task e4fb38c5-7bed-4a6f-8b4c-b21c9a90ffd7: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:16:07,038::task::495::TaskManager.Task::(_debug) Task e4fb38c5-7bed-4a6f-8b4c-b21c9a90ffd7: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:16:07,039::task::495::TaskManager.Task::(_debug) Task e4fb38c5-7bed-4a6f-8b4c-b21c9a90ffd7: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:16:07,039::task::495::TaskManager.Task::(_debug) Task e4fb38c5-7bed-4a6f-8b4c-b21c9a90ffd7: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:16:07,040::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:16:07,041::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:16:07,041::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:16:07,042::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:16:07,043::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:16:07,043::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:16:07,044::task::495::TaskManager.Task::(_debug) Task e4fb38c5-7bed-4a6f-8b4c-b21c9a90ffd7: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:16:07,044::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:16:07,045::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:16:07,046::task::495::TaskManager.Task::(_debug) Task 2aa22ee1-42b6-4393-8c00-9ddb235819f0: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:16:07,047::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`3992ff58-7c3d-498c-bbe8-b6419b9d5c1b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:16:07,047::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:16:07,048::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:16:07,049::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`3992ff58-7c3d-498c-bbe8-b6419b9d5c1b`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:16:07,050::task::495::TaskManager.Task::(_debug) Task 2aa22ee1-42b6-4393-8c00-9ddb235819f0: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:16:07,050::task::495::TaskManager.Task::(_debug) Task 2aa22ee1-42b6-4393-8c00-9ddb235819f0: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:16:07,051::task::495::TaskManager.Task::(_debug) Task 2aa22ee1-42b6-4393-8c00-9ddb235819f0: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:16:07,052::task::495::TaskManager.Task::(_debug) Task 2aa22ee1-42b6-4393-8c00-9ddb235819f0: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:16:07,052::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:16:07,053::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:16:07,053::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:16:07,054::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:16:07,055::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:16:07,055::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:16:07,056::task::495::TaskManager.Task::(_debug) Task 2aa22ee1-42b6-4393-8c00-9ddb235819f0: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:16:07,056::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:16:07,057::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:16:07,058::task::495::TaskManager.Task::(_debug) Task 33e27059-3986-4e7e-b6fd-6806cddc73ae: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:16:07,059::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`d6a2dee8-6028-414c-841a-9751bb2cc1d6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:16:07,059::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:16:07,060::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:16:07,061::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`d6a2dee8-6028-414c-841a-9751bb2cc1d6`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:16:07,062::task::495::TaskManager.Task::(_debug) Task 33e27059-3986-4e7e-b6fd-6806cddc73ae: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:16:07,062::task::495::TaskManager.Task::(_debug) Task 33e27059-3986-4e7e-b6fd-6806cddc73ae: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:16:07,063::task::495::TaskManager.Task::(_debug) Task 33e27059-3986-4e7e-b6fd-6806cddc73ae: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:16:07,063::task::495::TaskManager.Task::(_debug) Task 33e27059-3986-4e7e-b6fd-6806cddc73ae: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:16:07,064::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:16:07,065::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:16:07,065::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:16:07,066::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:16:07,066::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:16:07,067::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:16:07,067::task::495::TaskManager.Task::(_debug) Task 33e27059-3986-4e7e-b6fd-6806cddc73ae: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:16:07,068::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:16:07,069::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:16:07,069::task::495::TaskManager.Task::(_debug) Task bc0c406d-e9c5-49cc-80d0-c883e784f85c: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:16:07,071::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`20087400-d9d0-413e-b41d-cc33fa442329`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:16:07,071::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:16:07,072::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:16:07,072::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`20087400-d9d0-413e-b41d-cc33fa442329`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:16:07,073::task::495::TaskManager.Task::(_debug) Task bc0c406d-e9c5-49cc-80d0-c883e784f85c: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:16:07,074::task::495::TaskManager.Task::(_debug) Task bc0c406d-e9c5-49cc-80d0-c883e784f85c: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:16:07,075::task::495::TaskManager.Task::(_debug) Task bc0c406d-e9c5-49cc-80d0-c883e784f85c: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:16:07,075::task::495::TaskManager.Task::(_debug) Task bc0c406d-e9c5-49cc-80d0-c883e784f85c: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:16:07,076::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:16:07,076::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:16:07,077::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:16:07,077::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:16:07,078::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:16:07,078::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:16:07,079::task::495::TaskManager.Task::(_debug) Task bc0c406d-e9c5-49cc-80d0-c883e784f85c: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:16:07,080::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:16:07,286::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0716916 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:16:09,300::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:16:09,783::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0519543 s, 19.7 MB/s\n'; <rc> = 0 >Thread-15913874::INFO::2013-11-20 03:16:10,656::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913874::DEBUG::2013-11-20 03:16:10,657::task::495::TaskManager.Task::(_debug) Task bd47bb11-f156-4a30-b224-33c5eb1dbc4f: moving from state init -> state preparing >Thread-15913874::DEBUG::2013-11-20 03:16:10,658::task::495::TaskManager.Task::(_debug) Task bd47bb11-f156-4a30-b224-33c5eb1dbc4f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00037693977356', 'lastCheck': 1384917362.369127, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00123000144958', 'lastCheck': 1384917369.505178, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00136208534241', 'lastCheck': 1384917361.90255, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00277614593506', 'lastCheck': 1384917361.8993981, 'valid': True, 'code': 0}} >Thread-15913874::DEBUG::2013-11-20 03:16:10,659::task::495::TaskManager.Task::(_debug) Task bd47bb11-f156-4a30-b224-33c5eb1dbc4f: moving from state preparing -> state finished >Thread-15913874::DEBUG::2013-11-20 03:16:10,659::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913874::DEBUG::2013-11-20 03:16:10,660::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913874::DEBUG::2013-11-20 03:16:10,660::task::495::TaskManager.Task::(_debug) Task bd47bb11-f156-4a30-b224-33c5eb1dbc4f: ref 0 aborting False >Thread-15913874::INFO::2013-11-20 03:16:10,661::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00037693977356', 'lastCheck': 1384917362.369127, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00123000144958', 'lastCheck': 1384917369.505178, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00136208534241', 'lastCheck': 1384917361.90255, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00277614593506', 'lastCheck': 1384917361.8993981, 'valid': True, 'code': 0}} >Thread-15913876::DEBUG::2013-11-20 03:16:10,694::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913876::INFO::2013-11-20 03:16:10,695::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913876::DEBUG::2013-11-20 03:16:10,696::task::495::TaskManager.Task::(_debug) Task 30b5b5be-da61-4b75-92db-c64bdd7c1c89: moving from state init -> state preparing >Thread-15913876::DEBUG::2013-11-20 03:16:10,696::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913876::DEBUG::2013-11-20 03:16:10,697::task::495::TaskManager.Task::(_debug) Task 30b5b5be-da61-4b75-92db-c64bdd7c1c89: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913876::DEBUG::2013-11-20 03:16:10,697::task::495::TaskManager.Task::(_debug) Task 30b5b5be-da61-4b75-92db-c64bdd7c1c89: moving from state preparing -> state finished >Thread-15913876::DEBUG::2013-11-20 03:16:10,698::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913876::DEBUG::2013-11-20 03:16:10,698::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913876::DEBUG::2013-11-20 03:16:10,699::task::495::TaskManager.Task::(_debug) Task 30b5b5be-da61-4b75-92db-c64bdd7c1c89: ref 0 aborting False >Thread-15913876::INFO::2013-11-20 03:16:10,699::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913877::DEBUG::2013-11-20 03:16:10,704::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913877::INFO::2013-11-20 03:16:10,704::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913877::DEBUG::2013-11-20 03:16:10,705::task::495::TaskManager.Task::(_debug) Task 3d8ad319-9ebf-4d29-9048-b30900192f25: moving from state init -> state preparing >Thread-15913877::DEBUG::2013-11-20 03:16:10,706::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`22764006-5fa9-4e11-8c60-8b90a90bff06`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913877::DEBUG::2013-11-20 03:16:10,707::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913877::DEBUG::2013-11-20 03:16:10,708::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913877::DEBUG::2013-11-20 03:16:10,708::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`22764006-5fa9-4e11-8c60-8b90a90bff06`::Granted request >Thread-15913877::DEBUG::2013-11-20 03:16:10,717::task::495::TaskManager.Task::(_debug) Task 3d8ad319-9ebf-4d29-9048-b30900192f25: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913877::DEBUG::2013-11-20 03:16:10,718::task::495::TaskManager.Task::(_debug) Task 3d8ad319-9ebf-4d29-9048-b30900192f25: ref 1 aborting False >Thread-15913877::WARNING::2013-11-20 03:16:10,720::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913877::DEBUG::2013-11-20 03:16:10,721::task::495::TaskManager.Task::(_debug) Task 3d8ad319-9ebf-4d29-9048-b30900192f25: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913877::DEBUG::2013-11-20 03:16:10,721::task::495::TaskManager.Task::(_debug) Task 3d8ad319-9ebf-4d29-9048-b30900192f25: moving from state preparing -> state finished >Thread-15913877::DEBUG::2013-11-20 03:16:10,721::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913877::DEBUG::2013-11-20 03:16:10,722::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913877::DEBUG::2013-11-20 03:16:10,722::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913877::DEBUG::2013-11-20 03:16:10,723::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913877::DEBUG::2013-11-20 03:16:10,723::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913877::DEBUG::2013-11-20 03:16:10,723::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913877::DEBUG::2013-11-20 03:16:10,724::task::495::TaskManager.Task::(_debug) Task 3d8ad319-9ebf-4d29-9048-b30900192f25: ref 0 aborting False >Thread-15913877::INFO::2013-11-20 03:16:10,724::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:16:11,792::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:16:12,283::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0590973 s, 17.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:16:14,292::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:16:14,845::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0754751 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:16:16,861::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913639::INFO::2013-11-20 03:16:17,152::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:16:17,165::task::495::TaskManager.Task::(_debug) Task 14118175-771f-48bf-a2a2-b65a9c928280: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:16:17,171::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`556683fd-43f1-4cb9-874a-e8acedd5c062`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:16:17,172::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:16:17,174::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:16:17,174::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`556683fd-43f1-4cb9-874a-e8acedd5c062`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:16:17,176::task::495::TaskManager.Task::(_debug) Task 14118175-771f-48bf-a2a2-b65a9c928280: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:16:17,177::task::495::TaskManager.Task::(_debug) Task 14118175-771f-48bf-a2a2-b65a9c928280: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:16:17,178::task::495::TaskManager.Task::(_debug) Task 14118175-771f-48bf-a2a2-b65a9c928280: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:16:17,179::task::495::TaskManager.Task::(_debug) Task 14118175-771f-48bf-a2a2-b65a9c928280: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:16:17,180::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:16:17,181::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:16:17,182::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:16:17,183::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:16:17,184::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:16:17,184::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:16:17,185::task::495::TaskManager.Task::(_debug) Task 14118175-771f-48bf-a2a2-b65a9c928280: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:16:17,187::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:16:17,366::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0616544 s, 16.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:16:19,382::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:16:19,892::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0802159 s, 12.8 MB/s\n'; <rc> = 0 >Thread-15913881::DEBUG::2013-11-20 03:16:21,735::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913881::INFO::2013-11-20 03:16:21,736::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913881::DEBUG::2013-11-20 03:16:21,736::task::495::TaskManager.Task::(_debug) Task e6596153-2f46-4d27-bce7-5d4225730977: moving from state init -> state preparing >Thread-15913881::DEBUG::2013-11-20 03:16:21,737::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913881::DEBUG::2013-11-20 03:16:21,737::task::495::TaskManager.Task::(_debug) Task e6596153-2f46-4d27-bce7-5d4225730977: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913881::DEBUG::2013-11-20 03:16:21,738::task::495::TaskManager.Task::(_debug) Task e6596153-2f46-4d27-bce7-5d4225730977: moving from state preparing -> state finished >Thread-15913881::DEBUG::2013-11-20 03:16:21,738::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913881::DEBUG::2013-11-20 03:16:21,738::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913881::DEBUG::2013-11-20 03:16:21,739::task::495::TaskManager.Task::(_debug) Task e6596153-2f46-4d27-bce7-5d4225730977: ref 0 aborting False >Thread-15913881::INFO::2013-11-20 03:16:21,739::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913882::DEBUG::2013-11-20 03:16:21,742::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913882::INFO::2013-11-20 03:16:21,743::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913882::DEBUG::2013-11-20 03:16:21,743::task::495::TaskManager.Task::(_debug) Task 9d4f4856-535f-4248-a932-6ed14701e3d1: moving from state init -> state preparing >Thread-15913882::DEBUG::2013-11-20 03:16:21,744::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`64d445cb-9002-4981-9fe4-51dd6b545f67`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913882::DEBUG::2013-11-20 03:16:21,744::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913882::DEBUG::2013-11-20 03:16:21,744::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913882::DEBUG::2013-11-20 03:16:21,745::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`64d445cb-9002-4981-9fe4-51dd6b545f67`::Granted request >Thread-15913882::DEBUG::2013-11-20 03:16:21,745::task::495::TaskManager.Task::(_debug) Task 9d4f4856-535f-4248-a932-6ed14701e3d1: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913882::DEBUG::2013-11-20 03:16:21,746::task::495::TaskManager.Task::(_debug) Task 9d4f4856-535f-4248-a932-6ed14701e3d1: ref 1 aborting False >Thread-15913882::WARNING::2013-11-20 03:16:21,747::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913882::DEBUG::2013-11-20 03:16:21,747::task::495::TaskManager.Task::(_debug) Task 9d4f4856-535f-4248-a932-6ed14701e3d1: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913882::DEBUG::2013-11-20 03:16:21,748::task::495::TaskManager.Task::(_debug) Task 9d4f4856-535f-4248-a932-6ed14701e3d1: moving from state preparing -> state finished >Thread-15913882::DEBUG::2013-11-20 03:16:21,748::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913882::DEBUG::2013-11-20 03:16:21,749::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913882::DEBUG::2013-11-20 03:16:21,749::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913882::DEBUG::2013-11-20 03:16:21,749::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913882::DEBUG::2013-11-20 03:16:21,750::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913882::DEBUG::2013-11-20 03:16:21,750::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913882::DEBUG::2013-11-20 03:16:21,750::task::495::TaskManager.Task::(_debug) Task 9d4f4856-535f-4248-a932-6ed14701e3d1: ref 0 aborting False >Thread-15913882::INFO::2013-11-20 03:16:21,751::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:16:21,908::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:16:22,358::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0521314 s, 19.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:16:24,370::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:16:24,831::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0859019 s, 11.9 MB/s\n'; <rc> = 0 >Thread-15913884::INFO::2013-11-20 03:16:26,694::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913884::DEBUG::2013-11-20 03:16:26,695::task::495::TaskManager.Task::(_debug) Task d5ddeb8e-be38-479d-b0fb-045a7d7fd315: moving from state init -> state preparing >Thread-15913884::DEBUG::2013-11-20 03:16:26,696::task::495::TaskManager.Task::(_debug) Task d5ddeb8e-be38-479d-b0fb-045a7d7fd315: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00020694732666', 'lastCheck': 1384917382.563817, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0135819911957', 'lastCheck': 1384917379.5805919, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00110793113708', 'lastCheck': 1384917382.1207089, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00570297241211', 'lastCheck': 1384917382.1182389, 'valid': True, 'code': 0}} >Thread-15913884::DEBUG::2013-11-20 03:16:26,697::task::495::TaskManager.Task::(_debug) Task d5ddeb8e-be38-479d-b0fb-045a7d7fd315: moving from state preparing -> state finished >Thread-15913884::DEBUG::2013-11-20 03:16:26,697::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913884::DEBUG::2013-11-20 03:16:26,698::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913884::DEBUG::2013-11-20 03:16:26,698::task::495::TaskManager.Task::(_debug) Task d5ddeb8e-be38-479d-b0fb-045a7d7fd315: ref 0 aborting False >Thread-15913884::INFO::2013-11-20 03:16:26,699::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00020694732666', 'lastCheck': 1384917382.563817, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0135819911957', 'lastCheck': 1384917379.5805919, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00110793113708', 'lastCheck': 1384917382.1207089, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00570297241211', 'lastCheck': 1384917382.1182389, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:16:26,843::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:16:27,348::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0779604 s, 13.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:16:29,361::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:16:29,827::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0483567 s, 21.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:16:31,838::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913887::DEBUG::2013-11-20 03:16:32,042::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913887::INFO::2013-11-20 03:16:32,045::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913887::DEBUG::2013-11-20 03:16:32,045::task::495::TaskManager.Task::(_debug) Task f3582340-8488-4755-8da9-43d8ed7ef0cc: moving from state init -> state preparing >Thread-15913887::DEBUG::2013-11-20 03:16:32,047::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913887::DEBUG::2013-11-20 03:16:32,047::task::495::TaskManager.Task::(_debug) Task f3582340-8488-4755-8da9-43d8ed7ef0cc: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913887::DEBUG::2013-11-20 03:16:32,048::task::495::TaskManager.Task::(_debug) Task f3582340-8488-4755-8da9-43d8ed7ef0cc: moving from state preparing -> state finished >Thread-15913887::DEBUG::2013-11-20 03:16:32,049::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913887::DEBUG::2013-11-20 03:16:32,049::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913887::DEBUG::2013-11-20 03:16:32,050::task::495::TaskManager.Task::(_debug) Task f3582340-8488-4755-8da9-43d8ed7ef0cc: ref 0 aborting False >Thread-15913887::INFO::2013-11-20 03:16:32,050::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913888::DEBUG::2013-11-20 03:16:32,056::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913888::INFO::2013-11-20 03:16:32,057::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913888::DEBUG::2013-11-20 03:16:32,058::task::495::TaskManager.Task::(_debug) Task 353a193d-fb32-4a06-ad6a-92ddc332a4de: moving from state init -> state preparing >Thread-15913888::DEBUG::2013-11-20 03:16:32,059::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`03be33e9-2a32-4e32-940d-8d1e6bd899b9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913888::DEBUG::2013-11-20 03:16:32,060::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913888::DEBUG::2013-11-20 03:16:32,061::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913888::DEBUG::2013-11-20 03:16:32,062::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`03be33e9-2a32-4e32-940d-8d1e6bd899b9`::Granted request >Thread-15913888::DEBUG::2013-11-20 03:16:32,063::task::495::TaskManager.Task::(_debug) Task 353a193d-fb32-4a06-ad6a-92ddc332a4de: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913888::DEBUG::2013-11-20 03:16:32,064::task::495::TaskManager.Task::(_debug) Task 353a193d-fb32-4a06-ad6a-92ddc332a4de: ref 1 aborting False >Thread-15913888::WARNING::2013-11-20 03:16:32,067::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913888::DEBUG::2013-11-20 03:16:32,068::task::495::TaskManager.Task::(_debug) Task 353a193d-fb32-4a06-ad6a-92ddc332a4de: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913888::DEBUG::2013-11-20 03:16:32,069::task::495::TaskManager.Task::(_debug) Task 353a193d-fb32-4a06-ad6a-92ddc332a4de: moving from state preparing -> state finished >Thread-15913888::DEBUG::2013-11-20 03:16:32,070::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913888::DEBUG::2013-11-20 03:16:32,070::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913888::DEBUG::2013-11-20 03:16:32,071::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913888::DEBUG::2013-11-20 03:16:32,072::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913888::DEBUG::2013-11-20 03:16:32,073::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913888::DEBUG::2013-11-20 03:16:32,073::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913888::DEBUG::2013-11-20 03:16:32,074::task::495::TaskManager.Task::(_debug) Task 353a193d-fb32-4a06-ad6a-92ddc332a4de: ref 0 aborting False >Thread-15913888::INFO::2013-11-20 03:16:32,075::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:16:32,285::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0475622 s, 21.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:16:34,299::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913695::INFO::2013-11-20 03:16:34,634::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:16:34,635::task::495::TaskManager.Task::(_debug) Task 4a21c1e4-68ee-4c1c-b13a-dc6504e2d09e: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:16:34,636::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ddc18742-c0c5-4cdc-a343-5b1866e8378b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:16:34,637::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:16:34,638::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:16:34,639::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ddc18742-c0c5-4cdc-a343-5b1866e8378b`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:16:34,640::task::495::TaskManager.Task::(_debug) Task 4a21c1e4-68ee-4c1c-b13a-dc6504e2d09e: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:16:34,641::task::495::TaskManager.Task::(_debug) Task 4a21c1e4-68ee-4c1c-b13a-dc6504e2d09e: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:16:34,641::task::495::TaskManager.Task::(_debug) Task 4a21c1e4-68ee-4c1c-b13a-dc6504e2d09e: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:16:34,642::task::495::TaskManager.Task::(_debug) Task 4a21c1e4-68ee-4c1c-b13a-dc6504e2d09e: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:16:34,642::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:16:34,643::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:16:34,643::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:16:34,644::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:16:34,644::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:16:34,645::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:16:34,645::task::495::TaskManager.Task::(_debug) Task 4a21c1e4-68ee-4c1c-b13a-dc6504e2d09e: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:16:34,646::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:16:34,728::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0578738 s, 17.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:16:36,737::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:16:37,158::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.057075 s, 17.9 MB/s\n'; <rc> = 0 >Thread-15913658::INFO::2013-11-20 03:16:37,167::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:16:37,168::task::495::TaskManager.Task::(_debug) Task 22181d1b-7404-441e-acda-040b9eaf2272: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:16:37,169::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`639884ad-767c-4a94-949d-188ab78845d0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:16:37,169::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:16:37,170::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:16:37,170::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`639884ad-767c-4a94-949d-188ab78845d0`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:16:37,172::task::495::TaskManager.Task::(_debug) Task 22181d1b-7404-441e-acda-040b9eaf2272: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:16:37,172::task::495::TaskManager.Task::(_debug) Task 22181d1b-7404-441e-acda-040b9eaf2272: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:16:37,173::task::495::TaskManager.Task::(_debug) Task 22181d1b-7404-441e-acda-040b9eaf2272: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:16:37,174::task::495::TaskManager.Task::(_debug) Task 22181d1b-7404-441e-acda-040b9eaf2272: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:16:37,174::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:16:37,175::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:16:37,175::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:16:37,176::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:16:37,176::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:16:37,176::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:16:37,177::task::495::TaskManager.Task::(_debug) Task 22181d1b-7404-441e-acda-040b9eaf2272: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:16:37,177::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:16:39,169::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:16:39,719::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0861105 s, 11.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:16:41,730::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:16:42,333::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0566787 s, 18.1 MB/s\n'; <rc> = 0 >Thread-15913892::INFO::2013-11-20 03:16:42,409::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913892::DEBUG::2013-11-20 03:16:42,409::task::495::TaskManager.Task::(_debug) Task c61beebd-928c-404f-9fe4-890e4b63b57f: moving from state init -> state preparing >Thread-15913892::DEBUG::2013-11-20 03:16:42,410::task::495::TaskManager.Task::(_debug) Task c61beebd-928c-404f-9fe4-890e4b63b57f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000488996505737', 'lastCheck': 1384917392.5773411, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00140595436096', 'lastCheck': 1384917399.6175461, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0110759735107', 'lastCheck': 1384917402.180006, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000577211380005', 'lastCheck': 1384917402.142386, 'valid': True, 'code': 0}} >Thread-15913892::DEBUG::2013-11-20 03:16:42,410::task::495::TaskManager.Task::(_debug) Task c61beebd-928c-404f-9fe4-890e4b63b57f: moving from state preparing -> state finished >Thread-15913892::DEBUG::2013-11-20 03:16:42,411::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913892::DEBUG::2013-11-20 03:16:42,411::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913892::DEBUG::2013-11-20 03:16:42,411::task::495::TaskManager.Task::(_debug) Task c61beebd-928c-404f-9fe4-890e4b63b57f: ref 0 aborting False >Thread-15913892::INFO::2013-11-20 03:16:42,412::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000488996505737', 'lastCheck': 1384917392.5773411, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00140595436096', 'lastCheck': 1384917399.6175461, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0110759735107', 'lastCheck': 1384917402.180006, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000577211380005', 'lastCheck': 1384917402.142386, 'valid': True, 'code': 0}} >Thread-15913894::DEBUG::2013-11-20 03:16:42,532::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913894::INFO::2013-11-20 03:16:42,533::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913894::DEBUG::2013-11-20 03:16:42,534::task::495::TaskManager.Task::(_debug) Task b0163baa-d37e-46d7-8b59-7599145abd50: moving from state init -> state preparing >Thread-15913894::DEBUG::2013-11-20 03:16:42,535::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913894::DEBUG::2013-11-20 03:16:42,535::task::495::TaskManager.Task::(_debug) Task b0163baa-d37e-46d7-8b59-7599145abd50: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913894::DEBUG::2013-11-20 03:16:42,536::task::495::TaskManager.Task::(_debug) Task b0163baa-d37e-46d7-8b59-7599145abd50: moving from state preparing -> state finished >Thread-15913894::DEBUG::2013-11-20 03:16:42,537::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913894::DEBUG::2013-11-20 03:16:42,537::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913894::DEBUG::2013-11-20 03:16:42,538::task::495::TaskManager.Task::(_debug) Task b0163baa-d37e-46d7-8b59-7599145abd50: ref 0 aborting False >Thread-15913894::INFO::2013-11-20 03:16:42,538::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913895::DEBUG::2013-11-20 03:16:42,542::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913895::INFO::2013-11-20 03:16:42,543::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913895::DEBUG::2013-11-20 03:16:42,544::task::495::TaskManager.Task::(_debug) Task eb5629dc-6913-4b3c-901f-95c40d8d7ee7: moving from state init -> state preparing >Thread-15913895::DEBUG::2013-11-20 03:16:42,545::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`cf70a26a-df6f-457e-966b-172f06f1f463`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913895::DEBUG::2013-11-20 03:16:42,546::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913895::DEBUG::2013-11-20 03:16:42,547::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913895::DEBUG::2013-11-20 03:16:42,547::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`cf70a26a-df6f-457e-966b-172f06f1f463`::Granted request >Thread-15913895::DEBUG::2013-11-20 03:16:42,548::task::495::TaskManager.Task::(_debug) Task eb5629dc-6913-4b3c-901f-95c40d8d7ee7: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913895::DEBUG::2013-11-20 03:16:42,549::task::495::TaskManager.Task::(_debug) Task eb5629dc-6913-4b3c-901f-95c40d8d7ee7: ref 1 aborting False >Thread-15913895::WARNING::2013-11-20 03:16:42,552::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913895::DEBUG::2013-11-20 03:16:42,553::task::495::TaskManager.Task::(_debug) Task eb5629dc-6913-4b3c-901f-95c40d8d7ee7: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913895::DEBUG::2013-11-20 03:16:42,554::task::495::TaskManager.Task::(_debug) Task eb5629dc-6913-4b3c-901f-95c40d8d7ee7: moving from state preparing -> state finished >Thread-15913895::DEBUG::2013-11-20 03:16:42,554::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913895::DEBUG::2013-11-20 03:16:42,555::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913895::DEBUG::2013-11-20 03:16:42,556::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913895::DEBUG::2013-11-20 03:16:42,557::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15912392::INFO::2013-11-20 03:16:42,557::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15913895::DEBUG::2013-11-20 03:16:42,558::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:16:42,558::task::495::TaskManager.Task::(_debug) Task f80f0453-c70e-4381-881d-5cfc83d40e28: moving from state init -> state preparing >Thread-15913895::DEBUG::2013-11-20 03:16:42,559::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:16:42,560::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`8b94d1f0-6abf-47d0-91fa-867f41b5f854`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913895::DEBUG::2013-11-20 03:16:42,561::task::495::TaskManager.Task::(_debug) Task eb5629dc-6913-4b3c-901f-95c40d8d7ee7: ref 0 aborting False >Thread-15912392::DEBUG::2013-11-20 03:16:42,561::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913895::INFO::2013-11-20 03:16:42,562::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912392::DEBUG::2013-11-20 03:16:42,563::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:16:42,564::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`8b94d1f0-6abf-47d0-91fa-867f41b5f854`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:16:42,565::task::495::TaskManager.Task::(_debug) Task f80f0453-c70e-4381-881d-5cfc83d40e28: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:16:42,565::task::495::TaskManager.Task::(_debug) Task f80f0453-c70e-4381-881d-5cfc83d40e28: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:16:42,566::task::495::TaskManager.Task::(_debug) Task f80f0453-c70e-4381-881d-5cfc83d40e28: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:16:42,566::task::495::TaskManager.Task::(_debug) Task f80f0453-c70e-4381-881d-5cfc83d40e28: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:16:42,566::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:16:42,567::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:16:42,567::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:16:42,567::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:16:42,568::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:16:42,569::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:16:42,569::task::495::TaskManager.Task::(_debug) Task f80f0453-c70e-4381-881d-5cfc83d40e28: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:16:42,569::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-42838::DEBUG::2013-11-20 03:16:42,586::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 03:16:43,220::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:16:44,348::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:16:44,783::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0493433 s, 20.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:16:46,793::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:16:47,337::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.136662 s, 7.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:16:49,350::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:16:49,871::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0857151 s, 11.9 MB/s\n'; <rc> = 0 >Thread-15913518::INFO::2013-11-20 03:16:50,651::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:16:50,652::task::495::TaskManager.Task::(_debug) Task 33fa4453-2466-498e-8e72-07d7777b6bf8: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:16:50,653::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b9020216-4837-4fb6-83a9-34049e47ab3b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:16:50,653::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:16:50,654::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:16:50,654::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b9020216-4837-4fb6-83a9-34049e47ab3b`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:16:50,655::task::495::TaskManager.Task::(_debug) Task 33fa4453-2466-498e-8e72-07d7777b6bf8: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:16:50,655::task::495::TaskManager.Task::(_debug) Task 33fa4453-2466-498e-8e72-07d7777b6bf8: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:16:50,656::task::495::TaskManager.Task::(_debug) Task 33fa4453-2466-498e-8e72-07d7777b6bf8: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:16:50,665::task::495::TaskManager.Task::(_debug) Task 33fa4453-2466-498e-8e72-07d7777b6bf8: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:16:50,665::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:16:50,666::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:16:50,666::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:16:50,667::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:16:50,667::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:16:50,667::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:16:50,668::task::495::TaskManager.Task::(_debug) Task 33fa4453-2466-498e-8e72-07d7777b6bf8: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:16:50,668::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:16:50,669::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:16:50,669::task::495::TaskManager.Task::(_debug) Task 4232c3a0-47ba-4dd8-8b2b-8cf877fb148a: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:16:50,670::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`17a2a034-bbc9-413e-a290-98beb574c1f4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:16:50,671::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:16:50,671::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:16:50,671::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`17a2a034-bbc9-413e-a290-98beb574c1f4`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:16:50,672::task::495::TaskManager.Task::(_debug) Task 4232c3a0-47ba-4dd8-8b2b-8cf877fb148a: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:16:50,673::task::495::TaskManager.Task::(_debug) Task 4232c3a0-47ba-4dd8-8b2b-8cf877fb148a: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:16:50,673::task::495::TaskManager.Task::(_debug) Task 4232c3a0-47ba-4dd8-8b2b-8cf877fb148a: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:16:50,673::task::495::TaskManager.Task::(_debug) Task 4232c3a0-47ba-4dd8-8b2b-8cf877fb148a: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:16:50,674::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:16:50,674::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:16:50,675::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:16:50,675::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:16:50,675::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:16:50,676::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:16:50,676::task::495::TaskManager.Task::(_debug) Task 4232c3a0-47ba-4dd8-8b2b-8cf877fb148a: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:16:50,677::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:16:51,885::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:16:52,385::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0735257 s, 13.9 MB/s\n'; <rc> = 0 >Thread-15913899::DEBUG::2013-11-20 03:16:53,345::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913899::INFO::2013-11-20 03:16:53,346::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913899::DEBUG::2013-11-20 03:16:53,346::task::495::TaskManager.Task::(_debug) Task 98cfbd4c-46fa-469c-948d-ebad4b514f8b: moving from state init -> state preparing >Thread-15913899::DEBUG::2013-11-20 03:16:53,346::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913899::DEBUG::2013-11-20 03:16:53,347::task::495::TaskManager.Task::(_debug) Task 98cfbd4c-46fa-469c-948d-ebad4b514f8b: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913899::DEBUG::2013-11-20 03:16:53,347::task::495::TaskManager.Task::(_debug) Task 98cfbd4c-46fa-469c-948d-ebad4b514f8b: moving from state preparing -> state finished >Thread-15913899::DEBUG::2013-11-20 03:16:53,347::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913899::DEBUG::2013-11-20 03:16:53,348::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913899::DEBUG::2013-11-20 03:16:53,348::task::495::TaskManager.Task::(_debug) Task 98cfbd4c-46fa-469c-948d-ebad4b514f8b: ref 0 aborting False >Thread-15913899::INFO::2013-11-20 03:16:53,348::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913900::DEBUG::2013-11-20 03:16:53,374::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913900::INFO::2013-11-20 03:16:53,374::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913900::DEBUG::2013-11-20 03:16:53,375::task::495::TaskManager.Task::(_debug) Task e6aa34c2-b52a-46e3-825d-88c5ca4be965: moving from state init -> state preparing >Thread-15913900::DEBUG::2013-11-20 03:16:53,376::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`0e975c8b-0fa0-426e-9169-5f93d1ea49f3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913900::DEBUG::2013-11-20 03:16:53,376::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913900::DEBUG::2013-11-20 03:16:53,376::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913900::DEBUG::2013-11-20 03:16:53,377::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`0e975c8b-0fa0-426e-9169-5f93d1ea49f3`::Granted request >Thread-15913900::DEBUG::2013-11-20 03:16:53,377::task::495::TaskManager.Task::(_debug) Task e6aa34c2-b52a-46e3-825d-88c5ca4be965: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913900::DEBUG::2013-11-20 03:16:53,378::task::495::TaskManager.Task::(_debug) Task e6aa34c2-b52a-46e3-825d-88c5ca4be965: ref 1 aborting False >Thread-15913900::WARNING::2013-11-20 03:16:53,379::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913900::DEBUG::2013-11-20 03:16:53,379::task::495::TaskManager.Task::(_debug) Task e6aa34c2-b52a-46e3-825d-88c5ca4be965: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913900::DEBUG::2013-11-20 03:16:53,380::task::495::TaskManager.Task::(_debug) Task e6aa34c2-b52a-46e3-825d-88c5ca4be965: moving from state preparing -> state finished >Thread-15913900::DEBUG::2013-11-20 03:16:53,380::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913900::DEBUG::2013-11-20 03:16:53,381::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913900::DEBUG::2013-11-20 03:16:53,381::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913900::DEBUG::2013-11-20 03:16:53,382::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913900::DEBUG::2013-11-20 03:16:53,382::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913900::DEBUG::2013-11-20 03:16:53,382::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913900::DEBUG::2013-11-20 03:16:53,383::task::495::TaskManager.Task::(_debug) Task e6aa34c2-b52a-46e3-825d-88c5ca4be965: ref 0 aborting False >Thread-15913900::INFO::2013-11-20 03:16:53,383::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:16:54,400::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:16:54,894::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0837026 s, 12.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:16:56,909::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:16:57,375::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0720696 s, 14.2 MB/s\n'; <rc> = 0 >Thread-15913902::INFO::2013-11-20 03:16:58,311::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913902::DEBUG::2013-11-20 03:16:58,312::task::495::TaskManager.Task::(_debug) Task bac06feb-9145-4c33-959e-890ad2a0cf0d: moving from state init -> state preparing >Thread-15913902::DEBUG::2013-11-20 03:16:58,312::task::495::TaskManager.Task::(_debug) Task bac06feb-9145-4c33-959e-890ad2a0cf0d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000300884246826', 'lastCheck': 1384917413.2335579, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00216507911682', 'lastCheck': 1384917409.7565651, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00174403190613', 'lastCheck': 1384917412.197372, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000508069992065', 'lastCheck': 1384917412.1540329, 'valid': True, 'code': 0}} >Thread-15913902::DEBUG::2013-11-20 03:16:58,313::task::495::TaskManager.Task::(_debug) Task bac06feb-9145-4c33-959e-890ad2a0cf0d: moving from state preparing -> state finished >Thread-15913902::DEBUG::2013-11-20 03:16:58,313::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913902::DEBUG::2013-11-20 03:16:58,314::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913902::DEBUG::2013-11-20 03:16:58,314::task::495::TaskManager.Task::(_debug) Task bac06feb-9145-4c33-959e-890ad2a0cf0d: ref 0 aborting False >Thread-15913902::INFO::2013-11-20 03:16:58,314::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000300884246826', 'lastCheck': 1384917413.2335579, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00216507911682', 'lastCheck': 1384917409.7565651, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00174403190613', 'lastCheck': 1384917412.197372, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000508069992065', 'lastCheck': 1384917412.1540329, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:16:59,390::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:16:59,929::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0787665 s, 13.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:17:01,944::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:02,335::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0452756 s, 22.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:17:04,344::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913905::DEBUG::2013-11-20 03:17:04,500::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913905::INFO::2013-11-20 03:17:04,502::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913905::DEBUG::2013-11-20 03:17:04,503::task::495::TaskManager.Task::(_debug) Task 9a385d3b-1895-48dc-9075-1f680e21c556: moving from state init -> state preparing >Thread-15913905::DEBUG::2013-11-20 03:17:04,504::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913905::DEBUG::2013-11-20 03:17:04,504::task::495::TaskManager.Task::(_debug) Task 9a385d3b-1895-48dc-9075-1f680e21c556: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913905::DEBUG::2013-11-20 03:17:04,505::task::495::TaskManager.Task::(_debug) Task 9a385d3b-1895-48dc-9075-1f680e21c556: moving from state preparing -> state finished >Thread-15913905::DEBUG::2013-11-20 03:17:04,505::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913905::DEBUG::2013-11-20 03:17:04,506::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913905::DEBUG::2013-11-20 03:17:04,506::task::495::TaskManager.Task::(_debug) Task 9a385d3b-1895-48dc-9075-1f680e21c556: ref 0 aborting False >Thread-15913905::INFO::2013-11-20 03:17:04,507::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913906::DEBUG::2013-11-20 03:17:04,512::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913906::INFO::2013-11-20 03:17:04,513::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913906::DEBUG::2013-11-20 03:17:04,513::task::495::TaskManager.Task::(_debug) Task b59342f5-a065-4a0f-9a41-a4f5e9bcae81: moving from state init -> state preparing >Thread-15913906::DEBUG::2013-11-20 03:17:04,514::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9a2eaad9-4869-4533-97b3-075975989e53`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913906::DEBUG::2013-11-20 03:17:04,515::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913906::DEBUG::2013-11-20 03:17:04,515::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913906::DEBUG::2013-11-20 03:17:04,516::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9a2eaad9-4869-4533-97b3-075975989e53`::Granted request >Thread-15913906::DEBUG::2013-11-20 03:17:04,517::task::495::TaskManager.Task::(_debug) Task b59342f5-a065-4a0f-9a41-a4f5e9bcae81: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913906::DEBUG::2013-11-20 03:17:04,517::task::495::TaskManager.Task::(_debug) Task b59342f5-a065-4a0f-9a41-a4f5e9bcae81: ref 1 aborting False >Thread-15913906::WARNING::2013-11-20 03:17:04,519::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913906::DEBUG::2013-11-20 03:17:04,519::task::495::TaskManager.Task::(_debug) Task b59342f5-a065-4a0f-9a41-a4f5e9bcae81: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913906::DEBUG::2013-11-20 03:17:04,520::task::495::TaskManager.Task::(_debug) Task b59342f5-a065-4a0f-9a41-a4f5e9bcae81: moving from state preparing -> state finished >Thread-15913906::DEBUG::2013-11-20 03:17:04,520::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913906::DEBUG::2013-11-20 03:17:04,521::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913906::DEBUG::2013-11-20 03:17:04,521::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913906::DEBUG::2013-11-20 03:17:04,521::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913906::DEBUG::2013-11-20 03:17:04,522::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913906::DEBUG::2013-11-20 03:17:04,522::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913906::DEBUG::2013-11-20 03:17:04,523::task::495::TaskManager.Task::(_debug) Task b59342f5-a065-4a0f-9a41-a4f5e9bcae81: ref 0 aborting False >Thread-15913906::INFO::2013-11-20 03:17:04,523::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:17:04,824::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0679863 s, 15.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:17:06,844::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:07,241::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0455438 s, 22.5 MB/s\n'; <rc> = 0 >Thread-15913546::INFO::2013-11-20 03:17:07,621::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:17:07,621::task::495::TaskManager.Task::(_debug) Task 07b2d93d-3295-43d1-ba45-cb810bb0308a: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:17:07,637::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`a48b6f5d-6bce-4942-ac83-c0f39e45bd93`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:17:07,638::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:17:07,639::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:17:07,639::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`a48b6f5d-6bce-4942-ac83-c0f39e45bd93`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:17:07,640::task::495::TaskManager.Task::(_debug) Task 07b2d93d-3295-43d1-ba45-cb810bb0308a: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:17:07,641::task::495::TaskManager.Task::(_debug) Task 07b2d93d-3295-43d1-ba45-cb810bb0308a: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:17:07,641::task::495::TaskManager.Task::(_debug) Task 07b2d93d-3295-43d1-ba45-cb810bb0308a: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:17:07,642::task::495::TaskManager.Task::(_debug) Task 07b2d93d-3295-43d1-ba45-cb810bb0308a: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:17:07,642::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:17:07,642::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:17:07,643::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:17:07,643::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:17:07,644::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:17:07,644::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:17:07,644::task::495::TaskManager.Task::(_debug) Task 07b2d93d-3295-43d1-ba45-cb810bb0308a: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:17:07,645::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:17:07,645::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:17:07,645::task::495::TaskManager.Task::(_debug) Task dd9b16bb-0f13-481b-8ed4-0d73b1055510: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:17:07,646::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`ef209a62-fda6-4c61-8449-e29e95bfa6d8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:17:07,646::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:17:07,647::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:17:07,647::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`ef209a62-fda6-4c61-8449-e29e95bfa6d8`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:17:07,648::task::495::TaskManager.Task::(_debug) Task dd9b16bb-0f13-481b-8ed4-0d73b1055510: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:17:07,648::task::495::TaskManager.Task::(_debug) Task dd9b16bb-0f13-481b-8ed4-0d73b1055510: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:17:07,649::task::495::TaskManager.Task::(_debug) Task dd9b16bb-0f13-481b-8ed4-0d73b1055510: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:17:07,649::task::495::TaskManager.Task::(_debug) Task dd9b16bb-0f13-481b-8ed4-0d73b1055510: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:17:07,649::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:17:07,650::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:17:07,650::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:17:07,650::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:17:07,651::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:17:07,651::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:17:07,651::task::495::TaskManager.Task::(_debug) Task dd9b16bb-0f13-481b-8ed4-0d73b1055510: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:17:07,652::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:17:07,652::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:17:07,653::task::495::TaskManager.Task::(_debug) Task b7306902-2dc1-4fb4-b42d-cb28fe856132: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:17:07,653::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`06b0a5a5-c7a1-4cb5-b4cf-6261537b8313`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:17:07,654::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:17:07,654::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:17:07,654::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`06b0a5a5-c7a1-4cb5-b4cf-6261537b8313`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:17:07,655::task::495::TaskManager.Task::(_debug) Task b7306902-2dc1-4fb4-b42d-cb28fe856132: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:17:07,655::task::495::TaskManager.Task::(_debug) Task b7306902-2dc1-4fb4-b42d-cb28fe856132: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:17:07,655::task::495::TaskManager.Task::(_debug) Task b7306902-2dc1-4fb4-b42d-cb28fe856132: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:17:07,656::task::495::TaskManager.Task::(_debug) Task b7306902-2dc1-4fb4-b42d-cb28fe856132: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:17:07,656::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:17:07,657::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:17:07,657::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:17:07,657::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:17:07,658::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:17:07,658::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:17:07,658::task::495::TaskManager.Task::(_debug) Task b7306902-2dc1-4fb4-b42d-cb28fe856132: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:17:07,659::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:17:07,659::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:17:07,659::task::495::TaskManager.Task::(_debug) Task 455f9286-30d3-4212-896a-8d85e3613f65: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:17:07,660::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`e75de393-321b-49e4-9922-fab82b2963c1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:17:07,660::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:17:07,661::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:17:07,661::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`e75de393-321b-49e4-9922-fab82b2963c1`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:17:07,662::task::495::TaskManager.Task::(_debug) Task 455f9286-30d3-4212-896a-8d85e3613f65: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:17:07,662::task::495::TaskManager.Task::(_debug) Task 455f9286-30d3-4212-896a-8d85e3613f65: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:17:07,662::task::495::TaskManager.Task::(_debug) Task 455f9286-30d3-4212-896a-8d85e3613f65: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:17:07,663::task::495::TaskManager.Task::(_debug) Task 455f9286-30d3-4212-896a-8d85e3613f65: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:17:07,663::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:17:07,663::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:17:07,664::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:17:07,664::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:17:07,664::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:17:07,665::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:17:07,665::task::495::TaskManager.Task::(_debug) Task 455f9286-30d3-4212-896a-8d85e3613f65: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:17:07,665::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:17:09,256::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:09,781::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0790301 s, 13.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:17:11,804::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:12,231::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0461483 s, 22.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:17:14,240::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:14,751::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0835102 s, 12.3 MB/s\n'; <rc> = 0 >Thread-15913910::INFO::2013-11-20 03:17:15,036::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913910::DEBUG::2013-11-20 03:17:15,038::task::495::TaskManager.Task::(_debug) Task 73940958-d6a1-4eeb-8c92-03dd1ac77809: moving from state init -> state preparing >Thread-15913910::DEBUG::2013-11-20 03:17:15,039::task::495::TaskManager.Task::(_debug) Task 73940958-d6a1-4eeb-8c92-03dd1ac77809: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000298023223877', 'lastCheck': 1384917433.2552011, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00583696365356', 'lastCheck': 1384917429.803071, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00137591362', 'lastCheck': 1384917432.246809, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000648021697998', 'lastCheck': 1384917432.1774869, 'valid': True, 'code': 0}} >Thread-15913910::DEBUG::2013-11-20 03:17:15,041::task::495::TaskManager.Task::(_debug) Task 73940958-d6a1-4eeb-8c92-03dd1ac77809: moving from state preparing -> state finished >Thread-15913910::DEBUG::2013-11-20 03:17:15,041::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913910::DEBUG::2013-11-20 03:17:15,042::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913910::DEBUG::2013-11-20 03:17:15,043::task::495::TaskManager.Task::(_debug) Task 73940958-d6a1-4eeb-8c92-03dd1ac77809: ref 0 aborting False >Thread-15913910::INFO::2013-11-20 03:17:15,044::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000298023223877', 'lastCheck': 1384917433.2552011, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00583696365356', 'lastCheck': 1384917429.803071, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00137591362', 'lastCheck': 1384917432.246809, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000648021697998', 'lastCheck': 1384917432.1774869, 'valid': True, 'code': 0}} >Thread-15913912::DEBUG::2013-11-20 03:17:15,213::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913912::INFO::2013-11-20 03:17:15,214::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913912::DEBUG::2013-11-20 03:17:15,215::task::495::TaskManager.Task::(_debug) Task a292b767-e240-4e06-8b4a-7380916feac6: moving from state init -> state preparing >Thread-15913912::DEBUG::2013-11-20 03:17:15,216::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913912::DEBUG::2013-11-20 03:17:15,216::task::495::TaskManager.Task::(_debug) Task a292b767-e240-4e06-8b4a-7380916feac6: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913912::DEBUG::2013-11-20 03:17:15,217::task::495::TaskManager.Task::(_debug) Task a292b767-e240-4e06-8b4a-7380916feac6: moving from state preparing -> state finished >Thread-15913912::DEBUG::2013-11-20 03:17:15,218::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913912::DEBUG::2013-11-20 03:17:15,219::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913912::DEBUG::2013-11-20 03:17:15,219::task::495::TaskManager.Task::(_debug) Task a292b767-e240-4e06-8b4a-7380916feac6: ref 0 aborting False >Thread-15913912::INFO::2013-11-20 03:17:15,220::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913913::DEBUG::2013-11-20 03:17:15,224::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913913::INFO::2013-11-20 03:17:15,225::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913913::DEBUG::2013-11-20 03:17:15,226::task::495::TaskManager.Task::(_debug) Task 235d4292-374d-46b1-88ea-1359b5503d73: moving from state init -> state preparing >Thread-15913913::DEBUG::2013-11-20 03:17:15,227::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1168bf05-b7be-4394-b844-cbc476d0983a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913913::DEBUG::2013-11-20 03:17:15,228::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913913::DEBUG::2013-11-20 03:17:15,229::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913913::DEBUG::2013-11-20 03:17:15,229::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1168bf05-b7be-4394-b844-cbc476d0983a`::Granted request >Thread-15913913::DEBUG::2013-11-20 03:17:15,231::task::495::TaskManager.Task::(_debug) Task 235d4292-374d-46b1-88ea-1359b5503d73: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913913::DEBUG::2013-11-20 03:17:15,231::task::495::TaskManager.Task::(_debug) Task 235d4292-374d-46b1-88ea-1359b5503d73: ref 1 aborting False >Thread-15913913::WARNING::2013-11-20 03:17:15,233::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913913::DEBUG::2013-11-20 03:17:15,234::task::495::TaskManager.Task::(_debug) Task 235d4292-374d-46b1-88ea-1359b5503d73: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913913::DEBUG::2013-11-20 03:17:15,235::task::495::TaskManager.Task::(_debug) Task 235d4292-374d-46b1-88ea-1359b5503d73: moving from state preparing -> state finished >Thread-15913913::DEBUG::2013-11-20 03:17:15,236::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913913::DEBUG::2013-11-20 03:17:15,236::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913913::DEBUG::2013-11-20 03:17:15,237::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913913::DEBUG::2013-11-20 03:17:15,238::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913913::DEBUG::2013-11-20 03:17:15,238::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913913::DEBUG::2013-11-20 03:17:15,239::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913913::DEBUG::2013-11-20 03:17:15,239::task::495::TaskManager.Task::(_debug) Task 235d4292-374d-46b1-88ea-1359b5503d73: ref 0 aborting False >Thread-15913913::INFO::2013-11-20 03:17:15,240::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:17:16,764::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:17,296::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0956234 s, 10.7 MB/s\n'; <rc> = 0 >Thread-15913639::INFO::2013-11-20 03:17:17,658::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:17:17,660::task::495::TaskManager.Task::(_debug) Task b47e925c-6d94-4228-8204-dc34fba051c7: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:17:17,669::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`660a27db-bd59-47d0-a5e6-b18ae2545b92`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:17:17,670::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:17:17,671::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:17:17,672::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`660a27db-bd59-47d0-a5e6-b18ae2545b92`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:17:17,674::task::495::TaskManager.Task::(_debug) Task b47e925c-6d94-4228-8204-dc34fba051c7: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:17:17,674::task::495::TaskManager.Task::(_debug) Task b47e925c-6d94-4228-8204-dc34fba051c7: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:17:17,676::task::495::TaskManager.Task::(_debug) Task b47e925c-6d94-4228-8204-dc34fba051c7: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:17:17,676::task::495::TaskManager.Task::(_debug) Task b47e925c-6d94-4228-8204-dc34fba051c7: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:17:17,677::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:17:17,678::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:17:17,678::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:17:17,679::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:17:17,680::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:17:17,680::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:17:17,681::task::495::TaskManager.Task::(_debug) Task b47e925c-6d94-4228-8204-dc34fba051c7: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:17:17,681::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:17:19,305::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:19,846::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0783429 s, 13.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:17:21,859::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:22,367::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0555871 s, 18.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:17:24,381::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:24,962::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0755565 s, 13.6 MB/s\n'; <rc> = 0 >Thread-15913917::DEBUG::2013-11-20 03:17:25,679::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913917::INFO::2013-11-20 03:17:25,681::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913917::DEBUG::2013-11-20 03:17:25,682::task::495::TaskManager.Task::(_debug) Task b4a6040d-1196-4e1a-9939-6d9a563e0280: moving from state init -> state preparing >Thread-15913917::DEBUG::2013-11-20 03:17:25,683::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913917::DEBUG::2013-11-20 03:17:25,684::task::495::TaskManager.Task::(_debug) Task b4a6040d-1196-4e1a-9939-6d9a563e0280: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913917::DEBUG::2013-11-20 03:17:25,685::task::495::TaskManager.Task::(_debug) Task b4a6040d-1196-4e1a-9939-6d9a563e0280: moving from state preparing -> state finished >Thread-15913917::DEBUG::2013-11-20 03:17:25,686::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913917::DEBUG::2013-11-20 03:17:25,687::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913917::DEBUG::2013-11-20 03:17:25,688::task::495::TaskManager.Task::(_debug) Task b4a6040d-1196-4e1a-9939-6d9a563e0280: ref 0 aborting False >Thread-15913917::INFO::2013-11-20 03:17:25,688::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913918::DEBUG::2013-11-20 03:17:25,699::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913918::INFO::2013-11-20 03:17:25,701::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913918::DEBUG::2013-11-20 03:17:25,701::task::495::TaskManager.Task::(_debug) Task 9af1b9db-99f1-4847-b8de-fc1e64abea00: moving from state init -> state preparing >Thread-15913918::DEBUG::2013-11-20 03:17:25,703::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fc291d07-4779-4879-a0e5-bc866d98d2c1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913918::DEBUG::2013-11-20 03:17:25,703::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913918::DEBUG::2013-11-20 03:17:25,704::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913918::DEBUG::2013-11-20 03:17:25,705::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fc291d07-4779-4879-a0e5-bc866d98d2c1`::Granted request >Thread-15913918::DEBUG::2013-11-20 03:17:25,706::task::495::TaskManager.Task::(_debug) Task 9af1b9db-99f1-4847-b8de-fc1e64abea00: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913918::DEBUG::2013-11-20 03:17:25,707::task::495::TaskManager.Task::(_debug) Task 9af1b9db-99f1-4847-b8de-fc1e64abea00: ref 1 aborting False >Thread-15913918::WARNING::2013-11-20 03:17:25,709::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913918::DEBUG::2013-11-20 03:17:25,710::task::495::TaskManager.Task::(_debug) Task 9af1b9db-99f1-4847-b8de-fc1e64abea00: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913918::DEBUG::2013-11-20 03:17:25,711::task::495::TaskManager.Task::(_debug) Task 9af1b9db-99f1-4847-b8de-fc1e64abea00: moving from state preparing -> state finished >Thread-15913918::DEBUG::2013-11-20 03:17:25,711::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913918::DEBUG::2013-11-20 03:17:25,712::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913918::DEBUG::2013-11-20 03:17:25,712::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913918::DEBUG::2013-11-20 03:17:25,713::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913918::DEBUG::2013-11-20 03:17:25,713::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913918::DEBUG::2013-11-20 03:17:25,714::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913918::DEBUG::2013-11-20 03:17:25,715::task::495::TaskManager.Task::(_debug) Task 9af1b9db-99f1-4847-b8de-fc1e64abea00: ref 0 aborting False >Thread-15913918::INFO::2013-11-20 03:17:25,715::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:17:26,975::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:27,483::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0814004 s, 12.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:17:29,499::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:30,065::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0784806 s, 13.0 MB/s\n'; <rc> = 0 >Thread-15913920::INFO::2013-11-20 03:17:30,857::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913920::DEBUG::2013-11-20 03:17:30,857::task::495::TaskManager.Task::(_debug) Task eac4ceb4-db08-4a7c-aa86-56c956e26358: moving from state init -> state preparing >Thread-15913920::DEBUG::2013-11-20 03:17:30,858::task::495::TaskManager.Task::(_debug) Task eac4ceb4-db08-4a7c-aa86-56c956e26358: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000340938568115', 'lastCheck': 1384917443.511121, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00126314163208', 'lastCheck': 1384917450.1647019, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0016782283783', 'lastCheck': 1384917442.5143099, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000478029251099', 'lastCheck': 1384917442.190304, 'valid': True, 'code': 0}} >Thread-15913920::DEBUG::2013-11-20 03:17:30,858::task::495::TaskManager.Task::(_debug) Task eac4ceb4-db08-4a7c-aa86-56c956e26358: moving from state preparing -> state finished >Thread-15913920::DEBUG::2013-11-20 03:17:30,859::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913920::DEBUG::2013-11-20 03:17:30,859::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913920::DEBUG::2013-11-20 03:17:30,860::task::495::TaskManager.Task::(_debug) Task eac4ceb4-db08-4a7c-aa86-56c956e26358: ref 0 aborting False >Thread-15913920::INFO::2013-11-20 03:17:30,860::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000340938568115', 'lastCheck': 1384917443.511121, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00126314163208', 'lastCheck': 1384917450.1647019, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0016782283783', 'lastCheck': 1384917442.5143099, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000478029251099', 'lastCheck': 1384917442.190304, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:17:32,081::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:32,631::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.081181 s, 12.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:17:34,647::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:35,145::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0750451 s, 13.6 MB/s\n'; <rc> = 0 >Thread-15913695::INFO::2013-11-20 03:17:35,202::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:17:35,203::task::495::TaskManager.Task::(_debug) Task a26ddb7e-7e7d-4a7a-affd-ac2317f45474: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:17:35,205::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d9be0dd8-d847-4a3c-8071-1c78cd6230dc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:17:35,205::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:17:35,206::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:17:35,207::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d9be0dd8-d847-4a3c-8071-1c78cd6230dc`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:17:35,208::task::495::TaskManager.Task::(_debug) Task a26ddb7e-7e7d-4a7a-affd-ac2317f45474: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:17:35,208::task::495::TaskManager.Task::(_debug) Task a26ddb7e-7e7d-4a7a-affd-ac2317f45474: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:17:35,209::task::495::TaskManager.Task::(_debug) Task a26ddb7e-7e7d-4a7a-affd-ac2317f45474: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:17:35,209::task::495::TaskManager.Task::(_debug) Task a26ddb7e-7e7d-4a7a-affd-ac2317f45474: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:17:35,210::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:17:35,210::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:17:35,211::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:17:35,211::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:17:35,212::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:17:35,212::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:17:35,213::task::495::TaskManager.Task::(_debug) Task a26ddb7e-7e7d-4a7a-affd-ac2317f45474: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:17:35,213::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913923::DEBUG::2013-11-20 03:17:35,956::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913923::INFO::2013-11-20 03:17:35,958::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913923::DEBUG::2013-11-20 03:17:35,959::task::495::TaskManager.Task::(_debug) Task b17a5d63-4b78-43ef-8d48-c73a976bbf88: moving from state init -> state preparing >Thread-15913923::DEBUG::2013-11-20 03:17:35,960::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913923::DEBUG::2013-11-20 03:17:35,960::task::495::TaskManager.Task::(_debug) Task b17a5d63-4b78-43ef-8d48-c73a976bbf88: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913923::DEBUG::2013-11-20 03:17:35,961::task::495::TaskManager.Task::(_debug) Task b17a5d63-4b78-43ef-8d48-c73a976bbf88: moving from state preparing -> state finished >Thread-15913923::DEBUG::2013-11-20 03:17:35,962::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913923::DEBUG::2013-11-20 03:17:35,963::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913923::DEBUG::2013-11-20 03:17:35,963::task::495::TaskManager.Task::(_debug) Task b17a5d63-4b78-43ef-8d48-c73a976bbf88: ref 0 aborting False >Thread-15913923::INFO::2013-11-20 03:17:35,964::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913924::DEBUG::2013-11-20 03:17:35,969::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913924::INFO::2013-11-20 03:17:35,970::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913924::DEBUG::2013-11-20 03:17:35,971::task::495::TaskManager.Task::(_debug) Task 8dc1dea2-fb90-4fac-ab43-4a89a4207bf3: moving from state init -> state preparing >Thread-15913924::DEBUG::2013-11-20 03:17:35,972::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5e7a3b1b-8610-441c-8b93-990403b585c1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913924::DEBUG::2013-11-20 03:17:35,973::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913924::DEBUG::2013-11-20 03:17:35,973::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913924::DEBUG::2013-11-20 03:17:35,974::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5e7a3b1b-8610-441c-8b93-990403b585c1`::Granted request >Thread-15913924::DEBUG::2013-11-20 03:17:35,975::task::495::TaskManager.Task::(_debug) Task 8dc1dea2-fb90-4fac-ab43-4a89a4207bf3: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913924::DEBUG::2013-11-20 03:17:35,975::task::495::TaskManager.Task::(_debug) Task 8dc1dea2-fb90-4fac-ab43-4a89a4207bf3: ref 1 aborting False >Thread-15913924::WARNING::2013-11-20 03:17:35,977::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913924::DEBUG::2013-11-20 03:17:35,978::task::495::TaskManager.Task::(_debug) Task 8dc1dea2-fb90-4fac-ab43-4a89a4207bf3: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913924::DEBUG::2013-11-20 03:17:35,979::task::495::TaskManager.Task::(_debug) Task 8dc1dea2-fb90-4fac-ab43-4a89a4207bf3: moving from state preparing -> state finished >Thread-15913924::DEBUG::2013-11-20 03:17:35,980::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913924::DEBUG::2013-11-20 03:17:35,980::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913924::DEBUG::2013-11-20 03:17:35,981::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913924::DEBUG::2013-11-20 03:17:35,981::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913924::DEBUG::2013-11-20 03:17:35,982::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913924::DEBUG::2013-11-20 03:17:35,982::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913924::DEBUG::2013-11-20 03:17:35,983::task::495::TaskManager.Task::(_debug) Task 8dc1dea2-fb90-4fac-ab43-4a89a4207bf3: ref 0 aborting False >Thread-15913924::INFO::2013-11-20 03:17:35,984::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:17:37,162::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:37,685::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0749641 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15913658::INFO::2013-11-20 03:17:37,741::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:17:37,741::task::495::TaskManager.Task::(_debug) Task 2decd3dd-cd0b-4e91-95a2-8c68dd2e5b38: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:17:37,743::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`918832c0-7e5a-4db0-9ee3-75047ec0ba06`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:17:37,744::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:17:37,744::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:17:37,745::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`918832c0-7e5a-4db0-9ee3-75047ec0ba06`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:17:37,746::task::495::TaskManager.Task::(_debug) Task 2decd3dd-cd0b-4e91-95a2-8c68dd2e5b38: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:17:37,746::task::495::TaskManager.Task::(_debug) Task 2decd3dd-cd0b-4e91-95a2-8c68dd2e5b38: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:17:37,747::task::495::TaskManager.Task::(_debug) Task 2decd3dd-cd0b-4e91-95a2-8c68dd2e5b38: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:17:37,748::task::495::TaskManager.Task::(_debug) Task 2decd3dd-cd0b-4e91-95a2-8c68dd2e5b38: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:17:37,748::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:17:37,748::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:17:37,749::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:17:37,750::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:17:37,750::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:17:37,751::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:17:37,752::task::495::TaskManager.Task::(_debug) Task 2decd3dd-cd0b-4e91-95a2-8c68dd2e5b38: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:17:37,752::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:17:39,700::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:40,239::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.091199 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:17:42,256::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:42,777::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0775109 s, 13.2 MB/s\n'; <rc> = 0 >Thread-15912392::INFO::2013-11-20 03:17:42,990::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:17:42,991::task::495::TaskManager.Task::(_debug) Task 987bf7e3-5e4a-4387-b04b-737161dad3ac: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:17:42,993::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d439fdd6-c6a1-4ef8-8c86-901cbdafb3fd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:17:42,993::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:17:42,994::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:17:42,995::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d439fdd6-c6a1-4ef8-8c86-901cbdafb3fd`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:17:42,996::task::495::TaskManager.Task::(_debug) Task 987bf7e3-5e4a-4387-b04b-737161dad3ac: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:17:42,997::task::495::TaskManager.Task::(_debug) Task 987bf7e3-5e4a-4387-b04b-737161dad3ac: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:17:42,997::task::495::TaskManager.Task::(_debug) Task 987bf7e3-5e4a-4387-b04b-737161dad3ac: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:17:42,998::task::495::TaskManager.Task::(_debug) Task 987bf7e3-5e4a-4387-b04b-737161dad3ac: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:17:42,998::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:17:42,999::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:17:43,000::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:17:43,001::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:17:43,001::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:17:43,002::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:17:43,002::task::495::TaskManager.Task::(_debug) Task 987bf7e3-5e4a-4387-b04b-737161dad3ac: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:17:43,003::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:17:44,793::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:45,291::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0609346 s, 16.8 MB/s\n'; <rc> = 0 >Thread-15913928::DEBUG::2013-11-20 03:17:46,366::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913928::INFO::2013-11-20 03:17:46,368::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913928::DEBUG::2013-11-20 03:17:46,368::task::495::TaskManager.Task::(_debug) Task 9bcb81f7-aba3-4b8d-9d16-c4977a91569f: moving from state init -> state preparing >Thread-15913928::DEBUG::2013-11-20 03:17:46,369::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913928::DEBUG::2013-11-20 03:17:46,370::task::495::TaskManager.Task::(_debug) Task 9bcb81f7-aba3-4b8d-9d16-c4977a91569f: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913928::DEBUG::2013-11-20 03:17:46,371::task::495::TaskManager.Task::(_debug) Task 9bcb81f7-aba3-4b8d-9d16-c4977a91569f: moving from state preparing -> state finished >Thread-15913928::DEBUG::2013-11-20 03:17:46,371::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913928::DEBUG::2013-11-20 03:17:46,372::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913928::DEBUG::2013-11-20 03:17:46,373::task::495::TaskManager.Task::(_debug) Task 9bcb81f7-aba3-4b8d-9d16-c4977a91569f: ref 0 aborting False >Thread-15913928::INFO::2013-11-20 03:17:46,373::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913929::DEBUG::2013-11-20 03:17:46,378::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913929::INFO::2013-11-20 03:17:46,379::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913929::DEBUG::2013-11-20 03:17:46,380::task::495::TaskManager.Task::(_debug) Task 09972135-a189-43cd-b7c9-1ea7672fe889: moving from state init -> state preparing >Thread-15913929::DEBUG::2013-11-20 03:17:46,381::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6e9c9ee2-7056-4d35-ba2c-54c4b47e337a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913929::DEBUG::2013-11-20 03:17:46,381::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913929::DEBUG::2013-11-20 03:17:46,381::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913929::DEBUG::2013-11-20 03:17:46,382::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6e9c9ee2-7056-4d35-ba2c-54c4b47e337a`::Granted request >Thread-15913929::DEBUG::2013-11-20 03:17:46,382::task::495::TaskManager.Task::(_debug) Task 09972135-a189-43cd-b7c9-1ea7672fe889: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913929::DEBUG::2013-11-20 03:17:46,383::task::495::TaskManager.Task::(_debug) Task 09972135-a189-43cd-b7c9-1ea7672fe889: ref 1 aborting False >Thread-15913929::WARNING::2013-11-20 03:17:46,384::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913929::DEBUG::2013-11-20 03:17:46,385::task::495::TaskManager.Task::(_debug) Task 09972135-a189-43cd-b7c9-1ea7672fe889: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913929::DEBUG::2013-11-20 03:17:46,385::task::495::TaskManager.Task::(_debug) Task 09972135-a189-43cd-b7c9-1ea7672fe889: moving from state preparing -> state finished >Thread-15913929::DEBUG::2013-11-20 03:17:46,386::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913929::DEBUG::2013-11-20 03:17:46,386::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913929::DEBUG::2013-11-20 03:17:46,387::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913929::DEBUG::2013-11-20 03:17:46,387::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913929::DEBUG::2013-11-20 03:17:46,387::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913929::DEBUG::2013-11-20 03:17:46,388::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913929::DEBUG::2013-11-20 03:17:46,388::task::495::TaskManager.Task::(_debug) Task 09972135-a189-43cd-b7c9-1ea7672fe889: ref 0 aborting False >Thread-15913929::INFO::2013-11-20 03:17:46,388::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913930::INFO::2013-11-20 03:17:46,737::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913930::DEBUG::2013-11-20 03:17:46,738::task::495::TaskManager.Task::(_debug) Task 7683faca-a357-4ae1-83a4-bddadf9392f6: moving from state init -> state preparing >Thread-15913930::DEBUG::2013-11-20 03:17:46,739::task::495::TaskManager.Task::(_debug) Task 7683faca-a357-4ae1-83a4-bddadf9392f6: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000300884246826', 'lastCheck': 1384917463.9221289, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00141906738281', 'lastCheck': 1384917460.3949161, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0400328636169', 'lastCheck': 1384917462.9638219, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000539064407349', 'lastCheck': 1384917462.4630251, 'valid': True, 'code': 0}} >Thread-15913930::DEBUG::2013-11-20 03:17:46,740::task::495::TaskManager.Task::(_debug) Task 7683faca-a357-4ae1-83a4-bddadf9392f6: moving from state preparing -> state finished >Thread-15913930::DEBUG::2013-11-20 03:17:46,740::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913930::DEBUG::2013-11-20 03:17:46,741::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913930::DEBUG::2013-11-20 03:17:46,741::task::495::TaskManager.Task::(_debug) Task 7683faca-a357-4ae1-83a4-bddadf9392f6: ref 0 aborting False >Thread-15913930::INFO::2013-11-20 03:17:46,742::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000300884246826', 'lastCheck': 1384917463.9221289, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00141906738281', 'lastCheck': 1384917460.3949161, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0400328636169', 'lastCheck': 1384917462.9638219, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000539064407349', 'lastCheck': 1384917462.4630251, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:17:47,307::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:48,231::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.45613 s, 2.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:17:50,245::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:50,834::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0763936 s, 13.4 MB/s\n'; <rc> = 0 >Thread-15913518::INFO::2013-11-20 03:17:51,653::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:17:51,654::task::495::TaskManager.Task::(_debug) Task 45700514-a39d-4371-b6cd-eccfaab8c2bb: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:17:51,656::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c2fea0da-da87-4762-a8ef-b8f9d9ba5889`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:17:51,657::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:17:51,658::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:17:51,659::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c2fea0da-da87-4762-a8ef-b8f9d9ba5889`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:17:51,660::task::495::TaskManager.Task::(_debug) Task 45700514-a39d-4371-b6cd-eccfaab8c2bb: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:17:51,660::task::495::TaskManager.Task::(_debug) Task 45700514-a39d-4371-b6cd-eccfaab8c2bb: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:17:51,661::task::495::TaskManager.Task::(_debug) Task 45700514-a39d-4371-b6cd-eccfaab8c2bb: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:17:51,662::task::495::TaskManager.Task::(_debug) Task 45700514-a39d-4371-b6cd-eccfaab8c2bb: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:17:51,662::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:17:51,663::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:17:51,664::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:17:51,664::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:17:51,665::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:17:51,666::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:17:51,666::task::495::TaskManager.Task::(_debug) Task 45700514-a39d-4371-b6cd-eccfaab8c2bb: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:17:51,667::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:17:51,668::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:17:51,668::task::495::TaskManager.Task::(_debug) Task cd0008ad-dd6f-42da-865e-b65fb5fa107d: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:17:51,670::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`405dca0b-8edd-410c-99f6-dffb2f0b6bbc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:17:51,670::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:17:51,671::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:17:51,671::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`405dca0b-8edd-410c-99f6-dffb2f0b6bbc`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:17:51,672::task::495::TaskManager.Task::(_debug) Task cd0008ad-dd6f-42da-865e-b65fb5fa107d: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:17:51,673::task::495::TaskManager.Task::(_debug) Task cd0008ad-dd6f-42da-865e-b65fb5fa107d: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:17:51,674::task::495::TaskManager.Task::(_debug) Task cd0008ad-dd6f-42da-865e-b65fb5fa107d: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:17:51,674::task::495::TaskManager.Task::(_debug) Task cd0008ad-dd6f-42da-865e-b65fb5fa107d: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:17:51,675::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:17:51,675::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:17:51,676::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:17:51,677::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:17:51,677::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:17:51,678::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:17:51,679::task::495::TaskManager.Task::(_debug) Task cd0008ad-dd6f-42da-865e-b65fb5fa107d: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:17:51,679::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:17:52,847::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:53,380::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0873444 s, 11.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:17:55,396::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:55,932::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0518697 s, 19.7 MB/s\n'; <rc> = 0 >Thread-15913935::DEBUG::2013-11-20 03:17:56,608::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913935::INFO::2013-11-20 03:17:56,611::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913935::DEBUG::2013-11-20 03:17:56,613::task::495::TaskManager.Task::(_debug) Task 7a0093b4-f1ad-4299-8308-b7156afb5193: moving from state init -> state preparing >Thread-15913935::DEBUG::2013-11-20 03:17:56,616::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913935::DEBUG::2013-11-20 03:17:56,618::task::495::TaskManager.Task::(_debug) Task 7a0093b4-f1ad-4299-8308-b7156afb5193: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913935::DEBUG::2013-11-20 03:17:56,619::task::495::TaskManager.Task::(_debug) Task 7a0093b4-f1ad-4299-8308-b7156afb5193: moving from state preparing -> state finished >Thread-15913935::DEBUG::2013-11-20 03:17:56,621::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913935::DEBUG::2013-11-20 03:17:56,622::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913935::DEBUG::2013-11-20 03:17:56,625::task::495::TaskManager.Task::(_debug) Task 7a0093b4-f1ad-4299-8308-b7156afb5193: ref 0 aborting False >Thread-15913935::INFO::2013-11-20 03:17:56,628::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913936::DEBUG::2013-11-20 03:17:56,644::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913936::INFO::2013-11-20 03:17:56,655::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913936::DEBUG::2013-11-20 03:17:56,656::task::495::TaskManager.Task::(_debug) Task 23428aa1-0ace-4bfe-bc90-9c74356ceb1d: moving from state init -> state preparing >Thread-15913936::DEBUG::2013-11-20 03:17:56,658::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fcd869e2-233b-4a22-ad09-54665136cab0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913936::DEBUG::2013-11-20 03:17:56,658::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913936::DEBUG::2013-11-20 03:17:56,659::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913936::DEBUG::2013-11-20 03:17:56,660::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fcd869e2-233b-4a22-ad09-54665136cab0`::Granted request >Thread-15913936::DEBUG::2013-11-20 03:17:56,661::task::495::TaskManager.Task::(_debug) Task 23428aa1-0ace-4bfe-bc90-9c74356ceb1d: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913936::DEBUG::2013-11-20 03:17:56,662::task::495::TaskManager.Task::(_debug) Task 23428aa1-0ace-4bfe-bc90-9c74356ceb1d: ref 1 aborting False >Thread-15913936::WARNING::2013-11-20 03:17:56,664::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913936::DEBUG::2013-11-20 03:17:56,666::task::495::TaskManager.Task::(_debug) Task 23428aa1-0ace-4bfe-bc90-9c74356ceb1d: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913936::DEBUG::2013-11-20 03:17:56,667::task::495::TaskManager.Task::(_debug) Task 23428aa1-0ace-4bfe-bc90-9c74356ceb1d: moving from state preparing -> state finished >Thread-15913936::DEBUG::2013-11-20 03:17:56,667::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913936::DEBUG::2013-11-20 03:17:56,668::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913936::DEBUG::2013-11-20 03:17:56,669::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913936::DEBUG::2013-11-20 03:17:56,669::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913936::DEBUG::2013-11-20 03:17:56,670::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913936::DEBUG::2013-11-20 03:17:56,671::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913936::DEBUG::2013-11-20 03:17:56,671::task::495::TaskManager.Task::(_debug) Task 23428aa1-0ace-4bfe-bc90-9c74356ceb1d: ref 0 aborting False >Thread-15913936::INFO::2013-11-20 03:17:56,672::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:17:57,946::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:17:58,448::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.061842 s, 16.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:18:00,459::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:18:01,021::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0777301 s, 13.2 MB/s\n'; <rc> = 0 >Thread-15913938::INFO::2013-11-20 03:18:02,201::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913938::DEBUG::2013-11-20 03:18:02,202::task::495::TaskManager.Task::(_debug) Task 85d9e1a0-deeb-499a-abba-4a8b9712100e: moving from state init -> state preparing >Thread-15913938::DEBUG::2013-11-20 03:18:02,203::task::495::TaskManager.Task::(_debug) Task 85d9e1a0-deeb-499a-abba-4a8b9712100e: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000453948974609', 'lastCheck': 1384917474.0994971, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0089910030365', 'lastCheck': 1384917480.6765549, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00166487693787', 'lastCheck': 1384917473.104147, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000359058380127', 'lastCheck': 1384917472.5292311, 'valid': True, 'code': 0}} >Thread-15913938::DEBUG::2013-11-20 03:18:02,204::task::495::TaskManager.Task::(_debug) Task 85d9e1a0-deeb-499a-abba-4a8b9712100e: moving from state preparing -> state finished >Thread-15913938::DEBUG::2013-11-20 03:18:02,205::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913938::DEBUG::2013-11-20 03:18:02,206::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913938::DEBUG::2013-11-20 03:18:02,207::task::495::TaskManager.Task::(_debug) Task 85d9e1a0-deeb-499a-abba-4a8b9712100e: ref 0 aborting False >Thread-15913938::INFO::2013-11-20 03:18:02,208::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000453948974609', 'lastCheck': 1384917474.0994971, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0089910030365', 'lastCheck': 1384917480.6765549, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00166487693787', 'lastCheck': 1384917473.104147, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000359058380127', 'lastCheck': 1384917472.5292311, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:18:03,037::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:18:03,591::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0752684 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:18:05,602::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:18:06,117::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0753832 s, 13.6 MB/s\n'; <rc> = 0 >Thread-15913941::DEBUG::2013-11-20 03:18:06,928::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913941::INFO::2013-11-20 03:18:06,929::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913941::DEBUG::2013-11-20 03:18:06,930::task::495::TaskManager.Task::(_debug) Task 207eac8b-9f65-4d2a-b9f5-9e601cddfa89: moving from state init -> state preparing >Thread-15913941::DEBUG::2013-11-20 03:18:06,931::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913941::DEBUG::2013-11-20 03:18:06,931::task::495::TaskManager.Task::(_debug) Task 207eac8b-9f65-4d2a-b9f5-9e601cddfa89: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913941::DEBUG::2013-11-20 03:18:06,932::task::495::TaskManager.Task::(_debug) Task 207eac8b-9f65-4d2a-b9f5-9e601cddfa89: moving from state preparing -> state finished >Thread-15913941::DEBUG::2013-11-20 03:18:06,933::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913941::DEBUG::2013-11-20 03:18:06,933::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913941::DEBUG::2013-11-20 03:18:06,934::task::495::TaskManager.Task::(_debug) Task 207eac8b-9f65-4d2a-b9f5-9e601cddfa89: ref 0 aborting False >Thread-15913941::INFO::2013-11-20 03:18:06,934::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913942::DEBUG::2013-11-20 03:18:06,939::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913942::INFO::2013-11-20 03:18:06,940::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913942::DEBUG::2013-11-20 03:18:06,941::task::495::TaskManager.Task::(_debug) Task b9818109-1a3c-4d22-9c70-125be7fbcbdd: moving from state init -> state preparing >Thread-15913942::DEBUG::2013-11-20 03:18:06,943::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`39b7f09c-2453-47f3-83f0-6a7e692c5f05`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913942::DEBUG::2013-11-20 03:18:06,944::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913942::DEBUG::2013-11-20 03:18:06,945::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913942::DEBUG::2013-11-20 03:18:06,946::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`39b7f09c-2453-47f3-83f0-6a7e692c5f05`::Granted request >Thread-15913942::DEBUG::2013-11-20 03:18:06,948::task::495::TaskManager.Task::(_debug) Task b9818109-1a3c-4d22-9c70-125be7fbcbdd: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913942::DEBUG::2013-11-20 03:18:06,948::task::495::TaskManager.Task::(_debug) Task b9818109-1a3c-4d22-9c70-125be7fbcbdd: ref 1 aborting False >Thread-15913942::WARNING::2013-11-20 03:18:06,952::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913942::DEBUG::2013-11-20 03:18:06,953::task::495::TaskManager.Task::(_debug) Task b9818109-1a3c-4d22-9c70-125be7fbcbdd: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913942::DEBUG::2013-11-20 03:18:06,954::task::495::TaskManager.Task::(_debug) Task b9818109-1a3c-4d22-9c70-125be7fbcbdd: moving from state preparing -> state finished >Thread-15913942::DEBUG::2013-11-20 03:18:06,955::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913942::DEBUG::2013-11-20 03:18:06,955::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913942::DEBUG::2013-11-20 03:18:06,956::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913942::DEBUG::2013-11-20 03:18:06,956::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913942::DEBUG::2013-11-20 03:18:06,956::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913942::DEBUG::2013-11-20 03:18:06,957::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913942::DEBUG::2013-11-20 03:18:06,958::task::495::TaskManager.Task::(_debug) Task b9818109-1a3c-4d22-9c70-125be7fbcbdd: ref 0 aborting False >Thread-15913942::INFO::2013-11-20 03:18:06,958::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:18:08,133::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913546::INFO::2013-11-20 03:18:08,328::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:18:08,329::task::495::TaskManager.Task::(_debug) Task 00831382-e8db-4e73-ac90-781a272248ff: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:18:08,333::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`31fe8bc7-03c0-49fd-8a78-3622cdd1a9ac`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:18:08,334::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:18:08,336::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:18:08,337::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`31fe8bc7-03c0-49fd-8a78-3622cdd1a9ac`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:18:08,338::task::495::TaskManager.Task::(_debug) Task 00831382-e8db-4e73-ac90-781a272248ff: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:18:08,339::task::495::TaskManager.Task::(_debug) Task 00831382-e8db-4e73-ac90-781a272248ff: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:18:08,340::task::495::TaskManager.Task::(_debug) Task 00831382-e8db-4e73-ac90-781a272248ff: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:18:08,340::task::495::TaskManager.Task::(_debug) Task 00831382-e8db-4e73-ac90-781a272248ff: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:18:08,341::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:18:08,341::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:18:08,342::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:18:08,343::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:18:08,344::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:18:08,344::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:18:08,345::task::495::TaskManager.Task::(_debug) Task 00831382-e8db-4e73-ac90-781a272248ff: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:18:08,345::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:18:08,346::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:18:08,347::task::495::TaskManager.Task::(_debug) Task 7080fb05-e3b2-4185-bc01-130302102f41: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:18:08,349::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`40e9d801-b197-46e2-9a16-0e10d9d1be7f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:18:08,350::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:18:08,351::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:18:08,352::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`40e9d801-b197-46e2-9a16-0e10d9d1be7f`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:18:08,353::task::495::TaskManager.Task::(_debug) Task 7080fb05-e3b2-4185-bc01-130302102f41: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:18:08,354::task::495::TaskManager.Task::(_debug) Task 7080fb05-e3b2-4185-bc01-130302102f41: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:18:08,355::task::495::TaskManager.Task::(_debug) Task 7080fb05-e3b2-4185-bc01-130302102f41: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:18:08,355::task::495::TaskManager.Task::(_debug) Task 7080fb05-e3b2-4185-bc01-130302102f41: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:18:08,356::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:18:08,356::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:18:08,357::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:18:08,357::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:18:08,358::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:18:08,358::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:18:08,359::task::495::TaskManager.Task::(_debug) Task 7080fb05-e3b2-4185-bc01-130302102f41: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:18:08,359::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:18:08,360::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:18:08,360::task::495::TaskManager.Task::(_debug) Task 0d53c51e-e666-477c-a992-ded599dc7c6a: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:18:08,361::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fb1642aa-52ac-4d89-9399-085e43b0ea51`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:18:08,362::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:18:08,362::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:18:08,363::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fb1642aa-52ac-4d89-9399-085e43b0ea51`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:18:08,363::task::495::TaskManager.Task::(_debug) Task 0d53c51e-e666-477c-a992-ded599dc7c6a: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:18:08,364::task::495::TaskManager.Task::(_debug) Task 0d53c51e-e666-477c-a992-ded599dc7c6a: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:18:08,364::task::495::TaskManager.Task::(_debug) Task 0d53c51e-e666-477c-a992-ded599dc7c6a: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:18:08,365::task::495::TaskManager.Task::(_debug) Task 0d53c51e-e666-477c-a992-ded599dc7c6a: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:18:08,365::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:18:08,366::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:18:08,366::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:18:08,367::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:18:08,367::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:18:08,368::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:18:08,368::task::495::TaskManager.Task::(_debug) Task 0d53c51e-e666-477c-a992-ded599dc7c6a: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:18:08,369::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:18:08,369::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:18:08,370::task::495::TaskManager.Task::(_debug) Task 4209f755-bd4a-486d-a569-c3cb963e63a4: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:18:08,371::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`bf6d7259-5137-476f-95da-25927047e6df`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:18:08,371::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:18:08,371::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:18:08,372::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`bf6d7259-5137-476f-95da-25927047e6df`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:18:08,373::task::495::TaskManager.Task::(_debug) Task 4209f755-bd4a-486d-a569-c3cb963e63a4: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:18:08,373::task::495::TaskManager.Task::(_debug) Task 4209f755-bd4a-486d-a569-c3cb963e63a4: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:18:08,374::task::495::TaskManager.Task::(_debug) Task 4209f755-bd4a-486d-a569-c3cb963e63a4: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:18:08,374::task::495::TaskManager.Task::(_debug) Task 4209f755-bd4a-486d-a569-c3cb963e63a4: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:18:08,374::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:18:08,375::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:18:08,375::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:18:08,376::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:18:08,376::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:18:08,377::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:18:08,377::task::495::TaskManager.Task::(_debug) Task 4209f755-bd4a-486d-a569-c3cb963e63a4: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:18:08,378::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:18:08,690::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.119183 s, 8.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:18:10,706::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:18:11,229::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0751744 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:18:13,243::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:18:13,818::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0970897 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:18:15,835::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:18:16,388::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.083207 s, 12.3 MB/s\n'; <rc> = 0 >Thread-15913946::DEBUG::2013-11-20 03:18:17,169::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913946::INFO::2013-11-20 03:18:17,170::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913946::DEBUG::2013-11-20 03:18:17,171::task::495::TaskManager.Task::(_debug) Task 6cdc11ce-a576-4dbe-b98c-1f2da384e22c: moving from state init -> state preparing >Thread-15913946::DEBUG::2013-11-20 03:18:17,172::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913946::DEBUG::2013-11-20 03:18:17,173::task::495::TaskManager.Task::(_debug) Task 6cdc11ce-a576-4dbe-b98c-1f2da384e22c: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913946::DEBUG::2013-11-20 03:18:17,174::task::495::TaskManager.Task::(_debug) Task 6cdc11ce-a576-4dbe-b98c-1f2da384e22c: moving from state preparing -> state finished >Thread-15913946::DEBUG::2013-11-20 03:18:17,174::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913946::DEBUG::2013-11-20 03:18:17,175::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913946::DEBUG::2013-11-20 03:18:17,175::task::495::TaskManager.Task::(_debug) Task 6cdc11ce-a576-4dbe-b98c-1f2da384e22c: ref 0 aborting False >Thread-15913946::INFO::2013-11-20 03:18:17,176::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913947::DEBUG::2013-11-20 03:18:17,180::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913947::INFO::2013-11-20 03:18:17,181::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913947::DEBUG::2013-11-20 03:18:17,182::task::495::TaskManager.Task::(_debug) Task 919bce8b-8b7a-4aeb-a74f-80b77aaad493: moving from state init -> state preparing >Thread-15913947::DEBUG::2013-11-20 03:18:17,191::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9c5b9410-52d1-42bd-b508-da97201260f8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913947::DEBUG::2013-11-20 03:18:17,192::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913947::DEBUG::2013-11-20 03:18:17,193::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913947::DEBUG::2013-11-20 03:18:17,193::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9c5b9410-52d1-42bd-b508-da97201260f8`::Granted request >Thread-15913947::DEBUG::2013-11-20 03:18:17,194::task::495::TaskManager.Task::(_debug) Task 919bce8b-8b7a-4aeb-a74f-80b77aaad493: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913947::DEBUG::2013-11-20 03:18:17,195::task::495::TaskManager.Task::(_debug) Task 919bce8b-8b7a-4aeb-a74f-80b77aaad493: ref 1 aborting False >Thread-15913947::WARNING::2013-11-20 03:18:17,197::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913947::DEBUG::2013-11-20 03:18:17,198::task::495::TaskManager.Task::(_debug) Task 919bce8b-8b7a-4aeb-a74f-80b77aaad493: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913947::DEBUG::2013-11-20 03:18:17,199::task::495::TaskManager.Task::(_debug) Task 919bce8b-8b7a-4aeb-a74f-80b77aaad493: moving from state preparing -> state finished >Thread-15913947::DEBUG::2013-11-20 03:18:17,200::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913947::DEBUG::2013-11-20 03:18:17,200::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913947::DEBUG::2013-11-20 03:18:17,201::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913947::DEBUG::2013-11-20 03:18:17,202::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913947::DEBUG::2013-11-20 03:18:17,202::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913947::DEBUG::2013-11-20 03:18:17,203::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913947::DEBUG::2013-11-20 03:18:17,203::task::495::TaskManager.Task::(_debug) Task 919bce8b-8b7a-4aeb-a74f-80b77aaad493: ref 0 aborting False >Thread-15913947::INFO::2013-11-20 03:18:17,204::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913948::INFO::2013-11-20 03:18:17,927::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913948::DEBUG::2013-11-20 03:18:17,928::task::495::TaskManager.Task::(_debug) Task 133f1d8e-ebf4-4863-8c45-865870f7973d: moving from state init -> state preparing >Thread-15913948::DEBUG::2013-11-20 03:18:17,929::task::495::TaskManager.Task::(_debug) Task 133f1d8e-ebf4-4863-8c45-865870f7973d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000332832336426', 'lastCheck': 1384917494.4630909, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142002105713', 'lastCheck': 1384917490.899847, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0138359069824', 'lastCheck': 1384917493.4850581, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000556945800781', 'lastCheck': 1384917492.8946681, 'valid': True, 'code': 0}} >Thread-15913948::DEBUG::2013-11-20 03:18:17,930::task::495::TaskManager.Task::(_debug) Task 133f1d8e-ebf4-4863-8c45-865870f7973d: moving from state preparing -> state finished >Thread-15913948::DEBUG::2013-11-20 03:18:17,930::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913948::DEBUG::2013-11-20 03:18:17,931::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913948::DEBUG::2013-11-20 03:18:17,932::task::495::TaskManager.Task::(_debug) Task 133f1d8e-ebf4-4863-8c45-865870f7973d: ref 0 aborting False >Thread-15913948::INFO::2013-11-20 03:18:17,932::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000332832336426', 'lastCheck': 1384917494.4630909, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142002105713', 'lastCheck': 1384917490.899847, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0138359069824', 'lastCheck': 1384917493.4850581, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000556945800781', 'lastCheck': 1384917492.8946681, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:18:18,407::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913639::INFO::2013-11-20 03:18:18,633::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:18:18,636::task::495::TaskManager.Task::(_debug) Task 5fd1e375-f0b0-406c-8937-4b84cded6e87: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:18:18,639::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e3715110-47f4-42b3-89ab-97163ae8f8b0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:18:18,639::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:18:18,641::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:18:18,641::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e3715110-47f4-42b3-89ab-97163ae8f8b0`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:18:18,642::task::495::TaskManager.Task::(_debug) Task 5fd1e375-f0b0-406c-8937-4b84cded6e87: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:18:18,643::task::495::TaskManager.Task::(_debug) Task 5fd1e375-f0b0-406c-8937-4b84cded6e87: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:18:18,644::task::495::TaskManager.Task::(_debug) Task 5fd1e375-f0b0-406c-8937-4b84cded6e87: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:18:18,644::task::495::TaskManager.Task::(_debug) Task 5fd1e375-f0b0-406c-8937-4b84cded6e87: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:18:18,645::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:18:18,645::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:18:18,646::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:18:18,646::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:18:18,647::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:18:18,647::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:18:18,648::task::495::TaskManager.Task::(_debug) Task 5fd1e375-f0b0-406c-8937-4b84cded6e87: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:18:18,649::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:18:18,960::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0758006 s, 13.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:18:20,973::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:18:21,537::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0912544 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:18:23,553::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:18:24,102::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0554114 s, 18.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:18:26,117::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:18:26,717::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0764485 s, 13.4 MB/s\n'; <rc> = 0 >Thread-15913952::DEBUG::2013-11-20 03:18:27,358::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913952::INFO::2013-11-20 03:18:27,360::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913952::DEBUG::2013-11-20 03:18:27,360::task::495::TaskManager.Task::(_debug) Task e25b978b-3beb-4b13-a6c9-11977f811988: moving from state init -> state preparing >Thread-15913952::DEBUG::2013-11-20 03:18:27,361::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913952::DEBUG::2013-11-20 03:18:27,362::task::495::TaskManager.Task::(_debug) Task e25b978b-3beb-4b13-a6c9-11977f811988: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913952::DEBUG::2013-11-20 03:18:27,362::task::495::TaskManager.Task::(_debug) Task e25b978b-3beb-4b13-a6c9-11977f811988: moving from state preparing -> state finished >Thread-15913952::DEBUG::2013-11-20 03:18:27,363::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913952::DEBUG::2013-11-20 03:18:27,363::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913952::DEBUG::2013-11-20 03:18:27,364::task::495::TaskManager.Task::(_debug) Task e25b978b-3beb-4b13-a6c9-11977f811988: ref 0 aborting False >Thread-15913952::INFO::2013-11-20 03:18:27,365::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913953::DEBUG::2013-11-20 03:18:27,370::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913953::INFO::2013-11-20 03:18:27,371::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913953::DEBUG::2013-11-20 03:18:27,372::task::495::TaskManager.Task::(_debug) Task 1d70bfd7-7a64-410a-83ad-25b6bbab134b: moving from state init -> state preparing >Thread-15913953::DEBUG::2013-11-20 03:18:27,374::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e832fd39-d8cb-4657-9ffe-123dc5c4fce2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913953::DEBUG::2013-11-20 03:18:27,374::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913953::DEBUG::2013-11-20 03:18:27,375::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913953::DEBUG::2013-11-20 03:18:27,376::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e832fd39-d8cb-4657-9ffe-123dc5c4fce2`::Granted request >Thread-15913953::DEBUG::2013-11-20 03:18:27,377::task::495::TaskManager.Task::(_debug) Task 1d70bfd7-7a64-410a-83ad-25b6bbab134b: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913953::DEBUG::2013-11-20 03:18:27,378::task::495::TaskManager.Task::(_debug) Task 1d70bfd7-7a64-410a-83ad-25b6bbab134b: ref 1 aborting False >Thread-15913953::WARNING::2013-11-20 03:18:27,381::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913953::DEBUG::2013-11-20 03:18:27,383::task::495::TaskManager.Task::(_debug) Task 1d70bfd7-7a64-410a-83ad-25b6bbab134b: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913953::DEBUG::2013-11-20 03:18:27,384::task::495::TaskManager.Task::(_debug) Task 1d70bfd7-7a64-410a-83ad-25b6bbab134b: moving from state preparing -> state finished >Thread-15913953::DEBUG::2013-11-20 03:18:27,385::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913953::DEBUG::2013-11-20 03:18:27,386::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913953::DEBUG::2013-11-20 03:18:27,387::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913953::DEBUG::2013-11-20 03:18:27,388::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913953::DEBUG::2013-11-20 03:18:27,388::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913953::DEBUG::2013-11-20 03:18:27,389::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913953::DEBUG::2013-11-20 03:18:27,390::task::495::TaskManager.Task::(_debug) Task 1d70bfd7-7a64-410a-83ad-25b6bbab134b: ref 0 aborting False >Thread-15913953::INFO::2013-11-20 03:18:27,391::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:18:28,728::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:18:29,228::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0916001 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:18:31,243::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:18:31,768::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0527554 s, 19.4 MB/s\n'; <rc> = 0 >Thread-15913956::INFO::2013-11-20 03:18:33,471::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913956::DEBUG::2013-11-20 03:18:33,472::task::495::TaskManager.Task::(_debug) Task e663208b-22f3-495f-90c7-a08d879b114f: moving from state init -> state preparing >Thread-15913956::DEBUG::2013-11-20 03:18:33,473::task::495::TaskManager.Task::(_debug) Task e663208b-22f3-495f-90c7-a08d879b114f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000425100326538', 'lastCheck': 1384917504.795975, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154590606689', 'lastCheck': 1384917511.49985, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.017856836319', 'lastCheck': 1384917503.8996749, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000473976135254', 'lastCheck': 1384917503.1694419, 'valid': True, 'code': 0}} >Thread-15913956::DEBUG::2013-11-20 03:18:33,474::task::495::TaskManager.Task::(_debug) Task e663208b-22f3-495f-90c7-a08d879b114f: moving from state preparing -> state finished >Thread-15913956::DEBUG::2013-11-20 03:18:33,475::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913956::DEBUG::2013-11-20 03:18:33,475::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913956::DEBUG::2013-11-20 03:18:33,476::task::495::TaskManager.Task::(_debug) Task e663208b-22f3-495f-90c7-a08d879b114f: ref 0 aborting False >Thread-15913956::INFO::2013-11-20 03:18:33,477::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000425100326538', 'lastCheck': 1384917504.795975, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154590606689', 'lastCheck': 1384917511.49985, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.017856836319', 'lastCheck': 1384917503.8996749, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000473976135254', 'lastCheck': 1384917503.1694419, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:18:33,781::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:18:34,319::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.101118 s, 10.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:18:36,334::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913695::INFO::2013-11-20 03:18:36,538::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:18:36,540::task::495::TaskManager.Task::(_debug) Task a752e3e6-9741-449c-ace9-d787b0b3c58f: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:18:36,543::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`8a2e3a97-3f21-4080-ba16-d7353d50d65f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:18:36,544::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:18:36,545::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:18:36,546::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`8a2e3a97-3f21-4080-ba16-d7353d50d65f`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:18:36,548::task::495::TaskManager.Task::(_debug) Task a752e3e6-9741-449c-ace9-d787b0b3c58f: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:18:36,549::task::495::TaskManager.Task::(_debug) Task a752e3e6-9741-449c-ace9-d787b0b3c58f: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:18:36,550::task::495::TaskManager.Task::(_debug) Task a752e3e6-9741-449c-ace9-d787b0b3c58f: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:18:36,551::task::495::TaskManager.Task::(_debug) Task a752e3e6-9741-449c-ace9-d787b0b3c58f: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:18:36,552::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:18:36,553::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:18:36,554::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:18:36,555::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:18:36,556::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:18:36,557::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:18:36,558::task::495::TaskManager.Task::(_debug) Task a752e3e6-9741-449c-ace9-d787b0b3c58f: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:18:36,558::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:18:36,865::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.081769 s, 12.5 MB/s\n'; <rc> = 0 >Thread-15913959::DEBUG::2013-11-20 03:18:38,230::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913959::INFO::2013-11-20 03:18:38,231::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913959::DEBUG::2013-11-20 03:18:38,231::task::495::TaskManager.Task::(_debug) Task f71ae104-3ed0-43a0-834a-e06e6e549dc4: moving from state init -> state preparing >Thread-15913959::DEBUG::2013-11-20 03:18:38,232::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913959::DEBUG::2013-11-20 03:18:38,233::task::495::TaskManager.Task::(_debug) Task f71ae104-3ed0-43a0-834a-e06e6e549dc4: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913959::DEBUG::2013-11-20 03:18:38,233::task::495::TaskManager.Task::(_debug) Task f71ae104-3ed0-43a0-834a-e06e6e549dc4: moving from state preparing -> state finished >Thread-15913959::DEBUG::2013-11-20 03:18:38,234::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913959::DEBUG::2013-11-20 03:18:38,234::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913959::DEBUG::2013-11-20 03:18:38,235::task::495::TaskManager.Task::(_debug) Task f71ae104-3ed0-43a0-834a-e06e6e549dc4: ref 0 aborting False >Thread-15913959::INFO::2013-11-20 03:18:38,235::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913960::DEBUG::2013-11-20 03:18:38,240::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913960::INFO::2013-11-20 03:18:38,241::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913960::DEBUG::2013-11-20 03:18:38,241::task::495::TaskManager.Task::(_debug) Task fd5c2052-c7c4-439f-8075-991008a2d0e7: moving from state init -> state preparing >Thread-15913960::DEBUG::2013-11-20 03:18:38,242::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ddd272ae-edb6-434d-85ef-7109a792a534`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913960::DEBUG::2013-11-20 03:18:38,243::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913960::DEBUG::2013-11-20 03:18:38,244::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913960::DEBUG::2013-11-20 03:18:38,244::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ddd272ae-edb6-434d-85ef-7109a792a534`::Granted request >Thread-15913960::DEBUG::2013-11-20 03:18:38,245::task::495::TaskManager.Task::(_debug) Task fd5c2052-c7c4-439f-8075-991008a2d0e7: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913960::DEBUG::2013-11-20 03:18:38,246::task::495::TaskManager.Task::(_debug) Task fd5c2052-c7c4-439f-8075-991008a2d0e7: ref 1 aborting False >Thread-15913960::WARNING::2013-11-20 03:18:38,248::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913960::DEBUG::2013-11-20 03:18:38,248::task::495::TaskManager.Task::(_debug) Task fd5c2052-c7c4-439f-8075-991008a2d0e7: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913960::DEBUG::2013-11-20 03:18:38,249::task::495::TaskManager.Task::(_debug) Task fd5c2052-c7c4-439f-8075-991008a2d0e7: moving from state preparing -> state finished >Thread-15913960::DEBUG::2013-11-20 03:18:38,250::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913960::DEBUG::2013-11-20 03:18:38,251::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913960::DEBUG::2013-11-20 03:18:38,251::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913960::DEBUG::2013-11-20 03:18:38,252::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913960::DEBUG::2013-11-20 03:18:38,253::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913960::DEBUG::2013-11-20 03:18:38,253::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913960::DEBUG::2013-11-20 03:18:38,254::task::495::TaskManager.Task::(_debug) Task fd5c2052-c7c4-439f-8075-991008a2d0e7: ref 0 aborting False >Thread-15913960::INFO::2013-11-20 03:18:38,255::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:18:38,880::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913658::INFO::2013-11-20 03:18:39,134::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:18:39,137::task::495::TaskManager.Task::(_debug) Task e20f53ee-2903-4830-b672-6af2e246a87a: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:18:39,142::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e1c3a1bf-a20a-4acd-985a-df52b273cd3b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:18:39,143::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:18:39,144::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:18:39,145::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e1c3a1bf-a20a-4acd-985a-df52b273cd3b`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:18:39,146::task::495::TaskManager.Task::(_debug) Task e20f53ee-2903-4830-b672-6af2e246a87a: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:18:39,146::task::495::TaskManager.Task::(_debug) Task e20f53ee-2903-4830-b672-6af2e246a87a: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:18:39,147::task::495::TaskManager.Task::(_debug) Task e20f53ee-2903-4830-b672-6af2e246a87a: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:18:39,148::task::495::TaskManager.Task::(_debug) Task e20f53ee-2903-4830-b672-6af2e246a87a: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:18:39,148::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:18:39,149::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:18:39,150::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:18:39,150::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:18:39,151::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:18:39,151::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:18:39,152::task::495::TaskManager.Task::(_debug) Task e20f53ee-2903-4830-b672-6af2e246a87a: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:18:39,152::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:18:39,465::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0833504 s, 12.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:18:41,477::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:18:42,007::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0744888 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:18:44,022::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 03:18:44,245::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:18:44,249::task::495::TaskManager.Task::(_debug) Task 7b633167-aec8-4a83-bfc9-2aa4d78a8648: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:18:44,256::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`735fc08f-3878-42f6-b4e5-39c8dec66cbf`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:18:44,256::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:18:44,257::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:18:44,258::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`735fc08f-3878-42f6-b4e5-39c8dec66cbf`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:18:44,259::task::495::TaskManager.Task::(_debug) Task 7b633167-aec8-4a83-bfc9-2aa4d78a8648: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:18:44,260::task::495::TaskManager.Task::(_debug) Task 7b633167-aec8-4a83-bfc9-2aa4d78a8648: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:18:44,261::task::495::TaskManager.Task::(_debug) Task 7b633167-aec8-4a83-bfc9-2aa4d78a8648: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:18:44,261::task::495::TaskManager.Task::(_debug) Task 7b633167-aec8-4a83-bfc9-2aa4d78a8648: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:18:44,262::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:18:44,262::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:18:44,263::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:18:44,264::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:18:44,264::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:18:44,264::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:18:44,265::task::495::TaskManager.Task::(_debug) Task 7b633167-aec8-4a83-bfc9-2aa4d78a8648: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:18:44,266::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:18:44,596::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0717831 s, 14.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:18:46,609::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:18:47,209::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0872519 s, 11.7 MB/s\n'; <rc> = 0 >Thread-15913964::DEBUG::2013-11-20 03:18:48,452::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913964::INFO::2013-11-20 03:18:48,453::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913964::DEBUG::2013-11-20 03:18:48,454::task::495::TaskManager.Task::(_debug) Task 4cd402bb-bfc4-4cab-afef-75f08f1def6f: moving from state init -> state preparing >Thread-15913964::DEBUG::2013-11-20 03:18:48,455::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913964::DEBUG::2013-11-20 03:18:48,456::task::495::TaskManager.Task::(_debug) Task 4cd402bb-bfc4-4cab-afef-75f08f1def6f: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913964::DEBUG::2013-11-20 03:18:48,457::task::495::TaskManager.Task::(_debug) Task 4cd402bb-bfc4-4cab-afef-75f08f1def6f: moving from state preparing -> state finished >Thread-15913964::DEBUG::2013-11-20 03:18:48,458::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913964::DEBUG::2013-11-20 03:18:48,458::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913964::DEBUG::2013-11-20 03:18:48,459::task::495::TaskManager.Task::(_debug) Task 4cd402bb-bfc4-4cab-afef-75f08f1def6f: ref 0 aborting False >Thread-15913964::INFO::2013-11-20 03:18:48,459::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913965::DEBUG::2013-11-20 03:18:48,465::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913965::INFO::2013-11-20 03:18:48,466::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913965::DEBUG::2013-11-20 03:18:48,467::task::495::TaskManager.Task::(_debug) Task be864e9d-173a-4fe1-aa5a-cdf6456d136a: moving from state init -> state preparing >Thread-15913965::DEBUG::2013-11-20 03:18:48,468::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`85b5e3bc-9dad-457a-b9ac-04fead6daa6c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913965::DEBUG::2013-11-20 03:18:48,469::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913965::DEBUG::2013-11-20 03:18:48,470::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913965::DEBUG::2013-11-20 03:18:48,470::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`85b5e3bc-9dad-457a-b9ac-04fead6daa6c`::Granted request >Thread-15913965::DEBUG::2013-11-20 03:18:48,471::task::495::TaskManager.Task::(_debug) Task be864e9d-173a-4fe1-aa5a-cdf6456d136a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913965::DEBUG::2013-11-20 03:18:48,472::task::495::TaskManager.Task::(_debug) Task be864e9d-173a-4fe1-aa5a-cdf6456d136a: ref 1 aborting False >Thread-15913965::WARNING::2013-11-20 03:18:48,474::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913965::DEBUG::2013-11-20 03:18:48,475::task::495::TaskManager.Task::(_debug) Task be864e9d-173a-4fe1-aa5a-cdf6456d136a: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913965::DEBUG::2013-11-20 03:18:48,476::task::495::TaskManager.Task::(_debug) Task be864e9d-173a-4fe1-aa5a-cdf6456d136a: moving from state preparing -> state finished >Thread-15913965::DEBUG::2013-11-20 03:18:48,476::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913965::DEBUG::2013-11-20 03:18:48,477::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913965::DEBUG::2013-11-20 03:18:48,477::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913965::DEBUG::2013-11-20 03:18:48,478::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913965::DEBUG::2013-11-20 03:18:48,478::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913965::DEBUG::2013-11-20 03:18:48,479::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913965::DEBUG::2013-11-20 03:18:48,480::task::495::TaskManager.Task::(_debug) Task be864e9d-173a-4fe1-aa5a-cdf6456d136a: ref 0 aborting False >Thread-15913965::INFO::2013-11-20 03:18:48,480::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913966::INFO::2013-11-20 03:18:49,007::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913966::DEBUG::2013-11-20 03:18:49,008::task::495::TaskManager.Task::(_debug) Task 16129398-459b-4070-8a56-33d7e96bf264: moving from state init -> state preparing >Thread-15913966::DEBUG::2013-11-20 03:18:49,009::task::495::TaskManager.Task::(_debug) Task 16129398-459b-4070-8a56-33d7e96bf264: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000310897827148', 'lastCheck': 1384917525.245492, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00132393836975', 'lastCheck': 1384917521.69717, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0250089168549', 'lastCheck': 1384917524.2804661, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00034499168396', 'lastCheck': 1384917523.691849, 'valid': True, 'code': 0}} >Thread-15913966::DEBUG::2013-11-20 03:18:49,010::task::495::TaskManager.Task::(_debug) Task 16129398-459b-4070-8a56-33d7e96bf264: moving from state preparing -> state finished >Thread-15913966::DEBUG::2013-11-20 03:18:49,010::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913966::DEBUG::2013-11-20 03:18:49,011::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913966::DEBUG::2013-11-20 03:18:49,012::task::495::TaskManager.Task::(_debug) Task 16129398-459b-4070-8a56-33d7e96bf264: ref 0 aborting False >Thread-15913966::INFO::2013-11-20 03:18:49,012::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000310897827148', 'lastCheck': 1384917525.245492, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00132393836975', 'lastCheck': 1384917521.69717, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0250089168549', 'lastCheck': 1384917524.2804661, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00034499168396', 'lastCheck': 1384917523.691849, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:18:49,225::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:18:49,753::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.072798 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:18:51,768::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:18:52,359::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0820192 s, 12.5 MB/s\n'; <rc> = 0 >Thread-15913518::INFO::2013-11-20 03:18:52,955::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:18:52,956::task::495::TaskManager.Task::(_debug) Task eace0933-9966-4d5a-a1bb-154432f9adaf: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:18:52,958::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`1e35d681-8734-40fb-a032-619293dfde5e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:18:52,959::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:18:52,960::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:18:52,960::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`1e35d681-8734-40fb-a032-619293dfde5e`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:18:52,961::task::495::TaskManager.Task::(_debug) Task eace0933-9966-4d5a-a1bb-154432f9adaf: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:18:52,962::task::495::TaskManager.Task::(_debug) Task eace0933-9966-4d5a-a1bb-154432f9adaf: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:18:52,963::task::495::TaskManager.Task::(_debug) Task eace0933-9966-4d5a-a1bb-154432f9adaf: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:18:52,963::task::495::TaskManager.Task::(_debug) Task eace0933-9966-4d5a-a1bb-154432f9adaf: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:18:52,964::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:18:52,965::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:18:52,966::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:18:52,967::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:18:52,967::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:18:52,968::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:18:52,969::task::495::TaskManager.Task::(_debug) Task eace0933-9966-4d5a-a1bb-154432f9adaf: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:18:52,970::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:18:52,971::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:18:52,971::task::495::TaskManager.Task::(_debug) Task 29a91c77-d89c-4c7d-bc17-e571e5c53b62: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:18:52,973::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`25e9e545-a034-4012-b2c5-02eaac2a0c39`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:18:52,973::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:18:52,974::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:18:52,974::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`25e9e545-a034-4012-b2c5-02eaac2a0c39`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:18:52,975::task::495::TaskManager.Task::(_debug) Task 29a91c77-d89c-4c7d-bc17-e571e5c53b62: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:18:52,976::task::495::TaskManager.Task::(_debug) Task 29a91c77-d89c-4c7d-bc17-e571e5c53b62: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:18:52,977::task::495::TaskManager.Task::(_debug) Task 29a91c77-d89c-4c7d-bc17-e571e5c53b62: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:18:52,977::task::495::TaskManager.Task::(_debug) Task 29a91c77-d89c-4c7d-bc17-e571e5c53b62: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:18:52,978::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:18:52,978::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:18:52,979::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:18:52,980::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:18:52,980::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:18:52,980::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:18:52,981::task::495::TaskManager.Task::(_debug) Task 29a91c77-d89c-4c7d-bc17-e571e5c53b62: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:18:52,982::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:18:54,373::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:18:54,938::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.080938 s, 12.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:18:56,952::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:18:57,486::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0804723 s, 12.7 MB/s\n'; <rc> = 0 >Thread-15913971::DEBUG::2013-11-20 03:18:58,672::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913971::INFO::2013-11-20 03:18:58,674::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913971::DEBUG::2013-11-20 03:18:58,675::task::495::TaskManager.Task::(_debug) Task 8912b6e5-209c-42d2-aa9c-a0d845bcced7: moving from state init -> state preparing >Thread-15913971::DEBUG::2013-11-20 03:18:58,676::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913971::DEBUG::2013-11-20 03:18:58,676::task::495::TaskManager.Task::(_debug) Task 8912b6e5-209c-42d2-aa9c-a0d845bcced7: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913971::DEBUG::2013-11-20 03:18:58,677::task::495::TaskManager.Task::(_debug) Task 8912b6e5-209c-42d2-aa9c-a0d845bcced7: moving from state preparing -> state finished >Thread-15913971::DEBUG::2013-11-20 03:18:58,678::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913971::DEBUG::2013-11-20 03:18:58,678::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913971::DEBUG::2013-11-20 03:18:58,679::task::495::TaskManager.Task::(_debug) Task 8912b6e5-209c-42d2-aa9c-a0d845bcced7: ref 0 aborting False >Thread-15913971::INFO::2013-11-20 03:18:58,679::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913972::DEBUG::2013-11-20 03:18:58,684::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913972::INFO::2013-11-20 03:18:58,685::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913972::DEBUG::2013-11-20 03:18:58,685::task::495::TaskManager.Task::(_debug) Task 619067f8-c460-4e88-a513-8c0b53f48ec7: moving from state init -> state preparing >Thread-15913972::DEBUG::2013-11-20 03:18:58,686::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`106b6514-1b89-4fc3-952b-7b3f458b9e23`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913972::DEBUG::2013-11-20 03:18:58,687::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913972::DEBUG::2013-11-20 03:18:58,688::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913972::DEBUG::2013-11-20 03:18:58,689::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`106b6514-1b89-4fc3-952b-7b3f458b9e23`::Granted request >Thread-15913972::DEBUG::2013-11-20 03:18:58,690::task::495::TaskManager.Task::(_debug) Task 619067f8-c460-4e88-a513-8c0b53f48ec7: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913972::DEBUG::2013-11-20 03:18:58,690::task::495::TaskManager.Task::(_debug) Task 619067f8-c460-4e88-a513-8c0b53f48ec7: ref 1 aborting False >Thread-15913972::WARNING::2013-11-20 03:18:58,693::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913972::DEBUG::2013-11-20 03:18:58,694::task::495::TaskManager.Task::(_debug) Task 619067f8-c460-4e88-a513-8c0b53f48ec7: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913972::DEBUG::2013-11-20 03:18:58,695::task::495::TaskManager.Task::(_debug) Task 619067f8-c460-4e88-a513-8c0b53f48ec7: moving from state preparing -> state finished >Thread-15913972::DEBUG::2013-11-20 03:18:58,695::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913972::DEBUG::2013-11-20 03:18:58,696::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913972::DEBUG::2013-11-20 03:18:58,697::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913972::DEBUG::2013-11-20 03:18:58,698::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913972::DEBUG::2013-11-20 03:18:58,699::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913972::DEBUG::2013-11-20 03:18:58,700::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913972::DEBUG::2013-11-20 03:18:58,701::task::495::TaskManager.Task::(_debug) Task 619067f8-c460-4e88-a513-8c0b53f48ec7: ref 0 aborting False >Thread-15913972::INFO::2013-11-20 03:18:58,702::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:18:59,500::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:19:00,080::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0879906 s, 11.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:19:02,097::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:19:02,636::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0816534 s, 12.5 MB/s\n'; <rc> = 0 >Thread-15913974::INFO::2013-11-20 03:19:04,505::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913974::DEBUG::2013-11-20 03:19:04,505::task::495::TaskManager.Task::(_debug) Task 5fac1344-1c25-44de-a209-fec662f8e858: moving from state init -> state preparing >Thread-15913974::DEBUG::2013-11-20 03:19:04,506::task::495::TaskManager.Task::(_debug) Task 5fac1344-1c25-44de-a209-fec662f8e858: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00033712387085', 'lastCheck': 1384917535.599185, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00123691558838', 'lastCheck': 1384917542.297734, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00817799568176', 'lastCheck': 1384917534.6123581, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000500917434692', 'lastCheck': 1384917544.2906871, 'valid': True, 'code': 0}} >Thread-15913974::DEBUG::2013-11-20 03:19:04,507::task::495::TaskManager.Task::(_debug) Task 5fac1344-1c25-44de-a209-fec662f8e858: moving from state preparing -> state finished >Thread-15913974::DEBUG::2013-11-20 03:19:04,507::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913974::DEBUG::2013-11-20 03:19:04,508::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913974::DEBUG::2013-11-20 03:19:04,508::task::495::TaskManager.Task::(_debug) Task 5fac1344-1c25-44de-a209-fec662f8e858: ref 0 aborting False >Thread-15913974::INFO::2013-11-20 03:19:04,509::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00033712387085', 'lastCheck': 1384917535.599185, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00123691558838', 'lastCheck': 1384917542.297734, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00817799568176', 'lastCheck': 1384917534.6123581, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000500917434692', 'lastCheck': 1384917544.2906871, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:19:04,651::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:19:05,159::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0581533 s, 17.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:19:07,174::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:19:07,700::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0817327 s, 12.5 MB/s\n'; <rc> = 0 >Thread-15913977::DEBUG::2013-11-20 03:19:08,983::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913977::INFO::2013-11-20 03:19:08,985::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913977::DEBUG::2013-11-20 03:19:08,986::task::495::TaskManager.Task::(_debug) Task 3160eda0-c2d9-47cd-8301-e3d8645f47ff: moving from state init -> state preparing >Thread-15913977::DEBUG::2013-11-20 03:19:08,987::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913977::DEBUG::2013-11-20 03:19:08,988::task::495::TaskManager.Task::(_debug) Task 3160eda0-c2d9-47cd-8301-e3d8645f47ff: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913977::DEBUG::2013-11-20 03:19:08,989::task::495::TaskManager.Task::(_debug) Task 3160eda0-c2d9-47cd-8301-e3d8645f47ff: moving from state preparing -> state finished >Thread-15913977::DEBUG::2013-11-20 03:19:08,990::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913977::DEBUG::2013-11-20 03:19:08,991::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913977::DEBUG::2013-11-20 03:19:08,991::task::495::TaskManager.Task::(_debug) Task 3160eda0-c2d9-47cd-8301-e3d8645f47ff: ref 0 aborting False >Thread-15913977::INFO::2013-11-20 03:19:08,992::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913978::DEBUG::2013-11-20 03:19:08,998::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913978::INFO::2013-11-20 03:19:08,999::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913978::DEBUG::2013-11-20 03:19:08,999::task::495::TaskManager.Task::(_debug) Task 2bf2c554-d6f5-4a9f-9408-28157a334cc6: moving from state init -> state preparing >Thread-15913978::DEBUG::2013-11-20 03:19:09,001::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bb12d00e-c782-4e7f-a534-ded5822c92cc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913978::DEBUG::2013-11-20 03:19:09,001::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913978::DEBUG::2013-11-20 03:19:09,002::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913978::DEBUG::2013-11-20 03:19:09,003::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bb12d00e-c782-4e7f-a534-ded5822c92cc`::Granted request >Thread-15913978::DEBUG::2013-11-20 03:19:09,003::task::495::TaskManager.Task::(_debug) Task 2bf2c554-d6f5-4a9f-9408-28157a334cc6: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913978::DEBUG::2013-11-20 03:19:09,004::task::495::TaskManager.Task::(_debug) Task 2bf2c554-d6f5-4a9f-9408-28157a334cc6: ref 1 aborting False >Thread-15913978::WARNING::2013-11-20 03:19:09,006::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913978::DEBUG::2013-11-20 03:19:09,007::task::495::TaskManager.Task::(_debug) Task 2bf2c554-d6f5-4a9f-9408-28157a334cc6: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913978::DEBUG::2013-11-20 03:19:09,008::task::495::TaskManager.Task::(_debug) Task 2bf2c554-d6f5-4a9f-9408-28157a334cc6: moving from state preparing -> state finished >Thread-15913978::DEBUG::2013-11-20 03:19:09,008::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913978::DEBUG::2013-11-20 03:19:09,009::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913978::DEBUG::2013-11-20 03:19:09,009::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913978::DEBUG::2013-11-20 03:19:09,010::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913978::DEBUG::2013-11-20 03:19:09,011::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913978::DEBUG::2013-11-20 03:19:09,011::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913978::DEBUG::2013-11-20 03:19:09,012::task::495::TaskManager.Task::(_debug) Task 2bf2c554-d6f5-4a9f-9408-28157a334cc6: ref 0 aborting False >Thread-15913978::INFO::2013-11-20 03:19:09,012::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:19:09,714::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913546::INFO::2013-11-20 03:19:09,907::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:19:09,910::task::495::TaskManager.Task::(_debug) Task 05e8d057-4438-4b0c-b4e4-5706f7bedf28: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:19:09,914::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`7111abe8-1119-465f-b39f-fea89631c796`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:19:09,915::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:19:09,916::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:19:09,917::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`7111abe8-1119-465f-b39f-fea89631c796`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:19:09,918::task::495::TaskManager.Task::(_debug) Task 05e8d057-4438-4b0c-b4e4-5706f7bedf28: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:19:09,919::task::495::TaskManager.Task::(_debug) Task 05e8d057-4438-4b0c-b4e4-5706f7bedf28: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:19:09,919::task::495::TaskManager.Task::(_debug) Task 05e8d057-4438-4b0c-b4e4-5706f7bedf28: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:19:09,920::task::495::TaskManager.Task::(_debug) Task 05e8d057-4438-4b0c-b4e4-5706f7bedf28: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:19:09,921::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:19:09,921::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:19:09,922::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:19:09,923::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:19:09,923::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:19:09,924::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:19:09,924::task::495::TaskManager.Task::(_debug) Task 05e8d057-4438-4b0c-b4e4-5706f7bedf28: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:19:09,925::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:19:09,926::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:19:09,926::task::495::TaskManager.Task::(_debug) Task 169c92b2-da45-412c-b328-8ee5a474eea2: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:19:09,927::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`da1dcff9-b6bd-4765-a6e5-cdf9f828a1ad`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:19:09,928::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:19:09,929::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:19:09,929::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`da1dcff9-b6bd-4765-a6e5-cdf9f828a1ad`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:19:09,930::task::495::TaskManager.Task::(_debug) Task 169c92b2-da45-412c-b328-8ee5a474eea2: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:19:09,931::task::495::TaskManager.Task::(_debug) Task 169c92b2-da45-412c-b328-8ee5a474eea2: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:19:09,931::task::495::TaskManager.Task::(_debug) Task 169c92b2-da45-412c-b328-8ee5a474eea2: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:19:09,932::task::495::TaskManager.Task::(_debug) Task 169c92b2-da45-412c-b328-8ee5a474eea2: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:19:09,933::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:19:09,933::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:19:09,934::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:19:09,934::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:19:09,935::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:19:09,935::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:19:09,936::task::495::TaskManager.Task::(_debug) Task 169c92b2-da45-412c-b328-8ee5a474eea2: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:19:09,937::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:19:09,937::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:19:09,938::task::495::TaskManager.Task::(_debug) Task e9fc08af-a381-445e-8296-32380c77bb86: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:19:09,939::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`d48aeb09-88f0-4f76-800b-589301ac4dae`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:19:09,939::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:19:09,940::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:19:09,940::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`d48aeb09-88f0-4f76-800b-589301ac4dae`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:19:09,941::task::495::TaskManager.Task::(_debug) Task e9fc08af-a381-445e-8296-32380c77bb86: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:19:09,942::task::495::TaskManager.Task::(_debug) Task e9fc08af-a381-445e-8296-32380c77bb86: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:19:09,943::task::495::TaskManager.Task::(_debug) Task e9fc08af-a381-445e-8296-32380c77bb86: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:19:09,943::task::495::TaskManager.Task::(_debug) Task e9fc08af-a381-445e-8296-32380c77bb86: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:19:09,944::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:19:09,944::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:19:09,945::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:19:09,945::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:19:09,946::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:19:09,946::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:19:09,947::task::495::TaskManager.Task::(_debug) Task e9fc08af-a381-445e-8296-32380c77bb86: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:19:09,948::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:19:09,948::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:19:09,949::task::495::TaskManager.Task::(_debug) Task 51b612b9-0634-47f5-a037-3657c1d11467: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:19:09,950::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`f95a21b6-e2e7-4aad-a246-6727969f9e18`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:19:09,950::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:19:09,951::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:19:09,952::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`f95a21b6-e2e7-4aad-a246-6727969f9e18`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:19:09,952::task::495::TaskManager.Task::(_debug) Task 51b612b9-0634-47f5-a037-3657c1d11467: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:19:09,953::task::495::TaskManager.Task::(_debug) Task 51b612b9-0634-47f5-a037-3657c1d11467: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:19:09,953::task::495::TaskManager.Task::(_debug) Task 51b612b9-0634-47f5-a037-3657c1d11467: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:19:09,954::task::495::TaskManager.Task::(_debug) Task 51b612b9-0634-47f5-a037-3657c1d11467: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:19:09,954::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:19:09,955::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:19:09,955::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:19:09,956::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:19:09,957::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:19:09,957::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:19:09,958::task::495::TaskManager.Task::(_debug) Task 51b612b9-0634-47f5-a037-3657c1d11467: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:19:09,958::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:19:10,230::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0851448 s, 12.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:19:12,243::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:19:12,780::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0818085 s, 12.5 MB/s\n'; <rc> = 0 >Thread-42840::DEBUG::2013-11-20 03:19:14,468::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:19:14,796::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-42840::DEBUG::2013-11-20 03:19:15,240::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:19:15,359::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0807916 s, 12.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:19:17,374::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:19:17,941::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0994658 s, 10.3 MB/s\n'; <rc> = 0 >Thread-15913982::DEBUG::2013-11-20 03:19:19,217::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913982::INFO::2013-11-20 03:19:19,220::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913982::DEBUG::2013-11-20 03:19:19,221::task::495::TaskManager.Task::(_debug) Task ef92d118-fbe5-47e4-a531-729f96f2a200: moving from state init -> state preparing >Thread-15913982::DEBUG::2013-11-20 03:19:19,222::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913982::DEBUG::2013-11-20 03:19:19,222::task::495::TaskManager.Task::(_debug) Task ef92d118-fbe5-47e4-a531-729f96f2a200: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913982::DEBUG::2013-11-20 03:19:19,223::task::495::TaskManager.Task::(_debug) Task ef92d118-fbe5-47e4-a531-729f96f2a200: moving from state preparing -> state finished >Thread-15913982::DEBUG::2013-11-20 03:19:19,223::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913982::DEBUG::2013-11-20 03:19:19,224::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913982::DEBUG::2013-11-20 03:19:19,225::task::495::TaskManager.Task::(_debug) Task ef92d118-fbe5-47e4-a531-729f96f2a200: ref 0 aborting False >Thread-15913982::INFO::2013-11-20 03:19:19,225::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913983::DEBUG::2013-11-20 03:19:19,235::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913983::INFO::2013-11-20 03:19:19,236::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913983::DEBUG::2013-11-20 03:19:19,236::task::495::TaskManager.Task::(_debug) Task 75d04285-1595-4fb1-8ca9-aae17dc03ec5: moving from state init -> state preparing >Thread-15913983::DEBUG::2013-11-20 03:19:19,238::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3157e01e-2a63-4308-9b50-1115a7418a17`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913983::DEBUG::2013-11-20 03:19:19,238::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913983::DEBUG::2013-11-20 03:19:19,239::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913983::DEBUG::2013-11-20 03:19:19,240::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3157e01e-2a63-4308-9b50-1115a7418a17`::Granted request >Thread-15913983::DEBUG::2013-11-20 03:19:19,241::task::495::TaskManager.Task::(_debug) Task 75d04285-1595-4fb1-8ca9-aae17dc03ec5: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913983::DEBUG::2013-11-20 03:19:19,242::task::495::TaskManager.Task::(_debug) Task 75d04285-1595-4fb1-8ca9-aae17dc03ec5: ref 1 aborting False >Thread-15913983::WARNING::2013-11-20 03:19:19,244::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913983::DEBUG::2013-11-20 03:19:19,245::task::495::TaskManager.Task::(_debug) Task 75d04285-1595-4fb1-8ca9-aae17dc03ec5: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913983::DEBUG::2013-11-20 03:19:19,246::task::495::TaskManager.Task::(_debug) Task 75d04285-1595-4fb1-8ca9-aae17dc03ec5: moving from state preparing -> state finished >Thread-15913983::DEBUG::2013-11-20 03:19:19,246::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913983::DEBUG::2013-11-20 03:19:19,247::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913983::DEBUG::2013-11-20 03:19:19,248::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913983::DEBUG::2013-11-20 03:19:19,249::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913983::DEBUG::2013-11-20 03:19:19,249::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913983::DEBUG::2013-11-20 03:19:19,250::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913983::DEBUG::2013-11-20 03:19:19,251::task::495::TaskManager.Task::(_debug) Task 75d04285-1595-4fb1-8ca9-aae17dc03ec5: ref 0 aborting False >Thread-15913983::INFO::2013-11-20 03:19:19,252::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:19:19,956::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913639::INFO::2013-11-20 03:19:20,150::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:19:20,153::task::495::TaskManager.Task::(_debug) Task 1be2afb1-9daf-41cd-ad31-5655101e75a5: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:19:20,160::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0cdc2ce0-815f-4061-9903-f2fdf2e6ee9f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913984::INFO::2013-11-20 03:19:20,160::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913639::DEBUG::2013-11-20 03:19:20,161::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913984::DEBUG::2013-11-20 03:19:20,162::task::495::TaskManager.Task::(_debug) Task 5460c3b3-307d-4650-bb78-e3a3a770978e: moving from state init -> state preparing >Thread-15913984::DEBUG::2013-11-20 03:19:20,164::task::495::TaskManager.Task::(_debug) Task 5460c3b3-307d-4650-bb78-e3a3a770978e: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000264883041382', 'lastCheck': 1384917556.033457, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0114569664001', 'lastCheck': 1384917552.4826281, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0103697776794', 'lastCheck': 1384917555.1196179, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000441074371338', 'lastCheck': 1384917555.2432389, 'valid': True, 'code': 0}} >Thread-15913639::DEBUG::2013-11-20 03:19:20,164::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913984::DEBUG::2013-11-20 03:19:20,166::task::495::TaskManager.Task::(_debug) Task 5460c3b3-307d-4650-bb78-e3a3a770978e: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:19:20,166::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0cdc2ce0-815f-4061-9903-f2fdf2e6ee9f`::Granted request >Thread-15913984::DEBUG::2013-11-20 03:19:20,167::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913984::DEBUG::2013-11-20 03:19:20,169::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913984::DEBUG::2013-11-20 03:19:20,169::task::495::TaskManager.Task::(_debug) Task 5460c3b3-307d-4650-bb78-e3a3a770978e: ref 0 aborting False >Thread-15913984::INFO::2013-11-20 03:19:20,170::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000264883041382', 'lastCheck': 1384917556.033457, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0114569664001', 'lastCheck': 1384917552.4826281, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0103697776794', 'lastCheck': 1384917555.1196179, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000441074371338', 'lastCheck': 1384917555.2432389, 'valid': True, 'code': 0}} >Thread-15913639::DEBUG::2013-11-20 03:19:20,180::task::495::TaskManager.Task::(_debug) Task 1be2afb1-9daf-41cd-ad31-5655101e75a5: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:19:20,181::task::495::TaskManager.Task::(_debug) Task 1be2afb1-9daf-41cd-ad31-5655101e75a5: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:19:20,182::task::495::TaskManager.Task::(_debug) Task 1be2afb1-9daf-41cd-ad31-5655101e75a5: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:19:20,183::task::495::TaskManager.Task::(_debug) Task 1be2afb1-9daf-41cd-ad31-5655101e75a5: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:19:20,184::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:19:20,184::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:19:20,185::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:19:20,186::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:19:20,187::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:19:20,187::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:19:20,188::task::495::TaskManager.Task::(_debug) Task 1be2afb1-9daf-41cd-ad31-5655101e75a5: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:19:20,189::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:19:20,437::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0660423 s, 15.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:19:22,450::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:19:22,990::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0803598 s, 12.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:19:25,007::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:19:25,525::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0688275 s, 14.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:19:27,541::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:19:28,095::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0808873 s, 12.7 MB/s\n'; <rc> = 0 >Thread-15913988::DEBUG::2013-11-20 03:19:29,422::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913988::INFO::2013-11-20 03:19:29,424::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913988::DEBUG::2013-11-20 03:19:29,425::task::495::TaskManager.Task::(_debug) Task 57c2c696-cb51-4f94-b9e0-6671fa2a5a4a: moving from state init -> state preparing >Thread-15913988::DEBUG::2013-11-20 03:19:29,426::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913988::DEBUG::2013-11-20 03:19:29,426::task::495::TaskManager.Task::(_debug) Task 57c2c696-cb51-4f94-b9e0-6671fa2a5a4a: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913988::DEBUG::2013-11-20 03:19:29,427::task::495::TaskManager.Task::(_debug) Task 57c2c696-cb51-4f94-b9e0-6671fa2a5a4a: moving from state preparing -> state finished >Thread-15913988::DEBUG::2013-11-20 03:19:29,427::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913988::DEBUG::2013-11-20 03:19:29,428::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913988::DEBUG::2013-11-20 03:19:29,428::task::495::TaskManager.Task::(_debug) Task 57c2c696-cb51-4f94-b9e0-6671fa2a5a4a: ref 0 aborting False >Thread-15913988::INFO::2013-11-20 03:19:29,429::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913989::DEBUG::2013-11-20 03:19:29,433::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913989::INFO::2013-11-20 03:19:29,434::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913989::DEBUG::2013-11-20 03:19:29,435::task::495::TaskManager.Task::(_debug) Task 064b9dff-eab5-442d-9176-3c7140b28aef: moving from state init -> state preparing >Thread-15913989::DEBUG::2013-11-20 03:19:29,436::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`83ed822b-4e1f-4d42-93ba-598cd04d4a24`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913989::DEBUG::2013-11-20 03:19:29,437::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913989::DEBUG::2013-11-20 03:19:29,437::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913989::DEBUG::2013-11-20 03:19:29,438::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`83ed822b-4e1f-4d42-93ba-598cd04d4a24`::Granted request >Thread-15913989::DEBUG::2013-11-20 03:19:29,439::task::495::TaskManager.Task::(_debug) Task 064b9dff-eab5-442d-9176-3c7140b28aef: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913989::DEBUG::2013-11-20 03:19:29,440::task::495::TaskManager.Task::(_debug) Task 064b9dff-eab5-442d-9176-3c7140b28aef: ref 1 aborting False >Thread-15913989::WARNING::2013-11-20 03:19:29,442::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913989::DEBUG::2013-11-20 03:19:29,443::task::495::TaskManager.Task::(_debug) Task 064b9dff-eab5-442d-9176-3c7140b28aef: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913989::DEBUG::2013-11-20 03:19:29,443::task::495::TaskManager.Task::(_debug) Task 064b9dff-eab5-442d-9176-3c7140b28aef: moving from state preparing -> state finished >Thread-15913989::DEBUG::2013-11-20 03:19:29,444::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913989::DEBUG::2013-11-20 03:19:29,445::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913989::DEBUG::2013-11-20 03:19:29,446::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913989::DEBUG::2013-11-20 03:19:29,447::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913989::DEBUG::2013-11-20 03:19:29,447::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913989::DEBUG::2013-11-20 03:19:29,448::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913989::DEBUG::2013-11-20 03:19:29,448::task::495::TaskManager.Task::(_debug) Task 064b9dff-eab5-442d-9176-3c7140b28aef: ref 0 aborting False >Thread-15913989::INFO::2013-11-20 03:19:29,449::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:19:30,105::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:19:30,665::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0854322 s, 12.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:19:32,681::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:19:33,156::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0542529 s, 18.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:19:35,171::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:19:35,719::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0780855 s, 13.1 MB/s\n'; <rc> = 0 >Thread-15913992::INFO::2013-11-20 03:19:37,676::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15913992::DEBUG::2013-11-20 03:19:37,677::task::495::TaskManager.Task::(_debug) Task 8444cf40-1757-4ed9-a820-3f02a6b9b078: moving from state init -> state preparing >Thread-15913992::DEBUG::2013-11-20 03:19:37,678::task::495::TaskManager.Task::(_debug) Task 8444cf40-1757-4ed9-a820-3f02a6b9b078: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00294208526611', 'lastCheck': 1384917576.3917291, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00105714797974', 'lastCheck': 1384917572.879241, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00302386283875', 'lastCheck': 1384917575.3908601, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00115394592285', 'lastCheck': 1384917575.3830309, 'valid': True, 'code': 0}} >Thread-15913992::DEBUG::2013-11-20 03:19:37,679::task::495::TaskManager.Task::(_debug) Task 8444cf40-1757-4ed9-a820-3f02a6b9b078: moving from state preparing -> state finished >Thread-15913992::DEBUG::2013-11-20 03:19:37,679::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913992::DEBUG::2013-11-20 03:19:37,680::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913992::DEBUG::2013-11-20 03:19:37,681::task::495::TaskManager.Task::(_debug) Task 8444cf40-1757-4ed9-a820-3f02a6b9b078: ref 0 aborting False >Thread-15913992::INFO::2013-11-20 03:19:37,681::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00294208526611', 'lastCheck': 1384917576.3917291, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00105714797974', 'lastCheck': 1384917572.879241, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00302386283875', 'lastCheck': 1384917575.3908601, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00115394592285', 'lastCheck': 1384917575.3830309, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:19:37,734::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913695::INFO::2013-11-20 03:19:37,992::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:19:37,994::task::495::TaskManager.Task::(_debug) Task 8718f080-ba78-46be-a756-8ace0b136472: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:19:37,996::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`feed16f0-2b30-46c0-b362-f7a854514276`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:19:37,997::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:19:37,998::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:19:37,998::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`feed16f0-2b30-46c0-b362-f7a854514276`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:19:38,000::task::495::TaskManager.Task::(_debug) Task 8718f080-ba78-46be-a756-8ace0b136472: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:19:38,000::task::495::TaskManager.Task::(_debug) Task 8718f080-ba78-46be-a756-8ace0b136472: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:19:38,002::task::495::TaskManager.Task::(_debug) Task 8718f080-ba78-46be-a756-8ace0b136472: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:19:38,003::task::495::TaskManager.Task::(_debug) Task 8718f080-ba78-46be-a756-8ace0b136472: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:19:38,004::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:19:38,005::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:19:38,006::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:19:38,006::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:19:38,007::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:19:38,007::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:19:38,008::task::495::TaskManager.Task::(_debug) Task 8718f080-ba78-46be-a756-8ace0b136472: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:19:38,008::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:19:38,307::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0846897 s, 12.1 MB/s\n'; <rc> = 0 >Thread-15913994::DEBUG::2013-11-20 03:19:40,148::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913994::INFO::2013-11-20 03:19:40,149::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913994::DEBUG::2013-11-20 03:19:40,150::task::495::TaskManager.Task::(_debug) Task 4df9ac09-aa24-4d1b-b628-4f2d169b5c74: moving from state init -> state preparing >Thread-15913994::DEBUG::2013-11-20 03:19:40,150::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913994::DEBUG::2013-11-20 03:19:40,151::task::495::TaskManager.Task::(_debug) Task 4df9ac09-aa24-4d1b-b628-4f2d169b5c74: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913994::DEBUG::2013-11-20 03:19:40,151::task::495::TaskManager.Task::(_debug) Task 4df9ac09-aa24-4d1b-b628-4f2d169b5c74: moving from state preparing -> state finished >Thread-15913994::DEBUG::2013-11-20 03:19:40,151::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913994::DEBUG::2013-11-20 03:19:40,152::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913994::DEBUG::2013-11-20 03:19:40,152::task::495::TaskManager.Task::(_debug) Task 4df9ac09-aa24-4d1b-b628-4f2d169b5c74: ref 0 aborting False >Thread-15913994::INFO::2013-11-20 03:19:40,152::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913995::DEBUG::2013-11-20 03:19:40,155::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913995::INFO::2013-11-20 03:19:40,156::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913995::DEBUG::2013-11-20 03:19:40,156::task::495::TaskManager.Task::(_debug) Task a7379d56-325c-443b-ba1f-7ae4ced6f98f: moving from state init -> state preparing >Thread-15913995::DEBUG::2013-11-20 03:19:40,157::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3e1b9906-4f8a-43c4-89fd-8443fb3c4b2e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15913995::DEBUG::2013-11-20 03:19:40,157::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15913995::DEBUG::2013-11-20 03:19:40,158::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15913995::DEBUG::2013-11-20 03:19:40,158::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3e1b9906-4f8a-43c4-89fd-8443fb3c4b2e`::Granted request >Thread-15913995::DEBUG::2013-11-20 03:19:40,159::task::495::TaskManager.Task::(_debug) Task a7379d56-325c-443b-ba1f-7ae4ced6f98f: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15913995::DEBUG::2013-11-20 03:19:40,159::task::495::TaskManager.Task::(_debug) Task a7379d56-325c-443b-ba1f-7ae4ced6f98f: ref 1 aborting False >Thread-15913995::WARNING::2013-11-20 03:19:40,160::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15913995::DEBUG::2013-11-20 03:19:40,161::task::495::TaskManager.Task::(_debug) Task a7379d56-325c-443b-ba1f-7ae4ced6f98f: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913995::DEBUG::2013-11-20 03:19:40,161::task::495::TaskManager.Task::(_debug) Task a7379d56-325c-443b-ba1f-7ae4ced6f98f: moving from state preparing -> state finished >Thread-15913995::DEBUG::2013-11-20 03:19:40,162::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15913995::DEBUG::2013-11-20 03:19:40,162::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913995::DEBUG::2013-11-20 03:19:40,163::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15913995::DEBUG::2013-11-20 03:19:40,163::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15913995::DEBUG::2013-11-20 03:19:40,164::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15913995::DEBUG::2013-11-20 03:19:40,164::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15913995::DEBUG::2013-11-20 03:19:40,164::task::495::TaskManager.Task::(_debug) Task a7379d56-325c-443b-ba1f-7ae4ced6f98f: ref 0 aborting False >Thread-15913995::INFO::2013-11-20 03:19:40,165::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:19:40,324::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913658::INFO::2013-11-20 03:19:40,539::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:19:40,545::task::495::TaskManager.Task::(_debug) Task b20fced8-ddf5-40a7-94d1-6a564d52b205: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:19:40,548::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`195a2405-25bd-49fd-b874-84f727cda47a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:19:40,549::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:19:40,550::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:19:40,551::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`195a2405-25bd-49fd-b874-84f727cda47a`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:19:40,552::task::495::TaskManager.Task::(_debug) Task b20fced8-ddf5-40a7-94d1-6a564d52b205: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:19:40,552::task::495::TaskManager.Task::(_debug) Task b20fced8-ddf5-40a7-94d1-6a564d52b205: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:19:40,553::task::495::TaskManager.Task::(_debug) Task b20fced8-ddf5-40a7-94d1-6a564d52b205: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:19:40,554::task::495::TaskManager.Task::(_debug) Task b20fced8-ddf5-40a7-94d1-6a564d52b205: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:19:40,555::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:19:40,555::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:19:40,556::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:19:40,557::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:19:40,557::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:19:40,558::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:19:40,558::task::495::TaskManager.Task::(_debug) Task b20fced8-ddf5-40a7-94d1-6a564d52b205: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:19:40,559::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:19:40,868::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0870199 s, 11.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:19:42,882::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:19:43,335::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0687333 s, 14.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:19:45,346::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 03:19:45,561::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:19:45,595::task::495::TaskManager.Task::(_debug) Task 30e4e431-18eb-4d24-811c-a2bf66db40e9: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:19:45,604::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c77db48b-441f-4d3b-98f3-61cd31195ad6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:19:45,606::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:19:45,608::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:19:45,609::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c77db48b-441f-4d3b-98f3-61cd31195ad6`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:19:45,611::task::495::TaskManager.Task::(_debug) Task 30e4e431-18eb-4d24-811c-a2bf66db40e9: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:19:45,612::task::495::TaskManager.Task::(_debug) Task 30e4e431-18eb-4d24-811c-a2bf66db40e9: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:19:45,613::task::495::TaskManager.Task::(_debug) Task 30e4e431-18eb-4d24-811c-a2bf66db40e9: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:19:45,614::task::495::TaskManager.Task::(_debug) Task 30e4e431-18eb-4d24-811c-a2bf66db40e9: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:19:45,615::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:19:45,616::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:19:45,617::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:19:45,618::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:19:45,619::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:19:45,620::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:19:45,621::task::495::TaskManager.Task::(_debug) Task 30e4e431-18eb-4d24-811c-a2bf66db40e9: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:19:45,622::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:19:45,895::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0846956 s, 12.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:19:47,912::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:19:48,464::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0860339 s, 11.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:19:50,476::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913999::DEBUG::2013-11-20 03:19:50,730::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15913999::INFO::2013-11-20 03:19:50,732::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15913999::DEBUG::2013-11-20 03:19:50,733::task::495::TaskManager.Task::(_debug) Task ed3bf178-0569-4265-b32a-46a027c4fb79: moving from state init -> state preparing >Thread-15913999::DEBUG::2013-11-20 03:19:50,734::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15913999::DEBUG::2013-11-20 03:19:50,734::task::495::TaskManager.Task::(_debug) Task ed3bf178-0569-4265-b32a-46a027c4fb79: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15913999::DEBUG::2013-11-20 03:19:50,735::task::495::TaskManager.Task::(_debug) Task ed3bf178-0569-4265-b32a-46a027c4fb79: moving from state preparing -> state finished >Thread-15913999::DEBUG::2013-11-20 03:19:50,736::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15913999::DEBUG::2013-11-20 03:19:50,736::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913999::DEBUG::2013-11-20 03:19:50,737::task::495::TaskManager.Task::(_debug) Task ed3bf178-0569-4265-b32a-46a027c4fb79: ref 0 aborting False >Thread-15913999::INFO::2013-11-20 03:19:50,738::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914000::DEBUG::2013-11-20 03:19:50,743::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914000::INFO::2013-11-20 03:19:50,744::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914000::DEBUG::2013-11-20 03:19:50,745::task::495::TaskManager.Task::(_debug) Task 1b9db192-69b5-4cc7-b8b0-e19353604fae: moving from state init -> state preparing >Thread-15914000::DEBUG::2013-11-20 03:19:50,746::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9eb2d4a5-200f-45f9-a881-852030b29b68`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914000::DEBUG::2013-11-20 03:19:50,747::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914000::DEBUG::2013-11-20 03:19:50,748::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914000::DEBUG::2013-11-20 03:19:50,748::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9eb2d4a5-200f-45f9-a881-852030b29b68`::Granted request >Thread-15914000::DEBUG::2013-11-20 03:19:50,750::task::495::TaskManager.Task::(_debug) Task 1b9db192-69b5-4cc7-b8b0-e19353604fae: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914000::DEBUG::2013-11-20 03:19:50,750::task::495::TaskManager.Task::(_debug) Task 1b9db192-69b5-4cc7-b8b0-e19353604fae: ref 1 aborting False >Thread-15914000::WARNING::2013-11-20 03:19:50,753::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914000::DEBUG::2013-11-20 03:19:50,754::task::495::TaskManager.Task::(_debug) Task 1b9db192-69b5-4cc7-b8b0-e19353604fae: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914000::DEBUG::2013-11-20 03:19:50,755::task::495::TaskManager.Task::(_debug) Task 1b9db192-69b5-4cc7-b8b0-e19353604fae: moving from state preparing -> state finished >Thread-15914000::DEBUG::2013-11-20 03:19:50,755::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914000::DEBUG::2013-11-20 03:19:50,756::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914000::DEBUG::2013-11-20 03:19:50,757::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914000::DEBUG::2013-11-20 03:19:50,758::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914000::DEBUG::2013-11-20 03:19:50,758::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914000::DEBUG::2013-11-20 03:19:50,759::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914000::DEBUG::2013-11-20 03:19:50,759::task::495::TaskManager.Task::(_debug) Task 1b9db192-69b5-4cc7-b8b0-e19353604fae: ref 0 aborting False >Thread-15914000::INFO::2013-11-20 03:19:50,760::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:19:51,012::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0769805 s, 13.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:19:53,029::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:19:53,594::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0843319 s, 12.1 MB/s\n'; <rc> = 0 >Thread-15914002::INFO::2013-11-20 03:19:53,815::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914002::DEBUG::2013-11-20 03:19:53,816::task::495::TaskManager.Task::(_debug) Task 5069d3fc-21a9-4cf8-b5bf-d7dda9af0d3c: moving from state init -> state preparing >Thread-15914002::DEBUG::2013-11-20 03:19:53,817::task::495::TaskManager.Task::(_debug) Task 5069d3fc-21a9-4cf8-b5bf-d7dda9af0d3c: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000272035598755', 'lastCheck': 1384917586.5507591, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00218605995178', 'lastCheck': 1384917593.2640851, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00219297409058', 'lastCheck': 1384917585.6078019, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0113520622253', 'lastCheck': 1384917585.586195, 'valid': True, 'code': 0}} >Thread-15914002::DEBUG::2013-11-20 03:19:53,818::task::495::TaskManager.Task::(_debug) Task 5069d3fc-21a9-4cf8-b5bf-d7dda9af0d3c: moving from state preparing -> state finished >Thread-15914002::DEBUG::2013-11-20 03:19:53,818::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914002::DEBUG::2013-11-20 03:19:53,819::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914002::DEBUG::2013-11-20 03:19:53,819::task::495::TaskManager.Task::(_debug) Task 5069d3fc-21a9-4cf8-b5bf-d7dda9af0d3c: ref 0 aborting False >Thread-15914002::INFO::2013-11-20 03:19:53,820::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000272035598755', 'lastCheck': 1384917586.5507591, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00218605995178', 'lastCheck': 1384917593.2640851, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00219297409058', 'lastCheck': 1384917585.6078019, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0113520622253', 'lastCheck': 1384917585.586195, 'valid': True, 'code': 0}} >Thread-15913518::INFO::2013-11-20 03:19:54,237::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:19:54,238::task::495::TaskManager.Task::(_debug) Task 1e795189-534a-469b-9c94-7fd0360fbdf4: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:19:54,239::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`dcad802c-9353-4478-8758-f22240c34ad0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:19:54,240::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:19:54,240::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:19:54,241::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`dcad802c-9353-4478-8758-f22240c34ad0`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:19:54,241::task::495::TaskManager.Task::(_debug) Task 1e795189-534a-469b-9c94-7fd0360fbdf4: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:19:54,242::task::495::TaskManager.Task::(_debug) Task 1e795189-534a-469b-9c94-7fd0360fbdf4: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:19:54,242::task::495::TaskManager.Task::(_debug) Task 1e795189-534a-469b-9c94-7fd0360fbdf4: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:19:54,243::task::495::TaskManager.Task::(_debug) Task 1e795189-534a-469b-9c94-7fd0360fbdf4: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:19:54,243::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:19:54,244::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:19:54,244::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:19:54,245::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:19:54,245::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:19:54,245::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:19:54,246::task::495::TaskManager.Task::(_debug) Task 1e795189-534a-469b-9c94-7fd0360fbdf4: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:19:54,246::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:19:54,247::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:19:54,247::task::495::TaskManager.Task::(_debug) Task e96fb13e-457b-49cb-abe7-a249773757ef: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:19:54,248::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a996dafc-4b14-4716-a894-ec9a4fdbb7e5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:19:54,248::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:19:54,249::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:19:54,249::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a996dafc-4b14-4716-a894-ec9a4fdbb7e5`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:19:54,250::task::495::TaskManager.Task::(_debug) Task e96fb13e-457b-49cb-abe7-a249773757ef: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:19:54,250::task::495::TaskManager.Task::(_debug) Task e96fb13e-457b-49cb-abe7-a249773757ef: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:19:54,251::task::495::TaskManager.Task::(_debug) Task e96fb13e-457b-49cb-abe7-a249773757ef: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:19:54,251::task::495::TaskManager.Task::(_debug) Task e96fb13e-457b-49cb-abe7-a249773757ef: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:19:54,252::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:19:54,252::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:19:54,253::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:19:54,253::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:19:54,253::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:19:54,254::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:19:54,254::task::495::TaskManager.Task::(_debug) Task e96fb13e-457b-49cb-abe7-a249773757ef: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:19:54,255::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:19:55,604::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:19:56,117::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0804437 s, 12.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:19:58,133::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:19:58,659::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0759298 s, 13.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:20:00,675::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:20:01,151::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0591772 s, 17.3 MB/s\n'; <rc> = 0 >Thread-15914006::DEBUG::2013-11-20 03:20:01,239::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914006::INFO::2013-11-20 03:20:01,240::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914006::DEBUG::2013-11-20 03:20:01,240::task::495::TaskManager.Task::(_debug) Task 0a2679e9-0dd0-466a-b56a-34b1159fb3aa: moving from state init -> state preparing >Thread-15914006::DEBUG::2013-11-20 03:20:01,241::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914006::DEBUG::2013-11-20 03:20:01,242::task::495::TaskManager.Task::(_debug) Task 0a2679e9-0dd0-466a-b56a-34b1159fb3aa: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914006::DEBUG::2013-11-20 03:20:01,242::task::495::TaskManager.Task::(_debug) Task 0a2679e9-0dd0-466a-b56a-34b1159fb3aa: moving from state preparing -> state finished >Thread-15914006::DEBUG::2013-11-20 03:20:01,243::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914006::DEBUG::2013-11-20 03:20:01,243::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914006::DEBUG::2013-11-20 03:20:01,244::task::495::TaskManager.Task::(_debug) Task 0a2679e9-0dd0-466a-b56a-34b1159fb3aa: ref 0 aborting False >Thread-15914006::INFO::2013-11-20 03:20:01,245::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914007::DEBUG::2013-11-20 03:20:01,252::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914007::INFO::2013-11-20 03:20:01,253::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914007::DEBUG::2013-11-20 03:20:01,254::task::495::TaskManager.Task::(_debug) Task 9367ec8c-119c-4eb3-ae55-0f7aa5f05588: moving from state init -> state preparing >Thread-15914007::DEBUG::2013-11-20 03:20:01,255::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d8ddcb42-da01-4670-8d89-7c3d3c850f27`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914007::DEBUG::2013-11-20 03:20:01,255::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914007::DEBUG::2013-11-20 03:20:01,256::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914007::DEBUG::2013-11-20 03:20:01,256::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d8ddcb42-da01-4670-8d89-7c3d3c850f27`::Granted request >Thread-15914007::DEBUG::2013-11-20 03:20:01,257::task::495::TaskManager.Task::(_debug) Task 9367ec8c-119c-4eb3-ae55-0f7aa5f05588: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914007::DEBUG::2013-11-20 03:20:01,258::task::495::TaskManager.Task::(_debug) Task 9367ec8c-119c-4eb3-ae55-0f7aa5f05588: ref 1 aborting False >Thread-15914007::WARNING::2013-11-20 03:20:01,259::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914007::DEBUG::2013-11-20 03:20:01,260::task::495::TaskManager.Task::(_debug) Task 9367ec8c-119c-4eb3-ae55-0f7aa5f05588: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914007::DEBUG::2013-11-20 03:20:01,261::task::495::TaskManager.Task::(_debug) Task 9367ec8c-119c-4eb3-ae55-0f7aa5f05588: moving from state preparing -> state finished >Thread-15914007::DEBUG::2013-11-20 03:20:01,261::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914007::DEBUG::2013-11-20 03:20:01,262::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914007::DEBUG::2013-11-20 03:20:01,263::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914007::DEBUG::2013-11-20 03:20:01,263::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914007::DEBUG::2013-11-20 03:20:01,264::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914007::DEBUG::2013-11-20 03:20:01,264::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914007::DEBUG::2013-11-20 03:20:01,265::task::495::TaskManager.Task::(_debug) Task 9367ec8c-119c-4eb3-ae55-0f7aa5f05588: ref 0 aborting False >Thread-15914007::INFO::2013-11-20 03:20:01,266::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:20:03,165::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:20:03,706::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100827 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:20:05,722::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:20:06,272::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.119678 s, 8.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:20:08,287::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:20:08,827::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0842232 s, 12.2 MB/s\n'; <rc> = 0 >Thread-15914010::INFO::2013-11-20 03:20:09,463::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914010::DEBUG::2013-11-20 03:20:09,463::task::495::TaskManager.Task::(_debug) Task 58272913-b99a-4d45-b8ef-c803d4390fee: moving from state init -> state preparing >Thread-15914010::DEBUG::2013-11-20 03:20:09,464::task::495::TaskManager.Task::(_debug) Task 58272913-b99a-4d45-b8ef-c803d4390fee: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000327110290527', 'lastCheck': 1384917606.922416, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00133490562439', 'lastCheck': 1384917603.3825209, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00126600265503', 'lastCheck': 1384917605.927319, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00577187538147', 'lastCheck': 1384917605.924561, 'valid': True, 'code': 0}} >Thread-15914010::DEBUG::2013-11-20 03:20:09,465::task::495::TaskManager.Task::(_debug) Task 58272913-b99a-4d45-b8ef-c803d4390fee: moving from state preparing -> state finished >Thread-15914010::DEBUG::2013-11-20 03:20:09,465::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914010::DEBUG::2013-11-20 03:20:09,465::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914010::DEBUG::2013-11-20 03:20:09,466::task::495::TaskManager.Task::(_debug) Task 58272913-b99a-4d45-b8ef-c803d4390fee: ref 0 aborting False >Thread-15914010::INFO::2013-11-20 03:20:09,466::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000327110290527', 'lastCheck': 1384917606.922416, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00133490562439', 'lastCheck': 1384917603.3825209, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00126600265503', 'lastCheck': 1384917605.927319, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00577187538147', 'lastCheck': 1384917605.924561, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:20:10,844::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913546::INFO::2013-11-20 03:20:11,040::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:20:11,042::task::495::TaskManager.Task::(_debug) Task 89f0a3fc-5a6f-4381-b87e-d34651919cd1: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:20:11,046::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`34b168f4-396c-4b7e-99f8-7f3aa11ef09b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:20:11,047::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:20:11,049::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:20:11,050::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`34b168f4-396c-4b7e-99f8-7f3aa11ef09b`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:20:11,051::task::495::TaskManager.Task::(_debug) Task 89f0a3fc-5a6f-4381-b87e-d34651919cd1: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:20:11,052::task::495::TaskManager.Task::(_debug) Task 89f0a3fc-5a6f-4381-b87e-d34651919cd1: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:20:11,053::task::495::TaskManager.Task::(_debug) Task 89f0a3fc-5a6f-4381-b87e-d34651919cd1: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:20:11,053::task::495::TaskManager.Task::(_debug) Task 89f0a3fc-5a6f-4381-b87e-d34651919cd1: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:20:11,054::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:20:11,055::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:20:11,055::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:20:11,056::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:20:11,057::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:20:11,057::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:20:11,058::task::495::TaskManager.Task::(_debug) Task 89f0a3fc-5a6f-4381-b87e-d34651919cd1: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:20:11,058::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:20:11,059::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:20:11,060::task::495::TaskManager.Task::(_debug) Task 7051634f-196e-458e-a293-b3659bf5cd03: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:20:11,061::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`1889dfc6-824c-4da3-93c9-95d2dbc15cbc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:20:11,062::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:20:11,062::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:20:11,063::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`1889dfc6-824c-4da3-93c9-95d2dbc15cbc`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:20:11,064::task::495::TaskManager.Task::(_debug) Task 7051634f-196e-458e-a293-b3659bf5cd03: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:20:11,065::task::495::TaskManager.Task::(_debug) Task 7051634f-196e-458e-a293-b3659bf5cd03: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:20:11,065::task::495::TaskManager.Task::(_debug) Task 7051634f-196e-458e-a293-b3659bf5cd03: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:20:11,066::task::495::TaskManager.Task::(_debug) Task 7051634f-196e-458e-a293-b3659bf5cd03: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:20:11,066::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:20:11,067::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:20:11,068::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:20:11,068::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:20:11,069::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:20:11,069::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:20:11,070::task::495::TaskManager.Task::(_debug) Task 7051634f-196e-458e-a293-b3659bf5cd03: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:20:11,070::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:20:11,071::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:20:11,072::task::495::TaskManager.Task::(_debug) Task 866b2280-451e-47f0-be03-6cdeabb56d50: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:20:11,073::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`40a87e90-4836-4016-bbd9-ea80e01bbba9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:20:11,074::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:20:11,074::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:20:11,075::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`40a87e90-4836-4016-bbd9-ea80e01bbba9`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:20:11,076::task::495::TaskManager.Task::(_debug) Task 866b2280-451e-47f0-be03-6cdeabb56d50: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:20:11,076::task::495::TaskManager.Task::(_debug) Task 866b2280-451e-47f0-be03-6cdeabb56d50: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:20:11,077::task::495::TaskManager.Task::(_debug) Task 866b2280-451e-47f0-be03-6cdeabb56d50: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:20:11,078::task::495::TaskManager.Task::(_debug) Task 866b2280-451e-47f0-be03-6cdeabb56d50: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:20:11,078::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:20:11,079::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:20:11,079::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:20:11,080::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:20:11,081::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:20:11,081::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:20:11,082::task::495::TaskManager.Task::(_debug) Task 866b2280-451e-47f0-be03-6cdeabb56d50: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:20:11,082::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:20:11,083::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:20:11,084::task::495::TaskManager.Task::(_debug) Task 9949b5a8-b99b-4256-8e77-800e022d8d7b: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:20:11,085::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`130522a1-fb78-4c73-a7e7-da20ad0ccc05`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:20:11,085::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:20:11,086::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:20:11,087::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`130522a1-fb78-4c73-a7e7-da20ad0ccc05`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:20:11,087::task::495::TaskManager.Task::(_debug) Task 9949b5a8-b99b-4256-8e77-800e022d8d7b: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:20:11,088::task::495::TaskManager.Task::(_debug) Task 9949b5a8-b99b-4256-8e77-800e022d8d7b: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:20:11,089::task::495::TaskManager.Task::(_debug) Task 9949b5a8-b99b-4256-8e77-800e022d8d7b: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:20:11,089::task::495::TaskManager.Task::(_debug) Task 9949b5a8-b99b-4256-8e77-800e022d8d7b: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:20:11,090::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:20:11,090::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:20:11,091::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:20:11,092::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:20:11,093::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:20:11,093::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:20:11,094::task::495::TaskManager.Task::(_debug) Task 9949b5a8-b99b-4256-8e77-800e022d8d7b: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:20:11,094::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:20:11,290::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0486868 s, 21.0 MB/s\n'; <rc> = 0 >Thread-15914012::DEBUG::2013-11-20 03:20:11,480::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914012::INFO::2013-11-20 03:20:11,481::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914012::DEBUG::2013-11-20 03:20:11,495::task::495::TaskManager.Task::(_debug) Task f54c7d0c-d121-469d-bf1e-6d4ace275e48: moving from state init -> state preparing >Thread-15914012::DEBUG::2013-11-20 03:20:11,496::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914012::DEBUG::2013-11-20 03:20:11,496::task::495::TaskManager.Task::(_debug) Task f54c7d0c-d121-469d-bf1e-6d4ace275e48: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914012::DEBUG::2013-11-20 03:20:11,497::task::495::TaskManager.Task::(_debug) Task f54c7d0c-d121-469d-bf1e-6d4ace275e48: moving from state preparing -> state finished >Thread-15914012::DEBUG::2013-11-20 03:20:11,497::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914012::DEBUG::2013-11-20 03:20:11,498::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914012::DEBUG::2013-11-20 03:20:11,498::task::495::TaskManager.Task::(_debug) Task f54c7d0c-d121-469d-bf1e-6d4ace275e48: ref 0 aborting False >Thread-15914012::INFO::2013-11-20 03:20:11,499::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914013::DEBUG::2013-11-20 03:20:11,504::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914013::INFO::2013-11-20 03:20:11,505::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914013::DEBUG::2013-11-20 03:20:11,506::task::495::TaskManager.Task::(_debug) Task b7a40350-33f9-4509-8cad-f84bc9f579b2: moving from state init -> state preparing >Thread-15914013::DEBUG::2013-11-20 03:20:11,516::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9e065e7e-7c50-4e04-b373-ed3daa7bd061`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914013::DEBUG::2013-11-20 03:20:11,517::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914013::DEBUG::2013-11-20 03:20:11,517::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914013::DEBUG::2013-11-20 03:20:11,518::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9e065e7e-7c50-4e04-b373-ed3daa7bd061`::Granted request >Thread-15914013::DEBUG::2013-11-20 03:20:11,519::task::495::TaskManager.Task::(_debug) Task b7a40350-33f9-4509-8cad-f84bc9f579b2: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914013::DEBUG::2013-11-20 03:20:11,519::task::495::TaskManager.Task::(_debug) Task b7a40350-33f9-4509-8cad-f84bc9f579b2: ref 1 aborting False >Thread-15914013::WARNING::2013-11-20 03:20:11,521::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914013::DEBUG::2013-11-20 03:20:11,522::task::495::TaskManager.Task::(_debug) Task b7a40350-33f9-4509-8cad-f84bc9f579b2: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914013::DEBUG::2013-11-20 03:20:11,523::task::495::TaskManager.Task::(_debug) Task b7a40350-33f9-4509-8cad-f84bc9f579b2: moving from state preparing -> state finished >Thread-15914013::DEBUG::2013-11-20 03:20:11,523::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914013::DEBUG::2013-11-20 03:20:11,524::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914013::DEBUG::2013-11-20 03:20:11,524::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914013::DEBUG::2013-11-20 03:20:11,525::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914013::DEBUG::2013-11-20 03:20:11,525::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914013::DEBUG::2013-11-20 03:20:11,526::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914013::DEBUG::2013-11-20 03:20:11,526::task::495::TaskManager.Task::(_debug) Task b7a40350-33f9-4509-8cad-f84bc9f579b2: ref 0 aborting False >Thread-15914013::INFO::2013-11-20 03:20:11,527::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:20:13,307::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:20:13,830::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0497964 s, 20.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:20:15,839::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:20:16,369::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0788496 s, 13.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:20:18,380::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:20:18,888::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0751605 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:20:20,903::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913639::INFO::2013-11-20 03:20:21,136::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:20:21,141::task::495::TaskManager.Task::(_debug) Task 1408587d-e27a-4abf-b2eb-eeb2efcccde5: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:20:21,148::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`22d8ad39-906e-4327-a973-0045b1c6af7f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:20:21,158::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:20:21,175::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:20:21,176::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`22d8ad39-906e-4327-a973-0045b1c6af7f`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:20:21,177::task::495::TaskManager.Task::(_debug) Task 1408587d-e27a-4abf-b2eb-eeb2efcccde5: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:20:21,178::task::495::TaskManager.Task::(_debug) Task 1408587d-e27a-4abf-b2eb-eeb2efcccde5: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:20:21,178::task::495::TaskManager.Task::(_debug) Task 1408587d-e27a-4abf-b2eb-eeb2efcccde5: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:20:21,179::task::495::TaskManager.Task::(_debug) Task 1408587d-e27a-4abf-b2eb-eeb2efcccde5: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:20:21,179::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:20:21,180::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:20:21,180::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:20:21,181::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:20:21,181::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:20:21,181::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:20:21,182::task::495::TaskManager.Task::(_debug) Task 1408587d-e27a-4abf-b2eb-eeb2efcccde5: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:20:21,182::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:20:21,459::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0831684 s, 12.3 MB/s\n'; <rc> = 0 >Thread-15914018::DEBUG::2013-11-20 03:20:21,961::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914018::INFO::2013-11-20 03:20:21,963::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914018::DEBUG::2013-11-20 03:20:21,973::task::495::TaskManager.Task::(_debug) Task 1207e0a7-41cc-4580-86ec-aed1c10cd2e4: moving from state init -> state preparing >Thread-15914018::DEBUG::2013-11-20 03:20:21,974::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914018::DEBUG::2013-11-20 03:20:21,974::task::495::TaskManager.Task::(_debug) Task 1207e0a7-41cc-4580-86ec-aed1c10cd2e4: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914018::DEBUG::2013-11-20 03:20:21,975::task::495::TaskManager.Task::(_debug) Task 1207e0a7-41cc-4580-86ec-aed1c10cd2e4: moving from state preparing -> state finished >Thread-15914018::DEBUG::2013-11-20 03:20:21,975::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914018::DEBUG::2013-11-20 03:20:21,976::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914018::DEBUG::2013-11-20 03:20:21,976::task::495::TaskManager.Task::(_debug) Task 1207e0a7-41cc-4580-86ec-aed1c10cd2e4: ref 0 aborting False >Thread-15914018::INFO::2013-11-20 03:20:21,977::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914019::DEBUG::2013-11-20 03:20:21,981::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914019::INFO::2013-11-20 03:20:21,982::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914019::DEBUG::2013-11-20 03:20:21,983::task::495::TaskManager.Task::(_debug) Task 21392bca-a196-4f2c-bc36-eb9d28193d0c: moving from state init -> state preparing >Thread-15914019::DEBUG::2013-11-20 03:20:21,984::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d97b70a1-cbae-492b-9af4-9895b138e9df`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914019::DEBUG::2013-11-20 03:20:21,984::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914019::DEBUG::2013-11-20 03:20:21,985::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914019::DEBUG::2013-11-20 03:20:21,985::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d97b70a1-cbae-492b-9af4-9895b138e9df`::Granted request >Thread-15914019::DEBUG::2013-11-20 03:20:21,986::task::495::TaskManager.Task::(_debug) Task 21392bca-a196-4f2c-bc36-eb9d28193d0c: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914019::DEBUG::2013-11-20 03:20:21,987::task::495::TaskManager.Task::(_debug) Task 21392bca-a196-4f2c-bc36-eb9d28193d0c: ref 1 aborting False >Thread-15914019::WARNING::2013-11-20 03:20:21,989::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914019::DEBUG::2013-11-20 03:20:21,989::task::495::TaskManager.Task::(_debug) Task 21392bca-a196-4f2c-bc36-eb9d28193d0c: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914019::DEBUG::2013-11-20 03:20:21,990::task::495::TaskManager.Task::(_debug) Task 21392bca-a196-4f2c-bc36-eb9d28193d0c: moving from state preparing -> state finished >Thread-15914019::DEBUG::2013-11-20 03:20:21,990::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914019::DEBUG::2013-11-20 03:20:21,991::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914019::DEBUG::2013-11-20 03:20:21,991::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914019::DEBUG::2013-11-20 03:20:21,992::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914019::DEBUG::2013-11-20 03:20:21,993::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914019::DEBUG::2013-11-20 03:20:21,993::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914019::DEBUG::2013-11-20 03:20:21,994::task::495::TaskManager.Task::(_debug) Task 21392bca-a196-4f2c-bc36-eb9d28193d0c: ref 0 aborting False >Thread-15914019::INFO::2013-11-20 03:20:21,994::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:20:23,473::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:20:23,994::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0824173 s, 12.4 MB/s\n'; <rc> = 0 >Thread-15914020::INFO::2013-11-20 03:20:24,883::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914020::DEBUG::2013-11-20 03:20:24,884::task::495::TaskManager.Task::(_debug) Task d44d6209-35b5-470a-8b88-4703d026c1c5: moving from state init -> state preparing >Thread-15914020::DEBUG::2013-11-20 03:20:24,885::task::495::TaskManager.Task::(_debug) Task d44d6209-35b5-470a-8b88-4703d026c1c5: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000421047210693', 'lastCheck': 1384917617.0443921, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00104188919067', 'lastCheck': 1384917623.6470971, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00281882286072', 'lastCheck': 1384917616.0503371, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000764846801758', 'lastCheck': 1384917616.0410509, 'valid': True, 'code': 0}} >Thread-15914020::DEBUG::2013-11-20 03:20:24,886::task::495::TaskManager.Task::(_debug) Task d44d6209-35b5-470a-8b88-4703d026c1c5: moving from state preparing -> state finished >Thread-15914020::DEBUG::2013-11-20 03:20:24,886::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914020::DEBUG::2013-11-20 03:20:24,887::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914020::DEBUG::2013-11-20 03:20:24,887::task::495::TaskManager.Task::(_debug) Task d44d6209-35b5-470a-8b88-4703d026c1c5: ref 0 aborting False >Thread-15914020::INFO::2013-11-20 03:20:24,888::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000421047210693', 'lastCheck': 1384917617.0443921, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00104188919067', 'lastCheck': 1384917623.6470971, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00281882286072', 'lastCheck': 1384917616.0503371, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000764846801758', 'lastCheck': 1384917616.0410509, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:20:26,010::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:20:26,537::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0675244 s, 15.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:20:28,551::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:20:29,119::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.075188 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:20:31,134::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:20:31,693::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.102537 s, 10.0 MB/s\n'; <rc> = 0 >Thread-15914024::DEBUG::2013-11-20 03:20:32,280::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914024::INFO::2013-11-20 03:20:32,282::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914024::DEBUG::2013-11-20 03:20:32,283::task::495::TaskManager.Task::(_debug) Task 534a55bd-4fd2-4833-ace8-9e28427c180e: moving from state init -> state preparing >Thread-15914024::DEBUG::2013-11-20 03:20:32,285::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914024::DEBUG::2013-11-20 03:20:32,286::task::495::TaskManager.Task::(_debug) Task 534a55bd-4fd2-4833-ace8-9e28427c180e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914024::DEBUG::2013-11-20 03:20:32,287::task::495::TaskManager.Task::(_debug) Task 534a55bd-4fd2-4833-ace8-9e28427c180e: moving from state preparing -> state finished >Thread-15914024::DEBUG::2013-11-20 03:20:32,288::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914024::DEBUG::2013-11-20 03:20:32,288::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914024::DEBUG::2013-11-20 03:20:32,289::task::495::TaskManager.Task::(_debug) Task 534a55bd-4fd2-4833-ace8-9e28427c180e: ref 0 aborting False >Thread-15914024::INFO::2013-11-20 03:20:32,289::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914025::DEBUG::2013-11-20 03:20:32,294::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914025::INFO::2013-11-20 03:20:32,295::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914025::DEBUG::2013-11-20 03:20:32,296::task::495::TaskManager.Task::(_debug) Task 89280b1a-afc7-45d0-a726-b625bfeaa3cf: moving from state init -> state preparing >Thread-15914025::DEBUG::2013-11-20 03:20:32,297::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3149a381-fd1d-4f1a-b128-39700ae73241`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914025::DEBUG::2013-11-20 03:20:32,298::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914025::DEBUG::2013-11-20 03:20:32,299::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914025::DEBUG::2013-11-20 03:20:32,300::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3149a381-fd1d-4f1a-b128-39700ae73241`::Granted request >Thread-15914025::DEBUG::2013-11-20 03:20:32,301::task::495::TaskManager.Task::(_debug) Task 89280b1a-afc7-45d0-a726-b625bfeaa3cf: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914025::DEBUG::2013-11-20 03:20:32,302::task::495::TaskManager.Task::(_debug) Task 89280b1a-afc7-45d0-a726-b625bfeaa3cf: ref 1 aborting False >Thread-15914025::WARNING::2013-11-20 03:20:32,304::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914025::DEBUG::2013-11-20 03:20:32,304::task::495::TaskManager.Task::(_debug) Task 89280b1a-afc7-45d0-a726-b625bfeaa3cf: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914025::DEBUG::2013-11-20 03:20:32,305::task::495::TaskManager.Task::(_debug) Task 89280b1a-afc7-45d0-a726-b625bfeaa3cf: moving from state preparing -> state finished >Thread-15914025::DEBUG::2013-11-20 03:20:32,306::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914025::DEBUG::2013-11-20 03:20:32,306::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914025::DEBUG::2013-11-20 03:20:32,307::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914025::DEBUG::2013-11-20 03:20:32,308::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914025::DEBUG::2013-11-20 03:20:32,308::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914025::DEBUG::2013-11-20 03:20:32,309::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914025::DEBUG::2013-11-20 03:20:32,309::task::495::TaskManager.Task::(_debug) Task 89280b1a-afc7-45d0-a726-b625bfeaa3cf: ref 0 aborting False >Thread-15914025::INFO::2013-11-20 03:20:32,310::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:20:33,709::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:20:34,142::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0450451 s, 22.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:20:36,153::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:20:36,749::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.109806 s, 9.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:20:38,762::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913695::INFO::2013-11-20 03:20:38,977::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:20:38,980::task::495::TaskManager.Task::(_debug) Task 961399e5-a6f4-4183-b95e-6da64303ed56: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:20:38,982::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d1177cdb-e682-495a-aa28-2af89840378f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:20:38,983::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:20:38,984::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:20:38,985::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d1177cdb-e682-495a-aa28-2af89840378f`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:20:38,986::task::495::TaskManager.Task::(_debug) Task 961399e5-a6f4-4183-b95e-6da64303ed56: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:20:38,987::task::495::TaskManager.Task::(_debug) Task 961399e5-a6f4-4183-b95e-6da64303ed56: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:20:38,987::task::495::TaskManager.Task::(_debug) Task 961399e5-a6f4-4183-b95e-6da64303ed56: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:20:38,988::task::495::TaskManager.Task::(_debug) Task 961399e5-a6f4-4183-b95e-6da64303ed56: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:20:38,989::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:20:38,990::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:20:38,990::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:20:38,991::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:20:38,992::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:20:38,992::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:20:38,993::task::495::TaskManager.Task::(_debug) Task 961399e5-a6f4-4183-b95e-6da64303ed56: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:20:38,994::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:20:39,294::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0933445 s, 11.0 MB/s\n'; <rc> = 0 >Thread-15914028::INFO::2013-11-20 03:20:40,490::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914028::DEBUG::2013-11-20 03:20:40,491::task::495::TaskManager.Task::(_debug) Task e26e8354-38c6-459f-801b-f9d2ca05410b: moving from state init -> state preparing >Thread-15914028::DEBUG::2013-11-20 03:20:40,492::task::495::TaskManager.Task::(_debug) Task e26e8354-38c6-459f-801b-f9d2ca05410b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00049901008606', 'lastCheck': 1384917637.4107251, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.010794878006', 'lastCheck': 1384917633.93749, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0103280544281', 'lastCheck': 1384917636.439688, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0166292190552', 'lastCheck': 1384917636.426199, 'valid': True, 'code': 0}} >Thread-15914028::DEBUG::2013-11-20 03:20:40,493::task::495::TaskManager.Task::(_debug) Task e26e8354-38c6-459f-801b-f9d2ca05410b: moving from state preparing -> state finished >Thread-15914028::DEBUG::2013-11-20 03:20:40,493::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914028::DEBUG::2013-11-20 03:20:40,494::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914028::DEBUG::2013-11-20 03:20:40,495::task::495::TaskManager.Task::(_debug) Task e26e8354-38c6-459f-801b-f9d2ca05410b: ref 0 aborting False >Thread-15914028::INFO::2013-11-20 03:20:40,495::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00049901008606', 'lastCheck': 1384917637.4107251, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.010794878006', 'lastCheck': 1384917633.93749, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0103280544281', 'lastCheck': 1384917636.439688, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0166292190552', 'lastCheck': 1384917636.426199, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:20:41,310::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913658::INFO::2013-11-20 03:20:41,522::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:20:41,525::task::495::TaskManager.Task::(_debug) Task e60ad848-516b-4d67-8ffc-e96c3a7c7c67: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:20:41,528::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`8f1d8626-e6f3-47f0-badd-0cce2b77ca39`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:20:41,530::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:20:41,531::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:20:41,532::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`8f1d8626-e6f3-47f0-badd-0cce2b77ca39`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:20:41,533::task::495::TaskManager.Task::(_debug) Task e60ad848-516b-4d67-8ffc-e96c3a7c7c67: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:20:41,533::task::495::TaskManager.Task::(_debug) Task e60ad848-516b-4d67-8ffc-e96c3a7c7c67: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:20:41,534::task::495::TaskManager.Task::(_debug) Task e60ad848-516b-4d67-8ffc-e96c3a7c7c67: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:20:41,534::task::495::TaskManager.Task::(_debug) Task e60ad848-516b-4d67-8ffc-e96c3a7c7c67: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:20:41,534::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:20:41,535::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:20:41,535::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:20:41,536::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:20:41,536::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:20:41,537::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:20:41,537::task::495::TaskManager.Task::(_debug) Task e60ad848-516b-4d67-8ffc-e96c3a7c7c67: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:20:41,538::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:20:41,832::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0726305 s, 14.1 MB/s\n'; <rc> = 0 >Thread-15914030::DEBUG::2013-11-20 03:20:42,528::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914030::INFO::2013-11-20 03:20:42,529::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914030::DEBUG::2013-11-20 03:20:42,530::task::495::TaskManager.Task::(_debug) Task 5f55291c-4d94-414f-a7b9-01861bdab898: moving from state init -> state preparing >Thread-15914030::DEBUG::2013-11-20 03:20:42,531::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914030::DEBUG::2013-11-20 03:20:42,531::task::495::TaskManager.Task::(_debug) Task 5f55291c-4d94-414f-a7b9-01861bdab898: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914030::DEBUG::2013-11-20 03:20:42,532::task::495::TaskManager.Task::(_debug) Task 5f55291c-4d94-414f-a7b9-01861bdab898: moving from state preparing -> state finished >Thread-15914030::DEBUG::2013-11-20 03:20:42,533::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914030::DEBUG::2013-11-20 03:20:42,533::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914030::DEBUG::2013-11-20 03:20:42,534::task::495::TaskManager.Task::(_debug) Task 5f55291c-4d94-414f-a7b9-01861bdab898: ref 0 aborting False >Thread-15914030::INFO::2013-11-20 03:20:42,535::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914031::DEBUG::2013-11-20 03:20:42,540::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914031::INFO::2013-11-20 03:20:42,541::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914031::DEBUG::2013-11-20 03:20:42,542::task::495::TaskManager.Task::(_debug) Task 243b1d0c-256d-44fb-a60b-8dc50d176a5b: moving from state init -> state preparing >Thread-15914031::DEBUG::2013-11-20 03:20:42,543::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6ddf80b0-6c16-4469-b847-2136dfc5d785`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914031::DEBUG::2013-11-20 03:20:42,544::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914031::DEBUG::2013-11-20 03:20:42,545::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914031::DEBUG::2013-11-20 03:20:42,545::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6ddf80b0-6c16-4469-b847-2136dfc5d785`::Granted request >Thread-15914031::DEBUG::2013-11-20 03:20:42,547::task::495::TaskManager.Task::(_debug) Task 243b1d0c-256d-44fb-a60b-8dc50d176a5b: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914031::DEBUG::2013-11-20 03:20:42,547::task::495::TaskManager.Task::(_debug) Task 243b1d0c-256d-44fb-a60b-8dc50d176a5b: ref 1 aborting False >Thread-15914031::WARNING::2013-11-20 03:20:42,549::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914031::DEBUG::2013-11-20 03:20:42,550::task::495::TaskManager.Task::(_debug) Task 243b1d0c-256d-44fb-a60b-8dc50d176a5b: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914031::DEBUG::2013-11-20 03:20:42,551::task::495::TaskManager.Task::(_debug) Task 243b1d0c-256d-44fb-a60b-8dc50d176a5b: moving from state preparing -> state finished >Thread-15914031::DEBUG::2013-11-20 03:20:42,552::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914031::DEBUG::2013-11-20 03:20:42,552::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914031::DEBUG::2013-11-20 03:20:42,553::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914031::DEBUG::2013-11-20 03:20:42,554::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914031::DEBUG::2013-11-20 03:20:42,554::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914031::DEBUG::2013-11-20 03:20:42,555::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914031::DEBUG::2013-11-20 03:20:42,555::task::495::TaskManager.Task::(_debug) Task 243b1d0c-256d-44fb-a60b-8dc50d176a5b: ref 0 aborting False >Thread-15914031::INFO::2013-11-20 03:20:42,556::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:20:43,847::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:20:44,399::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0719003 s, 14.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:20:46,413::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 03:20:46,649::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:20:46,656::task::495::TaskManager.Task::(_debug) Task 89e05ea0-2526-4913-b65d-93d120698324: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:20:46,665::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`80cde197-80b3-4034-b9a7-4d1a86ae480d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:20:46,668::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:20:46,672::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:20:46,674::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`80cde197-80b3-4034-b9a7-4d1a86ae480d`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:20:46,676::task::495::TaskManager.Task::(_debug) Task 89e05ea0-2526-4913-b65d-93d120698324: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:20:46,677::task::495::TaskManager.Task::(_debug) Task 89e05ea0-2526-4913-b65d-93d120698324: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:20:46,678::task::495::TaskManager.Task::(_debug) Task 89e05ea0-2526-4913-b65d-93d120698324: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:20:46,679::task::495::TaskManager.Task::(_debug) Task 89e05ea0-2526-4913-b65d-93d120698324: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:20:46,680::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:20:46,681::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:20:46,682::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:20:46,683::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:20:46,684::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:20:46,685::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:20:46,686::task::495::TaskManager.Task::(_debug) Task 89e05ea0-2526-4913-b65d-93d120698324: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:20:46,687::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:20:46,969::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0844453 s, 12.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:20:48,984::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:20:49,531::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0822429 s, 12.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:20:51,547::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:20:52,098::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0767716 s, 13.3 MB/s\n'; <rc> = 0 >Thread-15914035::DEBUG::2013-11-20 03:20:52,749::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914035::INFO::2013-11-20 03:20:52,751::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914035::DEBUG::2013-11-20 03:20:52,751::task::495::TaskManager.Task::(_debug) Task ea5e4fa8-a88d-4fa8-ae6b-da8503ead8cf: moving from state init -> state preparing >Thread-15914035::DEBUG::2013-11-20 03:20:52,752::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914035::DEBUG::2013-11-20 03:20:52,753::task::495::TaskManager.Task::(_debug) Task ea5e4fa8-a88d-4fa8-ae6b-da8503ead8cf: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914035::DEBUG::2013-11-20 03:20:52,754::task::495::TaskManager.Task::(_debug) Task ea5e4fa8-a88d-4fa8-ae6b-da8503ead8cf: moving from state preparing -> state finished >Thread-15914035::DEBUG::2013-11-20 03:20:52,754::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914035::DEBUG::2013-11-20 03:20:52,755::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914035::DEBUG::2013-11-20 03:20:52,756::task::495::TaskManager.Task::(_debug) Task ea5e4fa8-a88d-4fa8-ae6b-da8503ead8cf: ref 0 aborting False >Thread-15914035::INFO::2013-11-20 03:20:52,756::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914036::DEBUG::2013-11-20 03:20:52,761::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914036::INFO::2013-11-20 03:20:52,762::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914036::DEBUG::2013-11-20 03:20:52,763::task::495::TaskManager.Task::(_debug) Task 303df4f6-3f3a-4037-b7ac-af03a8c7b52f: moving from state init -> state preparing >Thread-15914036::DEBUG::2013-11-20 03:20:52,764::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6e479bb4-be1a-42c6-9e81-4c4f60b58d90`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914036::DEBUG::2013-11-20 03:20:52,765::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914036::DEBUG::2013-11-20 03:20:52,766::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914036::DEBUG::2013-11-20 03:20:52,766::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6e479bb4-be1a-42c6-9e81-4c4f60b58d90`::Granted request >Thread-15914036::DEBUG::2013-11-20 03:20:52,767::task::495::TaskManager.Task::(_debug) Task 303df4f6-3f3a-4037-b7ac-af03a8c7b52f: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914036::DEBUG::2013-11-20 03:20:52,768::task::495::TaskManager.Task::(_debug) Task 303df4f6-3f3a-4037-b7ac-af03a8c7b52f: ref 1 aborting False >Thread-15914036::WARNING::2013-11-20 03:20:52,771::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914036::DEBUG::2013-11-20 03:20:52,772::task::495::TaskManager.Task::(_debug) Task 303df4f6-3f3a-4037-b7ac-af03a8c7b52f: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914036::DEBUG::2013-11-20 03:20:52,773::task::495::TaskManager.Task::(_debug) Task 303df4f6-3f3a-4037-b7ac-af03a8c7b52f: moving from state preparing -> state finished >Thread-15914036::DEBUG::2013-11-20 03:20:52,774::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914036::DEBUG::2013-11-20 03:20:52,774::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914036::DEBUG::2013-11-20 03:20:52,775::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914036::DEBUG::2013-11-20 03:20:52,776::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914036::DEBUG::2013-11-20 03:20:52,777::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914036::DEBUG::2013-11-20 03:20:52,777::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914036::DEBUG::2013-11-20 03:20:52,778::task::495::TaskManager.Task::(_debug) Task 303df4f6-3f3a-4037-b7ac-af03a8c7b52f: ref 0 aborting False >Thread-15914036::INFO::2013-11-20 03:20:52,779::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:20:54,112::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:20:54,651::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0615763 s, 16.6 MB/s\n'; <rc> = 0 >Thread-15913518::INFO::2013-11-20 03:20:55,287::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:20:55,288::task::495::TaskManager.Task::(_debug) Task 0b2c3537-1994-428c-be0f-8e8242efc19b: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:20:55,290::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`4ef2b657-a98d-4b5e-88eb-4df61d6c4549`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:20:55,290::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:20:55,291::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:20:55,292::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`4ef2b657-a98d-4b5e-88eb-4df61d6c4549`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:20:55,293::task::495::TaskManager.Task::(_debug) Task 0b2c3537-1994-428c-be0f-8e8242efc19b: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:20:55,294::task::495::TaskManager.Task::(_debug) Task 0b2c3537-1994-428c-be0f-8e8242efc19b: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:20:55,295::task::495::TaskManager.Task::(_debug) Task 0b2c3537-1994-428c-be0f-8e8242efc19b: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:20:55,295::task::495::TaskManager.Task::(_debug) Task 0b2c3537-1994-428c-be0f-8e8242efc19b: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:20:55,296::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:20:55,297::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:20:55,297::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:20:55,298::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:20:55,298::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:20:55,299::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:20:55,300::task::495::TaskManager.Task::(_debug) Task 0b2c3537-1994-428c-be0f-8e8242efc19b: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:20:55,300::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:20:55,301::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:20:55,302::task::495::TaskManager.Task::(_debug) Task c0a826bb-e0f0-4fc5-860c-ccae9afbbb2b: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:20:55,303::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ff5ff32e-2a18-4c6b-8d4e-e3088e8b7796`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:20:55,304::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:20:55,304::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:20:55,305::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ff5ff32e-2a18-4c6b-8d4e-e3088e8b7796`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:20:55,306::task::495::TaskManager.Task::(_debug) Task c0a826bb-e0f0-4fc5-860c-ccae9afbbb2b: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:20:55,307::task::495::TaskManager.Task::(_debug) Task c0a826bb-e0f0-4fc5-860c-ccae9afbbb2b: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:20:55,307::task::495::TaskManager.Task::(_debug) Task c0a826bb-e0f0-4fc5-860c-ccae9afbbb2b: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:20:55,308::task::495::TaskManager.Task::(_debug) Task c0a826bb-e0f0-4fc5-860c-ccae9afbbb2b: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:20:55,309::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:20:55,309::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:20:55,310::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:20:55,311::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:20:55,312::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:20:55,312::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:20:55,313::task::495::TaskManager.Task::(_debug) Task c0a826bb-e0f0-4fc5-860c-ccae9afbbb2b: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:20:55,314::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15914038::INFO::2013-11-20 03:20:55,950::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914038::DEBUG::2013-11-20 03:20:55,951::task::495::TaskManager.Task::(_debug) Task 0657b7cf-2320-4cb0-8e17-352623486176: moving from state init -> state preparing >Thread-15914038::DEBUG::2013-11-20 03:20:55,952::task::495::TaskManager.Task::(_debug) Task 0657b7cf-2320-4cb0-8e17-352623486176: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000513076782227', 'lastCheck': 1384917647.6363089, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146794319153', 'lastCheck': 1384917654.3721011, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00249600410461', 'lastCheck': 1384917646.6737411, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00871205329895', 'lastCheck': 1384917646.664993, 'valid': True, 'code': 0}} >Thread-15914038::DEBUG::2013-11-20 03:20:55,953::task::495::TaskManager.Task::(_debug) Task 0657b7cf-2320-4cb0-8e17-352623486176: moving from state preparing -> state finished >Thread-15914038::DEBUG::2013-11-20 03:20:55,954::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914038::DEBUG::2013-11-20 03:20:55,954::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914038::DEBUG::2013-11-20 03:20:55,955::task::495::TaskManager.Task::(_debug) Task 0657b7cf-2320-4cb0-8e17-352623486176: ref 0 aborting False >Thread-15914038::INFO::2013-11-20 03:20:55,955::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000513076782227', 'lastCheck': 1384917647.6363089, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00146794319153', 'lastCheck': 1384917654.3721011, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00249600410461', 'lastCheck': 1384917646.6737411, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00871205329895', 'lastCheck': 1384917646.664993, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:20:56,661::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:20:57,226::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0790887 s, 12.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:20:59,241::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:20:59,831::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0827599 s, 12.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:21:01,845::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:21:02,376::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0746215 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15914042::DEBUG::2013-11-20 03:21:03,017::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914042::INFO::2013-11-20 03:21:03,019::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914042::DEBUG::2013-11-20 03:21:03,020::task::495::TaskManager.Task::(_debug) Task ba255771-2527-4ece-8d08-cc23c5faa6e6: moving from state init -> state preparing >Thread-15914042::DEBUG::2013-11-20 03:21:03,021::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914042::DEBUG::2013-11-20 03:21:03,022::task::495::TaskManager.Task::(_debug) Task ba255771-2527-4ece-8d08-cc23c5faa6e6: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914042::DEBUG::2013-11-20 03:21:03,022::task::495::TaskManager.Task::(_debug) Task ba255771-2527-4ece-8d08-cc23c5faa6e6: moving from state preparing -> state finished >Thread-15914042::DEBUG::2013-11-20 03:21:03,023::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914042::DEBUG::2013-11-20 03:21:03,024::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914042::DEBUG::2013-11-20 03:21:03,024::task::495::TaskManager.Task::(_debug) Task ba255771-2527-4ece-8d08-cc23c5faa6e6: ref 0 aborting False >Thread-15914042::INFO::2013-11-20 03:21:03,025::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914043::DEBUG::2013-11-20 03:21:03,029::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914043::INFO::2013-11-20 03:21:03,030::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914043::DEBUG::2013-11-20 03:21:03,031::task::495::TaskManager.Task::(_debug) Task 5f3e2d8e-a154-4cb1-8282-258336de673f: moving from state init -> state preparing >Thread-15914043::DEBUG::2013-11-20 03:21:03,032::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ea6c56f1-d533-4125-a295-838752d6b8da`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914043::DEBUG::2013-11-20 03:21:03,033::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914043::DEBUG::2013-11-20 03:21:03,033::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914043::DEBUG::2013-11-20 03:21:03,034::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ea6c56f1-d533-4125-a295-838752d6b8da`::Granted request >Thread-15914043::DEBUG::2013-11-20 03:21:03,035::task::495::TaskManager.Task::(_debug) Task 5f3e2d8e-a154-4cb1-8282-258336de673f: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914043::DEBUG::2013-11-20 03:21:03,035::task::495::TaskManager.Task::(_debug) Task 5f3e2d8e-a154-4cb1-8282-258336de673f: ref 1 aborting False >Thread-15914043::WARNING::2013-11-20 03:21:03,037::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914043::DEBUG::2013-11-20 03:21:03,038::task::495::TaskManager.Task::(_debug) Task 5f3e2d8e-a154-4cb1-8282-258336de673f: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914043::DEBUG::2013-11-20 03:21:03,039::task::495::TaskManager.Task::(_debug) Task 5f3e2d8e-a154-4cb1-8282-258336de673f: moving from state preparing -> state finished >Thread-15914043::DEBUG::2013-11-20 03:21:03,040::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914043::DEBUG::2013-11-20 03:21:03,040::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914043::DEBUG::2013-11-20 03:21:03,041::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914043::DEBUG::2013-11-20 03:21:03,042::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914043::DEBUG::2013-11-20 03:21:03,042::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914043::DEBUG::2013-11-20 03:21:03,043::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914043::DEBUG::2013-11-20 03:21:03,043::task::495::TaskManager.Task::(_debug) Task 5f3e2d8e-a154-4cb1-8282-258336de673f: ref 0 aborting False >Thread-15914043::INFO::2013-11-20 03:21:03,044::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:21:04,391::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913659::DEBUG::2013-11-20 03:21:04,816::guestIF::97::vm.Vm::(_handleMessage) vmId=`0dc7e997-e884-4493-b701-790fb37b3d80`::u'AOS2Batch@brd-klee.dk' >Dummy-15912651::DEBUG::2013-11-20 03:21:04,987::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0853172 s, 12.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:21:07,006::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:21:07,564::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.077339 s, 13.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:21:09,579::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:21:10,093::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0711724 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15914046::INFO::2013-11-20 03:21:11,612::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914046::DEBUG::2013-11-20 03:21:11,613::task::495::TaskManager.Task::(_debug) Task 45e14a2e-29e1-4f82-8c9b-7ef0cb05adec: moving from state init -> state preparing >Thread-15914046::DEBUG::2013-11-20 03:21:11,614::task::495::TaskManager.Task::(_debug) Task 45e14a2e-29e1-4f82-8c9b-7ef0cb05adec: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000303030014038', 'lastCheck': 1384917668.2381959, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154900550842', 'lastCheck': 1384917664.6252999, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00170707702637', 'lastCheck': 1384917667.245239, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00131297111511', 'lastCheck': 1384917667.2373669, 'valid': True, 'code': 0}} >Thread-15914046::DEBUG::2013-11-20 03:21:11,615::task::495::TaskManager.Task::(_debug) Task 45e14a2e-29e1-4f82-8c9b-7ef0cb05adec: moving from state preparing -> state finished >Thread-15914046::DEBUG::2013-11-20 03:21:11,615::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914046::DEBUG::2013-11-20 03:21:11,616::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914046::DEBUG::2013-11-20 03:21:11,617::task::495::TaskManager.Task::(_debug) Task 45e14a2e-29e1-4f82-8c9b-7ef0cb05adec: ref 0 aborting False >Thread-15914046::INFO::2013-11-20 03:21:11,617::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000303030014038', 'lastCheck': 1384917668.2381959, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00154900550842', 'lastCheck': 1384917664.6252999, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00170707702637', 'lastCheck': 1384917667.245239, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00131297111511', 'lastCheck': 1384917667.2373669, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:21:12,102::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913546::INFO::2013-11-20 03:21:12,349::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:21:12,375::task::495::TaskManager.Task::(_debug) Task 688eee35-ee33-45a9-a835-13de89c77170: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:21:12,384::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`8ec865ed-a1f0-4882-8e77-29b8f78a8a3e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:21:12,385::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:21:12,386::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:21:12,387::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`8ec865ed-a1f0-4882-8e77-29b8f78a8a3e`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:21:12,388::task::495::TaskManager.Task::(_debug) Task 688eee35-ee33-45a9-a835-13de89c77170: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:21:12,389::task::495::TaskManager.Task::(_debug) Task 688eee35-ee33-45a9-a835-13de89c77170: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:21:12,390::task::495::TaskManager.Task::(_debug) Task 688eee35-ee33-45a9-a835-13de89c77170: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:21:12,391::task::495::TaskManager.Task::(_debug) Task 688eee35-ee33-45a9-a835-13de89c77170: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:21:12,391::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:21:12,392::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:21:12,393::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:21:12,393::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:21:12,394::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:21:12,395::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:21:12,396::task::495::TaskManager.Task::(_debug) Task 688eee35-ee33-45a9-a835-13de89c77170: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:21:12,396::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:21:12,398::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:21:12,399::task::495::TaskManager.Task::(_debug) Task d4de1189-f0f4-4419-b08b-bd5d5ef4cc8b: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:21:12,400::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`8658c9da-08da-4274-a48b-d24f4ba4a3d7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:21:12,400::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:21:12,401::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:21:12,402::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`8658c9da-08da-4274-a48b-d24f4ba4a3d7`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:21:12,403::task::495::TaskManager.Task::(_debug) Task d4de1189-f0f4-4419-b08b-bd5d5ef4cc8b: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:21:12,404::task::495::TaskManager.Task::(_debug) Task d4de1189-f0f4-4419-b08b-bd5d5ef4cc8b: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:21:12,405::task::495::TaskManager.Task::(_debug) Task d4de1189-f0f4-4419-b08b-bd5d5ef4cc8b: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:21:12,405::task::495::TaskManager.Task::(_debug) Task d4de1189-f0f4-4419-b08b-bd5d5ef4cc8b: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:21:12,406::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:21:12,407::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:21:12,408::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:21:12,409::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:21:12,410::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:21:12,411::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:21:12,412::task::495::TaskManager.Task::(_debug) Task d4de1189-f0f4-4419-b08b-bd5d5ef4cc8b: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:21:12,413::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:21:12,415::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:21:12,417::task::495::TaskManager.Task::(_debug) Task 59050bb4-ee60-4a36-8eca-df2a0893276d: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:21:12,418::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`f422bdf9-a81c-4deb-8d9f-9b06a1665381`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:21:12,421::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:21:12,422::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:21:12,422::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`f422bdf9-a81c-4deb-8d9f-9b06a1665381`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:21:12,425::task::495::TaskManager.Task::(_debug) Task 59050bb4-ee60-4a36-8eca-df2a0893276d: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:21:12,429::task::495::TaskManager.Task::(_debug) Task 59050bb4-ee60-4a36-8eca-df2a0893276d: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:21:12,432::task::495::TaskManager.Task::(_debug) Task 59050bb4-ee60-4a36-8eca-df2a0893276d: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:21:12,435::task::495::TaskManager.Task::(_debug) Task 59050bb4-ee60-4a36-8eca-df2a0893276d: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:21:12,437::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:21:12,437::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:21:12,438::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:21:12,439::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:21:12,439::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:21:12,440::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:21:12,440::task::495::TaskManager.Task::(_debug) Task 59050bb4-ee60-4a36-8eca-df2a0893276d: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:21:12,441::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:21:12,442::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:21:12,442::task::495::TaskManager.Task::(_debug) Task 5d23e1f7-c09d-4dc1-86ee-150a630dde2d: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:21:12,443::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`aaacb4e9-571d-40bf-ae95-9c92a6d300f8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:21:12,444::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:21:12,445::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:21:12,445::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`aaacb4e9-571d-40bf-ae95-9c92a6d300f8`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:21:12,446::task::495::TaskManager.Task::(_debug) Task 5d23e1f7-c09d-4dc1-86ee-150a630dde2d: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:21:12,447::task::495::TaskManager.Task::(_debug) Task 5d23e1f7-c09d-4dc1-86ee-150a630dde2d: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:21:12,447::task::495::TaskManager.Task::(_debug) Task 5d23e1f7-c09d-4dc1-86ee-150a630dde2d: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:21:12,448::task::495::TaskManager.Task::(_debug) Task 5d23e1f7-c09d-4dc1-86ee-150a630dde2d: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:21:12,449::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:21:12,450::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:21:12,450::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:21:12,451::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:21:12,452::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:21:12,452::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:21:12,453::task::495::TaskManager.Task::(_debug) Task 5d23e1f7-c09d-4dc1-86ee-150a630dde2d: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:21:12,454::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:21:12,674::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0814751 s, 12.6 MB/s\n'; <rc> = 0 >Thread-15914048::DEBUG::2013-11-20 03:21:13,316::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914048::INFO::2013-11-20 03:21:13,318::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914048::DEBUG::2013-11-20 03:21:13,318::task::495::TaskManager.Task::(_debug) Task 492e6444-35a9-4262-b9a0-1bc30bb67268: moving from state init -> state preparing >Thread-15914048::DEBUG::2013-11-20 03:21:13,319::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914048::DEBUG::2013-11-20 03:21:13,320::task::495::TaskManager.Task::(_debug) Task 492e6444-35a9-4262-b9a0-1bc30bb67268: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914048::DEBUG::2013-11-20 03:21:13,320::task::495::TaskManager.Task::(_debug) Task 492e6444-35a9-4262-b9a0-1bc30bb67268: moving from state preparing -> state finished >Thread-15914048::DEBUG::2013-11-20 03:21:13,321::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914048::DEBUG::2013-11-20 03:21:13,321::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914048::DEBUG::2013-11-20 03:21:13,322::task::495::TaskManager.Task::(_debug) Task 492e6444-35a9-4262-b9a0-1bc30bb67268: ref 0 aborting False >Thread-15914048::INFO::2013-11-20 03:21:13,322::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914049::DEBUG::2013-11-20 03:21:13,379::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914049::INFO::2013-11-20 03:21:13,380::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914049::DEBUG::2013-11-20 03:21:13,381::task::495::TaskManager.Task::(_debug) Task 45622d33-e363-4c28-81ae-5ffc722054b4: moving from state init -> state preparing >Thread-15914049::DEBUG::2013-11-20 03:21:13,382::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9de4f802-a701-424f-a194-4ad66559ce04`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914049::DEBUG::2013-11-20 03:21:13,382::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914049::DEBUG::2013-11-20 03:21:13,383::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914049::DEBUG::2013-11-20 03:21:13,384::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9de4f802-a701-424f-a194-4ad66559ce04`::Granted request >Thread-15914049::DEBUG::2013-11-20 03:21:13,384::task::495::TaskManager.Task::(_debug) Task 45622d33-e363-4c28-81ae-5ffc722054b4: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914049::DEBUG::2013-11-20 03:21:13,385::task::495::TaskManager.Task::(_debug) Task 45622d33-e363-4c28-81ae-5ffc722054b4: ref 1 aborting False >Thread-15914049::WARNING::2013-11-20 03:21:13,387::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914049::DEBUG::2013-11-20 03:21:13,388::task::495::TaskManager.Task::(_debug) Task 45622d33-e363-4c28-81ae-5ffc722054b4: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914049::DEBUG::2013-11-20 03:21:13,389::task::495::TaskManager.Task::(_debug) Task 45622d33-e363-4c28-81ae-5ffc722054b4: moving from state preparing -> state finished >Thread-15914049::DEBUG::2013-11-20 03:21:13,389::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914049::DEBUG::2013-11-20 03:21:13,390::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914049::DEBUG::2013-11-20 03:21:13,390::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914049::DEBUG::2013-11-20 03:21:13,391::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914049::DEBUG::2013-11-20 03:21:13,392::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914049::DEBUG::2013-11-20 03:21:13,392::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914049::DEBUG::2013-11-20 03:21:13,393::task::495::TaskManager.Task::(_debug) Task 45622d33-e363-4c28-81ae-5ffc722054b4: ref 0 aborting False >Thread-15914049::INFO::2013-11-20 03:21:13,393::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:21:14,690::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:21:15,256::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0812706 s, 12.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:21:17,272::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:21:17,843::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0833439 s, 12.3 MB/s\n'; <rc> = 0 >Thread-15913640::DEBUG::2013-11-20 03:21:19,532::guestIF::97::vm.Vm::(_handleMessage) vmId=`4760db3e-d3a8-4045-9153-12c36ec16ba2`::u'AOS1Batch@brd-klee.dk' >Dummy-15912651::DEBUG::2013-11-20 03:21:19,854::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:21:20,464::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0866953 s, 11.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:21:22,477::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913639::INFO::2013-11-20 03:21:22,744::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:21:22,745::task::495::TaskManager.Task::(_debug) Task d116add1-0f62-4315-9e24-299eec1e95a1: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:21:22,748::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`9c3a16c9-e0f9-4a04-975a-0e8cc200c521`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:21:22,749::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:21:22,750::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:21:22,751::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`9c3a16c9-e0f9-4a04-975a-0e8cc200c521`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:21:22,752::task::495::TaskManager.Task::(_debug) Task d116add1-0f62-4315-9e24-299eec1e95a1: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:21:22,753::task::495::TaskManager.Task::(_debug) Task d116add1-0f62-4315-9e24-299eec1e95a1: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:21:22,754::task::495::TaskManager.Task::(_debug) Task d116add1-0f62-4315-9e24-299eec1e95a1: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:21:22,754::task::495::TaskManager.Task::(_debug) Task d116add1-0f62-4315-9e24-299eec1e95a1: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:21:22,755::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:21:22,756::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:21:22,757::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:21:22,757::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:21:22,758::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:21:22,759::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:21:22,759::task::495::TaskManager.Task::(_debug) Task d116add1-0f62-4315-9e24-299eec1e95a1: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:21:22,760::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:21:23,070::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0801527 s, 12.8 MB/s\n'; <rc> = 0 >Thread-15914053::DEBUG::2013-11-20 03:21:23,865::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914053::INFO::2013-11-20 03:21:23,867::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914053::DEBUG::2013-11-20 03:21:23,868::task::495::TaskManager.Task::(_debug) Task 922343eb-dcde-4247-98e5-f57e5cd451e3: moving from state init -> state preparing >Thread-15914053::DEBUG::2013-11-20 03:21:23,869::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914053::DEBUG::2013-11-20 03:21:23,869::task::495::TaskManager.Task::(_debug) Task 922343eb-dcde-4247-98e5-f57e5cd451e3: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914053::DEBUG::2013-11-20 03:21:23,870::task::495::TaskManager.Task::(_debug) Task 922343eb-dcde-4247-98e5-f57e5cd451e3: moving from state preparing -> state finished >Thread-15914053::DEBUG::2013-11-20 03:21:23,871::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914053::DEBUG::2013-11-20 03:21:23,871::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914053::DEBUG::2013-11-20 03:21:23,872::task::495::TaskManager.Task::(_debug) Task 922343eb-dcde-4247-98e5-f57e5cd451e3: ref 0 aborting False >Thread-15914053::INFO::2013-11-20 03:21:23,873::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914054::DEBUG::2013-11-20 03:21:23,878::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914054::INFO::2013-11-20 03:21:23,880::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914054::DEBUG::2013-11-20 03:21:23,880::task::495::TaskManager.Task::(_debug) Task 6ca775b2-a3f6-493f-a094-4a0d20389158: moving from state init -> state preparing >Thread-15914054::DEBUG::2013-11-20 03:21:23,881::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`66b4e6e5-9684-48b6-81ad-3f41a09c2966`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914054::DEBUG::2013-11-20 03:21:23,882::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914054::DEBUG::2013-11-20 03:21:23,883::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914054::DEBUG::2013-11-20 03:21:23,884::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`66b4e6e5-9684-48b6-81ad-3f41a09c2966`::Granted request >Thread-15914054::DEBUG::2013-11-20 03:21:23,885::task::495::TaskManager.Task::(_debug) Task 6ca775b2-a3f6-493f-a094-4a0d20389158: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914054::DEBUG::2013-11-20 03:21:23,886::task::495::TaskManager.Task::(_debug) Task 6ca775b2-a3f6-493f-a094-4a0d20389158: ref 1 aborting False >Thread-15914054::WARNING::2013-11-20 03:21:23,888::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914054::DEBUG::2013-11-20 03:21:23,888::task::495::TaskManager.Task::(_debug) Task 6ca775b2-a3f6-493f-a094-4a0d20389158: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914054::DEBUG::2013-11-20 03:21:23,889::task::495::TaskManager.Task::(_debug) Task 6ca775b2-a3f6-493f-a094-4a0d20389158: moving from state preparing -> state finished >Thread-15914054::DEBUG::2013-11-20 03:21:23,890::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914054::DEBUG::2013-11-20 03:21:23,890::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914054::DEBUG::2013-11-20 03:21:23,891::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914054::DEBUG::2013-11-20 03:21:23,892::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914054::DEBUG::2013-11-20 03:21:23,892::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914054::DEBUG::2013-11-20 03:21:23,893::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914054::DEBUG::2013-11-20 03:21:23,894::task::495::TaskManager.Task::(_debug) Task 6ca775b2-a3f6-493f-a094-4a0d20389158: ref 0 aborting False >Thread-15914054::INFO::2013-11-20 03:21:23,894::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:21:25,087::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:21:25,632::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0855935 s, 12.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:21:27,648::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:21:28,250::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0843502 s, 12.1 MB/s\n'; <rc> = 0 >Thread-15914056::INFO::2013-11-20 03:21:28,505::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914056::DEBUG::2013-11-20 03:21:28,506::task::495::TaskManager.Task::(_debug) Task 0c722992-a67d-481b-ab98-0bbf5ce32ddc: moving from state init -> state preparing >Thread-15914056::DEBUG::2013-11-20 03:21:28,507::task::495::TaskManager.Task::(_debug) Task 0c722992-a67d-481b-ab98-0bbf5ce32ddc: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000383853912354', 'lastCheck': 1384917678.4852309, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00132513046265', 'lastCheck': 1384917685.309761, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00180387496948', 'lastCheck': 1384917687.949965, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0194098949432', 'lastCheck': 1384917687.933697, 'valid': True, 'code': 0}} >Thread-15914056::DEBUG::2013-11-20 03:21:28,507::task::495::TaskManager.Task::(_debug) Task 0c722992-a67d-481b-ab98-0bbf5ce32ddc: moving from state preparing -> state finished >Thread-15914056::DEBUG::2013-11-20 03:21:28,508::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914056::DEBUG::2013-11-20 03:21:28,508::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914056::DEBUG::2013-11-20 03:21:28,509::task::495::TaskManager.Task::(_debug) Task 0c722992-a67d-481b-ab98-0bbf5ce32ddc: ref 0 aborting False >Thread-15914056::INFO::2013-11-20 03:21:28,510::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000383853912354', 'lastCheck': 1384917678.4852309, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00132513046265', 'lastCheck': 1384917685.309761, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00180387496948', 'lastCheck': 1384917687.949965, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0194098949432', 'lastCheck': 1384917687.933697, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:21:30,267::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:21:30,848::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0888232 s, 11.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:21:32,859::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:21:33,373::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.054935 s, 18.6 MB/s\n'; <rc> = 0 >Thread-15914059::DEBUG::2013-11-20 03:21:35,227::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914059::INFO::2013-11-20 03:21:35,228::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914059::DEBUG::2013-11-20 03:21:35,228::task::495::TaskManager.Task::(_debug) Task 154299de-71c9-48ca-860c-674db4bac90a: moving from state init -> state preparing >Thread-15914059::DEBUG::2013-11-20 03:21:35,229::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914059::DEBUG::2013-11-20 03:21:35,229::task::495::TaskManager.Task::(_debug) Task 154299de-71c9-48ca-860c-674db4bac90a: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914059::DEBUG::2013-11-20 03:21:35,230::task::495::TaskManager.Task::(_debug) Task 154299de-71c9-48ca-860c-674db4bac90a: moving from state preparing -> state finished >Thread-15914059::DEBUG::2013-11-20 03:21:35,230::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914059::DEBUG::2013-11-20 03:21:35,230::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914059::DEBUG::2013-11-20 03:21:35,231::task::495::TaskManager.Task::(_debug) Task 154299de-71c9-48ca-860c-674db4bac90a: ref 0 aborting False >Thread-15914059::INFO::2013-11-20 03:21:35,231::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914060::DEBUG::2013-11-20 03:21:35,235::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914060::INFO::2013-11-20 03:21:35,236::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914060::DEBUG::2013-11-20 03:21:35,237::task::495::TaskManager.Task::(_debug) Task b2ec2752-562b-425b-9d5b-383722f4ac8f: moving from state init -> state preparing >Thread-15914060::DEBUG::2013-11-20 03:21:35,238::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`52b8ec82-7878-4735-ac38-e9f046c17fbb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914060::DEBUG::2013-11-20 03:21:35,239::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914060::DEBUG::2013-11-20 03:21:35,239::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914060::DEBUG::2013-11-20 03:21:35,240::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`52b8ec82-7878-4735-ac38-e9f046c17fbb`::Granted request >Thread-15914060::DEBUG::2013-11-20 03:21:35,241::task::495::TaskManager.Task::(_debug) Task b2ec2752-562b-425b-9d5b-383722f4ac8f: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914060::DEBUG::2013-11-20 03:21:35,242::task::495::TaskManager.Task::(_debug) Task b2ec2752-562b-425b-9d5b-383722f4ac8f: ref 1 aborting False >Thread-15914060::WARNING::2013-11-20 03:21:35,244::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914060::DEBUG::2013-11-20 03:21:35,245::task::495::TaskManager.Task::(_debug) Task b2ec2752-562b-425b-9d5b-383722f4ac8f: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914060::DEBUG::2013-11-20 03:21:35,246::task::495::TaskManager.Task::(_debug) Task b2ec2752-562b-425b-9d5b-383722f4ac8f: moving from state preparing -> state finished >Thread-15914060::DEBUG::2013-11-20 03:21:35,246::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914060::DEBUG::2013-11-20 03:21:35,247::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914060::DEBUG::2013-11-20 03:21:35,247::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914060::DEBUG::2013-11-20 03:21:35,248::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914060::DEBUG::2013-11-20 03:21:35,249::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914060::DEBUG::2013-11-20 03:21:35,249::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914060::DEBUG::2013-11-20 03:21:35,250::task::495::TaskManager.Task::(_debug) Task b2ec2752-562b-425b-9d5b-383722f4ac8f: ref 0 aborting False >Thread-15914060::INFO::2013-11-20 03:21:35,250::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:21:35,393::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:21:35,932::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0760732 s, 13.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:21:37,948::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:21:38,459::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0815302 s, 12.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:21:40,474::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913695::INFO::2013-11-20 03:21:40,692::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:21:40,694::task::495::TaskManager.Task::(_debug) Task 2b9a65b3-34fb-4833-b975-9ae90b2a034f: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:21:40,697::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`2b32bc57-341c-4890-aaa4-3da2ba55a4f6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:21:40,698::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:21:40,699::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:21:40,700::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`2b32bc57-341c-4890-aaa4-3da2ba55a4f6`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:21:40,701::task::495::TaskManager.Task::(_debug) Task 2b9a65b3-34fb-4833-b975-9ae90b2a034f: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:21:40,702::task::495::TaskManager.Task::(_debug) Task 2b9a65b3-34fb-4833-b975-9ae90b2a034f: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:21:40,703::task::495::TaskManager.Task::(_debug) Task 2b9a65b3-34fb-4833-b975-9ae90b2a034f: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:21:40,704::task::495::TaskManager.Task::(_debug) Task 2b9a65b3-34fb-4833-b975-9ae90b2a034f: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:21:40,704::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:21:40,705::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:21:40,706::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:21:40,707::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:21:40,707::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:21:40,708::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:21:40,709::task::495::TaskManager.Task::(_debug) Task 2b9a65b3-34fb-4833-b975-9ae90b2a034f: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:21:40,710::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:21:40,994::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0748524 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:21:43,007::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913658::INFO::2013-11-20 03:21:43,224::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:21:43,226::task::495::TaskManager.Task::(_debug) Task 06aee392-df4b-4d97-94b0-f0b19f97b1e9: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:21:43,230::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`cc52529c-73e2-4845-86ec-6af72b8a9e04`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:21:43,231::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:21:43,233::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:21:43,233::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`cc52529c-73e2-4845-86ec-6af72b8a9e04`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:21:43,234::task::495::TaskManager.Task::(_debug) Task 06aee392-df4b-4d97-94b0-f0b19f97b1e9: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:21:43,235::task::495::TaskManager.Task::(_debug) Task 06aee392-df4b-4d97-94b0-f0b19f97b1e9: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:21:43,236::task::495::TaskManager.Task::(_debug) Task 06aee392-df4b-4d97-94b0-f0b19f97b1e9: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:21:43,236::task::495::TaskManager.Task::(_debug) Task 06aee392-df4b-4d97-94b0-f0b19f97b1e9: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:21:43,237::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:21:43,237::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:21:43,238::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:21:43,239::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:21:43,239::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:21:43,239::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:21:43,240::task::495::TaskManager.Task::(_debug) Task 06aee392-df4b-4d97-94b0-f0b19f97b1e9: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:21:43,241::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:21:43,437::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0450977 s, 22.7 MB/s\n'; <rc> = 0 >Thread-15914064::INFO::2013-11-20 03:21:45,438::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914064::DEBUG::2013-11-20 03:21:45,439::task::495::TaskManager.Task::(_debug) Task 2df59309-c719-4efa-8da8-60995fb21028: moving from state init -> state preparing >Thread-15914064::DEBUG::2013-11-20 03:21:45,440::task::495::TaskManager.Task::(_debug) Task 2df59309-c719-4efa-8da8-60995fb21028: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000291109085083', 'lastCheck': 1384917699.1476021, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137090682983', 'lastCheck': 1384917695.6106739, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00110101699829', 'lastCheck': 1384917698.150955, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00251984596252', 'lastCheck': 1384917698.147362, 'valid': True, 'code': 0}} >Thread-15914064::DEBUG::2013-11-20 03:21:45,440::task::495::TaskManager.Task::(_debug) Task 2df59309-c719-4efa-8da8-60995fb21028: moving from state preparing -> state finished >Thread-15914064::DEBUG::2013-11-20 03:21:45,441::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914064::DEBUG::2013-11-20 03:21:45,441::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914064::DEBUG::2013-11-20 03:21:45,441::task::495::TaskManager.Task::(_debug) Task 2df59309-c719-4efa-8da8-60995fb21028: ref 0 aborting False >Thread-15914064::INFO::2013-11-20 03:21:45,442::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000291109085083', 'lastCheck': 1384917699.1476021, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00137090682983', 'lastCheck': 1384917695.6106739, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00110101699829', 'lastCheck': 1384917698.150955, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00251984596252', 'lastCheck': 1384917698.147362, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:21:45,448::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914066::DEBUG::2013-11-20 03:21:45,712::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914066::INFO::2013-11-20 03:21:45,715::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914066::DEBUG::2013-11-20 03:21:45,715::task::495::TaskManager.Task::(_debug) Task 5f338598-965d-4560-a0b2-b3ff14a72018: moving from state init -> state preparing >Thread-15914066::DEBUG::2013-11-20 03:21:45,717::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914066::DEBUG::2013-11-20 03:21:45,718::task::495::TaskManager.Task::(_debug) Task 5f338598-965d-4560-a0b2-b3ff14a72018: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914066::DEBUG::2013-11-20 03:21:45,725::task::495::TaskManager.Task::(_debug) Task 5f338598-965d-4560-a0b2-b3ff14a72018: moving from state preparing -> state finished >Thread-15914066::DEBUG::2013-11-20 03:21:45,726::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914066::DEBUG::2013-11-20 03:21:45,727::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914066::DEBUG::2013-11-20 03:21:45,728::task::495::TaskManager.Task::(_debug) Task 5f338598-965d-4560-a0b2-b3ff14a72018: ref 0 aborting False >Thread-15914066::INFO::2013-11-20 03:21:45,728::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914067::DEBUG::2013-11-20 03:21:45,749::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914067::INFO::2013-11-20 03:21:45,750::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914067::DEBUG::2013-11-20 03:21:45,751::task::495::TaskManager.Task::(_debug) Task 20d4ce52-0d18-4f03-84d5-97f6690d268f: moving from state init -> state preparing >Thread-15914067::DEBUG::2013-11-20 03:21:45,753::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`76ed6e25-0208-49f0-aa40-b8bb69700dc0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914067::DEBUG::2013-11-20 03:21:45,753::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914067::DEBUG::2013-11-20 03:21:45,754::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914067::DEBUG::2013-11-20 03:21:45,755::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`76ed6e25-0208-49f0-aa40-b8bb69700dc0`::Granted request >Thread-15914067::DEBUG::2013-11-20 03:21:45,762::task::495::TaskManager.Task::(_debug) Task 20d4ce52-0d18-4f03-84d5-97f6690d268f: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914067::DEBUG::2013-11-20 03:21:45,762::task::495::TaskManager.Task::(_debug) Task 20d4ce52-0d18-4f03-84d5-97f6690d268f: ref 1 aborting False >Thread-15914067::WARNING::2013-11-20 03:21:45,765::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914067::DEBUG::2013-11-20 03:21:45,772::task::495::TaskManager.Task::(_debug) Task 20d4ce52-0d18-4f03-84d5-97f6690d268f: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914067::DEBUG::2013-11-20 03:21:45,773::task::495::TaskManager.Task::(_debug) Task 20d4ce52-0d18-4f03-84d5-97f6690d268f: moving from state preparing -> state finished >Thread-15914067::DEBUG::2013-11-20 03:21:45,774::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914067::DEBUG::2013-11-20 03:21:45,775::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914067::DEBUG::2013-11-20 03:21:45,776::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914067::DEBUG::2013-11-20 03:21:45,777::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914067::DEBUG::2013-11-20 03:21:45,777::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914067::DEBUG::2013-11-20 03:21:45,778::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914067::DEBUG::2013-11-20 03:21:45,779::task::495::TaskManager.Task::(_debug) Task 20d4ce52-0d18-4f03-84d5-97f6690d268f: ref 0 aborting False >Thread-15914067::INFO::2013-11-20 03:21:45,779::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:21:46,022::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.113882 s, 9.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:21:48,037::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 03:21:48,240::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:21:48,246::task::495::TaskManager.Task::(_debug) Task 99d5ae1e-3228-4459-be49-3c32aed7d065: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:21:48,252::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`dd150457-5f61-4817-a198-6de97d51b9dc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:21:48,253::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:21:48,253::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:21:48,254::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`dd150457-5f61-4817-a198-6de97d51b9dc`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:21:48,254::task::495::TaskManager.Task::(_debug) Task 99d5ae1e-3228-4459-be49-3c32aed7d065: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:21:48,255::task::495::TaskManager.Task::(_debug) Task 99d5ae1e-3228-4459-be49-3c32aed7d065: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:21:48,255::task::495::TaskManager.Task::(_debug) Task 99d5ae1e-3228-4459-be49-3c32aed7d065: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:21:48,256::task::495::TaskManager.Task::(_debug) Task 99d5ae1e-3228-4459-be49-3c32aed7d065: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:21:48,257::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:21:48,257::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:21:48,257::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:21:48,258::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:21:48,258::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:21:48,259::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:21:48,259::task::495::TaskManager.Task::(_debug) Task 99d5ae1e-3228-4459-be49-3c32aed7d065: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:21:48,259::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:21:48,561::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0818734 s, 12.5 MB/s\n'; <rc> = 0 >Thread-42838::DEBUG::2013-11-20 03:21:49,244::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 03:21:49,857::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:21:50,578::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:21:51,047::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0870576 s, 11.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:21:53,062::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:21:53,586::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0844231 s, 12.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:21:55,598::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914071::DEBUG::2013-11-20 03:21:56,076::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914071::INFO::2013-11-20 03:21:56,079::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914071::DEBUG::2013-11-20 03:21:56,080::task::495::TaskManager.Task::(_debug) Task 400c6aaf-f266-4682-a003-54fba8f84993: moving from state init -> state preparing >Thread-15914071::DEBUG::2013-11-20 03:21:56,082::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914071::DEBUG::2013-11-20 03:21:56,083::task::495::TaskManager.Task::(_debug) Task 400c6aaf-f266-4682-a003-54fba8f84993: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914071::DEBUG::2013-11-20 03:21:56,084::task::495::TaskManager.Task::(_debug) Task 400c6aaf-f266-4682-a003-54fba8f84993: moving from state preparing -> state finished >Thread-15914071::DEBUG::2013-11-20 03:21:56,085::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914071::DEBUG::2013-11-20 03:21:56,086::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914071::DEBUG::2013-11-20 03:21:56,087::task::495::TaskManager.Task::(_debug) Task 400c6aaf-f266-4682-a003-54fba8f84993: ref 0 aborting False >Thread-15914071::INFO::2013-11-20 03:21:56,089::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914072::DEBUG::2013-11-20 03:21:56,096::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914072::INFO::2013-11-20 03:21:56,097::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914072::DEBUG::2013-11-20 03:21:56,099::task::495::TaskManager.Task::(_debug) Task ad5b3306-e7d4-4c72-82b4-d9a717212bab: moving from state init -> state preparing >Thread-15914072::DEBUG::2013-11-20 03:21:56,101::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b37d9e1b-3341-41d5-b7fd-7fcc5cdb7fdb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914072::DEBUG::2013-11-20 03:21:56,103::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914072::DEBUG::2013-11-20 03:21:56,104::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914072::DEBUG::2013-11-20 03:21:56,105::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b37d9e1b-3341-41d5-b7fd-7fcc5cdb7fdb`::Granted request >Thread-15914072::DEBUG::2013-11-20 03:21:56,107::task::495::TaskManager.Task::(_debug) Task ad5b3306-e7d4-4c72-82b4-d9a717212bab: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914072::DEBUG::2013-11-20 03:21:56,107::task::495::TaskManager.Task::(_debug) Task ad5b3306-e7d4-4c72-82b4-d9a717212bab: ref 1 aborting False >Thread-15914072::WARNING::2013-11-20 03:21:56,110::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914072::DEBUG::2013-11-20 03:21:56,112::task::495::TaskManager.Task::(_debug) Task ad5b3306-e7d4-4c72-82b4-d9a717212bab: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914072::DEBUG::2013-11-20 03:21:56,113::task::495::TaskManager.Task::(_debug) Task ad5b3306-e7d4-4c72-82b4-d9a717212bab: moving from state preparing -> state finished >Thread-15914072::DEBUG::2013-11-20 03:21:56,114::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914072::DEBUG::2013-11-20 03:21:56,115::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914072::DEBUG::2013-11-20 03:21:56,116::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914072::DEBUG::2013-11-20 03:21:56,117::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914072::DEBUG::2013-11-20 03:21:56,118::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914072::DEBUG::2013-11-20 03:21:56,119::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914072::DEBUG::2013-11-20 03:21:56,120::task::495::TaskManager.Task::(_debug) Task ad5b3306-e7d4-4c72-82b4-d9a717212bab: ref 0 aborting False >Thread-15914072::INFO::2013-11-20 03:21:56,121::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:21:56,138::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0773564 s, 13.2 MB/s\n'; <rc> = 0 >Thread-15913518::INFO::2013-11-20 03:21:56,502::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:21:56,503::task::495::TaskManager.Task::(_debug) Task ba714483-e062-4371-89c9-29aa95b16d74: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:21:56,504::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`bb4e134c-72d2-4c1b-80bf-2a0dd01b42d2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:21:56,505::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:21:56,505::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:21:56,506::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`bb4e134c-72d2-4c1b-80bf-2a0dd01b42d2`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:21:56,506::task::495::TaskManager.Task::(_debug) Task ba714483-e062-4371-89c9-29aa95b16d74: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:21:56,507::task::495::TaskManager.Task::(_debug) Task ba714483-e062-4371-89c9-29aa95b16d74: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:21:56,507::task::495::TaskManager.Task::(_debug) Task ba714483-e062-4371-89c9-29aa95b16d74: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:21:56,508::task::495::TaskManager.Task::(_debug) Task ba714483-e062-4371-89c9-29aa95b16d74: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:21:56,508::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:21:56,509::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:21:56,510::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:21:56,510::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:21:56,511::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:21:56,511::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:21:56,512::task::495::TaskManager.Task::(_debug) Task ba714483-e062-4371-89c9-29aa95b16d74: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:21:56,512::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:21:56,513::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:21:56,513::task::495::TaskManager.Task::(_debug) Task d07b1120-a050-457f-9083-76d5e015262e: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:21:56,514::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`173fc5be-2cbf-4513-ba7f-a0a885950d81`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:21:56,515::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:21:56,516::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:21:56,516::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`173fc5be-2cbf-4513-ba7f-a0a885950d81`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:21:56,517::task::495::TaskManager.Task::(_debug) Task d07b1120-a050-457f-9083-76d5e015262e: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:21:56,517::task::495::TaskManager.Task::(_debug) Task d07b1120-a050-457f-9083-76d5e015262e: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:21:56,518::task::495::TaskManager.Task::(_debug) Task d07b1120-a050-457f-9083-76d5e015262e: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:21:56,518::task::495::TaskManager.Task::(_debug) Task d07b1120-a050-457f-9083-76d5e015262e: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:21:56,518::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:21:56,519::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:21:56,520::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:21:56,520::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:21:56,520::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:21:56,521::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:21:56,522::task::495::TaskManager.Task::(_debug) Task d07b1120-a050-457f-9083-76d5e015262e: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:21:56,522::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:21:58,151::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:21:58,553::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0526587 s, 19.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:22:00,572::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:22:01,105::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0766989 s, 13.4 MB/s\n'; <rc> = 0 >Thread-15914074::INFO::2013-11-20 03:22:01,334::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914074::DEBUG::2013-11-20 03:22:01,335::task::495::TaskManager.Task::(_debug) Task 99a66120-90f8-48e3-9908-df15a8a92bb8: moving from state init -> state preparing >Thread-15914074::DEBUG::2013-11-20 03:22:01,336::task::495::TaskManager.Task::(_debug) Task 99a66120-90f8-48e3-9908-df15a8a92bb8: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000314950942993', 'lastCheck': 1384917719.8721621, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00195813179016', 'lastCheck': 1384917715.8134961, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00116014480591', 'lastCheck': 1384917718.4123321, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00225400924683', 'lastCheck': 1384917718.409816, 'valid': True, 'code': 0}} >Thread-15914074::DEBUG::2013-11-20 03:22:01,337::task::495::TaskManager.Task::(_debug) Task 99a66120-90f8-48e3-9908-df15a8a92bb8: moving from state preparing -> state finished >Thread-15914074::DEBUG::2013-11-20 03:22:01,337::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914074::DEBUG::2013-11-20 03:22:01,338::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914074::DEBUG::2013-11-20 03:22:01,338::task::495::TaskManager.Task::(_debug) Task 99a66120-90f8-48e3-9908-df15a8a92bb8: ref 0 aborting False >Thread-15914074::INFO::2013-11-20 03:22:01,339::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000314950942993', 'lastCheck': 1384917719.8721621, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00195813179016', 'lastCheck': 1384917715.8134961, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00116014480591', 'lastCheck': 1384917718.4123321, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00225400924683', 'lastCheck': 1384917718.409816, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:22:03,122::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:22:03,569::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0456477 s, 22.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:22:05,580::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:22:06,107::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0716224 s, 14.3 MB/s\n'; <rc> = 0 >Thread-15914077::DEBUG::2013-11-20 03:22:06,365::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914077::INFO::2013-11-20 03:22:06,368::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914077::DEBUG::2013-11-20 03:22:06,368::task::495::TaskManager.Task::(_debug) Task 0b9f633e-0482-43fe-955e-6b2a7d1102f8: moving from state init -> state preparing >Thread-15914077::DEBUG::2013-11-20 03:22:06,370::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914077::DEBUG::2013-11-20 03:22:06,371::task::495::TaskManager.Task::(_debug) Task 0b9f633e-0482-43fe-955e-6b2a7d1102f8: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914077::DEBUG::2013-11-20 03:22:06,371::task::495::TaskManager.Task::(_debug) Task 0b9f633e-0482-43fe-955e-6b2a7d1102f8: moving from state preparing -> state finished >Thread-15914077::DEBUG::2013-11-20 03:22:06,372::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914077::DEBUG::2013-11-20 03:22:06,372::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914077::DEBUG::2013-11-20 03:22:06,373::task::495::TaskManager.Task::(_debug) Task 0b9f633e-0482-43fe-955e-6b2a7d1102f8: ref 0 aborting False >Thread-15914077::INFO::2013-11-20 03:22:06,374::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914078::DEBUG::2013-11-20 03:22:06,379::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914078::INFO::2013-11-20 03:22:06,380::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914078::DEBUG::2013-11-20 03:22:06,380::task::495::TaskManager.Task::(_debug) Task 03be9948-e697-40a1-8464-61d25593a927: moving from state init -> state preparing >Thread-15914078::DEBUG::2013-11-20 03:22:06,381::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d38c3136-32a4-4632-b29a-e3f06affbf6c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914078::DEBUG::2013-11-20 03:22:06,382::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914078::DEBUG::2013-11-20 03:22:06,383::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914078::DEBUG::2013-11-20 03:22:06,384::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d38c3136-32a4-4632-b29a-e3f06affbf6c`::Granted request >Thread-15914078::DEBUG::2013-11-20 03:22:06,385::task::495::TaskManager.Task::(_debug) Task 03be9948-e697-40a1-8464-61d25593a927: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914078::DEBUG::2013-11-20 03:22:06,386::task::495::TaskManager.Task::(_debug) Task 03be9948-e697-40a1-8464-61d25593a927: ref 1 aborting False >Thread-15914078::WARNING::2013-11-20 03:22:06,389::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914078::DEBUG::2013-11-20 03:22:06,390::task::495::TaskManager.Task::(_debug) Task 03be9948-e697-40a1-8464-61d25593a927: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914078::DEBUG::2013-11-20 03:22:06,391::task::495::TaskManager.Task::(_debug) Task 03be9948-e697-40a1-8464-61d25593a927: moving from state preparing -> state finished >Thread-15914078::DEBUG::2013-11-20 03:22:06,392::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914078::DEBUG::2013-11-20 03:22:06,393::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914078::DEBUG::2013-11-20 03:22:06,394::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914078::DEBUG::2013-11-20 03:22:06,394::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914078::DEBUG::2013-11-20 03:22:06,395::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914078::DEBUG::2013-11-20 03:22:06,396::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914078::DEBUG::2013-11-20 03:22:06,396::task::495::TaskManager.Task::(_debug) Task 03be9948-e697-40a1-8464-61d25593a927: ref 0 aborting False >Thread-15914078::INFO::2013-11-20 03:22:06,397::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:22:08,123::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:22:08,678::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0797713 s, 12.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:22:10,694::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:22:11,211::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0749017 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:22:13,226::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913546::INFO::2013-11-20 03:22:13,434::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:22:13,436::task::495::TaskManager.Task::(_debug) Task 31c4a5b6-7200-487c-a578-30536d588221: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:22:13,438::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`e91ee52f-7f07-40ba-8c8e-259014671ccf`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:22:13,439::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:22:13,440::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:22:13,440::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`e91ee52f-7f07-40ba-8c8e-259014671ccf`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:22:13,441::task::495::TaskManager.Task::(_debug) Task 31c4a5b6-7200-487c-a578-30536d588221: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:22:13,441::task::495::TaskManager.Task::(_debug) Task 31c4a5b6-7200-487c-a578-30536d588221: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:22:13,442::task::495::TaskManager.Task::(_debug) Task 31c4a5b6-7200-487c-a578-30536d588221: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:22:13,442::task::495::TaskManager.Task::(_debug) Task 31c4a5b6-7200-487c-a578-30536d588221: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:22:13,443::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:22:13,443::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:22:13,444::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:22:13,444::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:22:13,444::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:22:13,445::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:22:13,445::task::495::TaskManager.Task::(_debug) Task 31c4a5b6-7200-487c-a578-30536d588221: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:22:13,446::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:22:13,446::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:22:13,447::task::495::TaskManager.Task::(_debug) Task d26692f0-d6ec-4e83-aaaf-09a695619c69: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:22:13,447::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`059605e4-7b3c-4a4b-a404-4d83c2146ca2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:22:13,448::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:22:13,448::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:22:13,448::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`059605e4-7b3c-4a4b-a404-4d83c2146ca2`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:22:13,449::task::495::TaskManager.Task::(_debug) Task d26692f0-d6ec-4e83-aaaf-09a695619c69: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:22:13,449::task::495::TaskManager.Task::(_debug) Task d26692f0-d6ec-4e83-aaaf-09a695619c69: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:22:13,450::task::495::TaskManager.Task::(_debug) Task d26692f0-d6ec-4e83-aaaf-09a695619c69: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:22:13,450::task::495::TaskManager.Task::(_debug) Task d26692f0-d6ec-4e83-aaaf-09a695619c69: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:22:13,451::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:22:13,451::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:22:13,451::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:22:13,452::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:22:13,452::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:22:13,453::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:22:13,453::task::495::TaskManager.Task::(_debug) Task d26692f0-d6ec-4e83-aaaf-09a695619c69: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:22:13,454::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:22:13,454::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:22:13,454::task::495::TaskManager.Task::(_debug) Task 8383056b-c7ff-487e-9435-a8042b5d1503: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:22:13,455::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`9dc90df2-27b8-46b7-9ff1-2ca074cd7a60`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:22:13,455::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:22:13,456::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:22:13,456::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`9dc90df2-27b8-46b7-9ff1-2ca074cd7a60`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:22:13,457::task::495::TaskManager.Task::(_debug) Task 8383056b-c7ff-487e-9435-a8042b5d1503: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:22:13,457::task::495::TaskManager.Task::(_debug) Task 8383056b-c7ff-487e-9435-a8042b5d1503: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:22:13,458::task::495::TaskManager.Task::(_debug) Task 8383056b-c7ff-487e-9435-a8042b5d1503: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:22:13,458::task::495::TaskManager.Task::(_debug) Task 8383056b-c7ff-487e-9435-a8042b5d1503: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:22:13,458::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:22:13,459::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:22:13,459::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:22:13,460::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:22:13,460::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:22:13,460::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:22:13,461::task::495::TaskManager.Task::(_debug) Task 8383056b-c7ff-487e-9435-a8042b5d1503: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:22:13,461::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:22:13,462::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:22:13,462::task::495::TaskManager.Task::(_debug) Task 9c9ae5c3-0e7a-441a-a3c3-57edd866c041: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:22:13,463::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`95d5f89f-3884-4bf8-bb64-242325217073`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:22:13,463::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:22:13,464::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:22:13,464::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`95d5f89f-3884-4bf8-bb64-242325217073`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:22:13,465::task::495::TaskManager.Task::(_debug) Task 9c9ae5c3-0e7a-441a-a3c3-57edd866c041: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:22:13,466::task::495::TaskManager.Task::(_debug) Task 9c9ae5c3-0e7a-441a-a3c3-57edd866c041: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:22:13,467::task::495::TaskManager.Task::(_debug) Task 9c9ae5c3-0e7a-441a-a3c3-57edd866c041: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:22:13,467::task::495::TaskManager.Task::(_debug) Task 9c9ae5c3-0e7a-441a-a3c3-57edd866c041: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:22:13,468::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:22:13,468::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:22:13,469::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:22:13,470::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:22:13,470::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:22:13,471::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:22:13,471::task::495::TaskManager.Task::(_debug) Task 9c9ae5c3-0e7a-441a-a3c3-57edd866c041: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:22:13,472::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:22:13,757::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0833508 s, 12.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:22:15,772::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:22:16,320::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0759192 s, 13.5 MB/s\n'; <rc> = 0 >Thread-15914082::DEBUG::2013-11-20 03:22:16,951::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914082::INFO::2013-11-20 03:22:16,954::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914082::DEBUG::2013-11-20 03:22:16,954::task::495::TaskManager.Task::(_debug) Task daac5c45-a020-45f0-91f5-3782c4945f1d: moving from state init -> state preparing >Thread-15914082::DEBUG::2013-11-20 03:22:16,955::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914082::DEBUG::2013-11-20 03:22:16,956::task::495::TaskManager.Task::(_debug) Task daac5c45-a020-45f0-91f5-3782c4945f1d: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914082::DEBUG::2013-11-20 03:22:16,957::task::495::TaskManager.Task::(_debug) Task daac5c45-a020-45f0-91f5-3782c4945f1d: moving from state preparing -> state finished >Thread-15914082::DEBUG::2013-11-20 03:22:16,957::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914082::DEBUG::2013-11-20 03:22:16,958::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914082::DEBUG::2013-11-20 03:22:16,958::task::495::TaskManager.Task::(_debug) Task daac5c45-a020-45f0-91f5-3782c4945f1d: ref 0 aborting False >Thread-15914082::INFO::2013-11-20 03:22:16,959::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914083::DEBUG::2013-11-20 03:22:16,963::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914083::INFO::2013-11-20 03:22:16,964::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914083::DEBUG::2013-11-20 03:22:16,965::task::495::TaskManager.Task::(_debug) Task a767ee14-a178-499e-89c6-57a2d115e00d: moving from state init -> state preparing >Thread-15914083::DEBUG::2013-11-20 03:22:16,966::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`8d24a512-1e08-4da2-9d3c-4ee8ab88091f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914083::DEBUG::2013-11-20 03:22:16,967::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914083::DEBUG::2013-11-20 03:22:16,968::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914083::DEBUG::2013-11-20 03:22:16,969::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`8d24a512-1e08-4da2-9d3c-4ee8ab88091f`::Granted request >Thread-15914083::DEBUG::2013-11-20 03:22:16,971::task::495::TaskManager.Task::(_debug) Task a767ee14-a178-499e-89c6-57a2d115e00d: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914083::DEBUG::2013-11-20 03:22:16,972::task::495::TaskManager.Task::(_debug) Task a767ee14-a178-499e-89c6-57a2d115e00d: ref 1 aborting False >Thread-15914083::WARNING::2013-11-20 03:22:16,974::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914083::DEBUG::2013-11-20 03:22:16,975::task::495::TaskManager.Task::(_debug) Task a767ee14-a178-499e-89c6-57a2d115e00d: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914083::DEBUG::2013-11-20 03:22:16,976::task::495::TaskManager.Task::(_debug) Task a767ee14-a178-499e-89c6-57a2d115e00d: moving from state preparing -> state finished >Thread-15914083::DEBUG::2013-11-20 03:22:16,977::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914083::DEBUG::2013-11-20 03:22:16,977::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914084::INFO::2013-11-20 03:22:16,980::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914084::DEBUG::2013-11-20 03:22:16,981::task::495::TaskManager.Task::(_debug) Task db48b610-cacf-4827-8309-49e1f1e7664c: moving from state init -> state preparing >Thread-15914083::DEBUG::2013-11-20 03:22:16,982::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914084::DEBUG::2013-11-20 03:22:16,983::task::495::TaskManager.Task::(_debug) Task db48b610-cacf-4827-8309-49e1f1e7664c: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000312089920044', 'lastCheck': 1384917729.8859291, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00130796432495', 'lastCheck': 1384917736.076524, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0169382095337', 'lastCheck': 1384917728.443157, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000749111175537', 'lastCheck': 1384917728.4223771, 'valid': True, 'code': 0}} >Thread-15914083::DEBUG::2013-11-20 03:22:16,984::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914084::DEBUG::2013-11-20 03:22:16,985::task::495::TaskManager.Task::(_debug) Task db48b610-cacf-4827-8309-49e1f1e7664c: moving from state preparing -> state finished >Thread-15914083::DEBUG::2013-11-20 03:22:16,985::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914084::DEBUG::2013-11-20 03:22:16,986::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914083::DEBUG::2013-11-20 03:22:16,987::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914084::DEBUG::2013-11-20 03:22:16,987::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914083::DEBUG::2013-11-20 03:22:16,989::task::495::TaskManager.Task::(_debug) Task a767ee14-a178-499e-89c6-57a2d115e00d: ref 0 aborting False >Thread-15914084::DEBUG::2013-11-20 03:22:16,989::task::495::TaskManager.Task::(_debug) Task db48b610-cacf-4827-8309-49e1f1e7664c: ref 0 aborting False >Thread-15914083::INFO::2013-11-20 03:22:16,990::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914084::INFO::2013-11-20 03:22:16,991::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000312089920044', 'lastCheck': 1384917729.8859291, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00130796432495', 'lastCheck': 1384917736.076524, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0169382095337', 'lastCheck': 1384917728.443157, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000749111175537', 'lastCheck': 1384917728.4223771, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:22:18,334::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:22:18,867::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0797632 s, 12.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:22:20,881::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:22:21,378::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0731363 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:22:23,394::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913639::INFO::2013-11-20 03:22:23,611::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:22:23,614::task::495::TaskManager.Task::(_debug) Task 0935ea2a-fcc0-4136-8de5-d4288a1f0d92: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:22:23,616::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d0dd6d7b-2aba-4b5d-9545-5ebc16812fbf`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:22:23,617::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:22:23,618::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:22:23,619::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d0dd6d7b-2aba-4b5d-9545-5ebc16812fbf`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:22:23,620::task::495::TaskManager.Task::(_debug) Task 0935ea2a-fcc0-4136-8de5-d4288a1f0d92: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:22:23,621::task::495::TaskManager.Task::(_debug) Task 0935ea2a-fcc0-4136-8de5-d4288a1f0d92: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:22:23,622::task::495::TaskManager.Task::(_debug) Task 0935ea2a-fcc0-4136-8de5-d4288a1f0d92: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:22:23,622::task::495::TaskManager.Task::(_debug) Task 0935ea2a-fcc0-4136-8de5-d4288a1f0d92: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:22:23,623::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:22:23,624::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:22:23,624::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:22:23,625::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:22:23,625::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:22:23,626::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:22:23,627::task::495::TaskManager.Task::(_debug) Task 0935ea2a-fcc0-4136-8de5-d4288a1f0d92: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:22:23,627::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:22:23,951::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.084958 s, 12.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:22:25,963::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:22:26,499::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0854991 s, 12.0 MB/s\n'; <rc> = 0 >Thread-15914089::DEBUG::2013-11-20 03:22:27,649::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914089::INFO::2013-11-20 03:22:27,651::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914089::DEBUG::2013-11-20 03:22:27,652::task::495::TaskManager.Task::(_debug) Task e005abf1-8132-4733-b9e2-637264af1967: moving from state init -> state preparing >Thread-15914089::DEBUG::2013-11-20 03:22:27,653::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914089::DEBUG::2013-11-20 03:22:27,653::task::495::TaskManager.Task::(_debug) Task e005abf1-8132-4733-b9e2-637264af1967: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914089::DEBUG::2013-11-20 03:22:27,654::task::495::TaskManager.Task::(_debug) Task e005abf1-8132-4733-b9e2-637264af1967: moving from state preparing -> state finished >Thread-15914089::DEBUG::2013-11-20 03:22:27,655::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914089::DEBUG::2013-11-20 03:22:27,656::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914089::DEBUG::2013-11-20 03:22:27,656::task::495::TaskManager.Task::(_debug) Task e005abf1-8132-4733-b9e2-637264af1967: ref 0 aborting False >Thread-15914089::INFO::2013-11-20 03:22:27,657::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914090::DEBUG::2013-11-20 03:22:27,662::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914090::INFO::2013-11-20 03:22:27,663::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914090::DEBUG::2013-11-20 03:22:27,664::task::495::TaskManager.Task::(_debug) Task 2bb10fab-9d72-4f51-b0a4-a2019f26ec12: moving from state init -> state preparing >Thread-15914090::DEBUG::2013-11-20 03:22:27,665::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c5dd1723-3b93-4e32-b355-b900cbc49449`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914090::DEBUG::2013-11-20 03:22:27,665::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914090::DEBUG::2013-11-20 03:22:27,666::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914090::DEBUG::2013-11-20 03:22:27,667::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`c5dd1723-3b93-4e32-b355-b900cbc49449`::Granted request >Thread-15914090::DEBUG::2013-11-20 03:22:27,668::task::495::TaskManager.Task::(_debug) Task 2bb10fab-9d72-4f51-b0a4-a2019f26ec12: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914090::DEBUG::2013-11-20 03:22:27,669::task::495::TaskManager.Task::(_debug) Task 2bb10fab-9d72-4f51-b0a4-a2019f26ec12: ref 1 aborting False >Thread-15914090::WARNING::2013-11-20 03:22:27,671::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914090::DEBUG::2013-11-20 03:22:27,672::task::495::TaskManager.Task::(_debug) Task 2bb10fab-9d72-4f51-b0a4-a2019f26ec12: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914090::DEBUG::2013-11-20 03:22:27,673::task::495::TaskManager.Task::(_debug) Task 2bb10fab-9d72-4f51-b0a4-a2019f26ec12: moving from state preparing -> state finished >Thread-15914090::DEBUG::2013-11-20 03:22:27,674::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914090::DEBUG::2013-11-20 03:22:27,675::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914090::DEBUG::2013-11-20 03:22:27,675::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914090::DEBUG::2013-11-20 03:22:27,676::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914090::DEBUG::2013-11-20 03:22:27,676::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914090::DEBUG::2013-11-20 03:22:27,677::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914090::DEBUG::2013-11-20 03:22:27,678::task::495::TaskManager.Task::(_debug) Task 2bb10fab-9d72-4f51-b0a4-a2019f26ec12: ref 0 aborting False >Thread-15914090::INFO::2013-11-20 03:22:27,678::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:22:28,510::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:22:29,045::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0805154 s, 12.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:22:31,062::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:22:31,637::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0850937 s, 12.0 MB/s\n'; <rc> = 0 >Thread-15914092::INFO::2013-11-20 03:22:32,975::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914092::DEBUG::2013-11-20 03:22:32,976::task::495::TaskManager.Task::(_debug) Task 10477832-f20e-44f9-a6d9-ed08af94bb6c: moving from state init -> state preparing >Thread-15914092::DEBUG::2013-11-20 03:22:32,977::task::495::TaskManager.Task::(_debug) Task 10477832-f20e-44f9-a6d9-ed08af94bb6c: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000281095504761', 'lastCheck': 1384917750.190949, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139403343201', 'lastCheck': 1384917746.1893401, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0019690990448', 'lastCheck': 1384917748.734725, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00334692001343', 'lastCheck': 1384917748.727541, 'valid': True, 'code': 0}} >Thread-15914092::DEBUG::2013-11-20 03:22:32,986::task::495::TaskManager.Task::(_debug) Task 10477832-f20e-44f9-a6d9-ed08af94bb6c: moving from state preparing -> state finished >Thread-15914092::DEBUG::2013-11-20 03:22:32,987::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914092::DEBUG::2013-11-20 03:22:32,988::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914092::DEBUG::2013-11-20 03:22:32,989::task::495::TaskManager.Task::(_debug) Task 10477832-f20e-44f9-a6d9-ed08af94bb6c: ref 0 aborting False >Thread-15914092::INFO::2013-11-20 03:22:32,989::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000281095504761', 'lastCheck': 1384917750.190949, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00139403343201', 'lastCheck': 1384917746.1893401, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0019690990448', 'lastCheck': 1384917748.734725, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00334692001343', 'lastCheck': 1384917748.727541, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:22:33,656::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:22:34,217::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100393 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:22:36,228::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:22:36,807::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0866096 s, 11.8 MB/s\n'; <rc> = 0 >Thread-15914095::DEBUG::2013-11-20 03:22:38,485::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914095::INFO::2013-11-20 03:22:38,487::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914095::DEBUG::2013-11-20 03:22:38,488::task::495::TaskManager.Task::(_debug) Task 80588929-b78a-4ecc-a4cb-2cc10fbb6b88: moving from state init -> state preparing >Thread-15914095::DEBUG::2013-11-20 03:22:38,489::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914095::DEBUG::2013-11-20 03:22:38,489::task::495::TaskManager.Task::(_debug) Task 80588929-b78a-4ecc-a4cb-2cc10fbb6b88: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914095::DEBUG::2013-11-20 03:22:38,490::task::495::TaskManager.Task::(_debug) Task 80588929-b78a-4ecc-a4cb-2cc10fbb6b88: moving from state preparing -> state finished >Thread-15914095::DEBUG::2013-11-20 03:22:38,491::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914095::DEBUG::2013-11-20 03:22:38,491::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914095::DEBUG::2013-11-20 03:22:38,492::task::495::TaskManager.Task::(_debug) Task 80588929-b78a-4ecc-a4cb-2cc10fbb6b88: ref 0 aborting False >Thread-15914095::INFO::2013-11-20 03:22:38,493::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914096::DEBUG::2013-11-20 03:22:38,500::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914096::INFO::2013-11-20 03:22:38,501::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914096::DEBUG::2013-11-20 03:22:38,502::task::495::TaskManager.Task::(_debug) Task 26abd79e-266a-4017-b64e-e5c9728f0773: moving from state init -> state preparing >Thread-15914096::DEBUG::2013-11-20 03:22:38,508::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e39ff4d1-67af-4a6f-b73a-ec8fa75b6800`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914096::DEBUG::2013-11-20 03:22:38,508::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914096::DEBUG::2013-11-20 03:22:38,509::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914096::DEBUG::2013-11-20 03:22:38,510::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e39ff4d1-67af-4a6f-b73a-ec8fa75b6800`::Granted request >Thread-15914096::DEBUG::2013-11-20 03:22:38,511::task::495::TaskManager.Task::(_debug) Task 26abd79e-266a-4017-b64e-e5c9728f0773: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914096::DEBUG::2013-11-20 03:22:38,511::task::495::TaskManager.Task::(_debug) Task 26abd79e-266a-4017-b64e-e5c9728f0773: ref 1 aborting False >Thread-15914096::WARNING::2013-11-20 03:22:38,513::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914096::DEBUG::2013-11-20 03:22:38,514::task::495::TaskManager.Task::(_debug) Task 26abd79e-266a-4017-b64e-e5c9728f0773: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914096::DEBUG::2013-11-20 03:22:38,515::task::495::TaskManager.Task::(_debug) Task 26abd79e-266a-4017-b64e-e5c9728f0773: moving from state preparing -> state finished >Thread-15914096::DEBUG::2013-11-20 03:22:38,516::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914096::DEBUG::2013-11-20 03:22:38,517::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914096::DEBUG::2013-11-20 03:22:38,517::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914096::DEBUG::2013-11-20 03:22:38,518::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914096::DEBUG::2013-11-20 03:22:38,519::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914096::DEBUG::2013-11-20 03:22:38,519::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914096::DEBUG::2013-11-20 03:22:38,520::task::495::TaskManager.Task::(_debug) Task 26abd79e-266a-4017-b64e-e5c9728f0773: ref 0 aborting False >Thread-15914096::INFO::2013-11-20 03:22:38,521::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:22:38,822::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:22:39,354::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0754735 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:22:41,370::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913695::INFO::2013-11-20 03:22:41,598::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:22:41,599::task::495::TaskManager.Task::(_debug) Task b72a84a5-3dd5-4200-a7d1-c75054822d8d: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:22:41,603::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`79d2f120-6d45-441a-8e92-a6a5e06df341`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:22:41,604::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:22:41,605::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:22:41,606::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`79d2f120-6d45-441a-8e92-a6a5e06df341`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:22:41,608::task::495::TaskManager.Task::(_debug) Task b72a84a5-3dd5-4200-a7d1-c75054822d8d: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:22:41,609::task::495::TaskManager.Task::(_debug) Task b72a84a5-3dd5-4200-a7d1-c75054822d8d: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:22:41,610::task::495::TaskManager.Task::(_debug) Task b72a84a5-3dd5-4200-a7d1-c75054822d8d: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:22:41,611::task::495::TaskManager.Task::(_debug) Task b72a84a5-3dd5-4200-a7d1-c75054822d8d: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:22:41,611::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:22:41,612::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:22:41,613::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:22:41,613::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:22:41,614::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:22:41,614::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:22:41,615::task::495::TaskManager.Task::(_debug) Task b72a84a5-3dd5-4200-a7d1-c75054822d8d: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:22:41,615::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:22:41,954::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0766965 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:22:43,970::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913658::INFO::2013-11-20 03:22:44,157::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:22:44,160::task::495::TaskManager.Task::(_debug) Task a58bd9cd-70f3-40f5-b59c-054e43d28628: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:22:44,164::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`383a632d-ebc1-4d0e-90e6-96907c637e94`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:22:44,165::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:22:44,166::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:22:44,167::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`383a632d-ebc1-4d0e-90e6-96907c637e94`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:22:44,168::task::495::TaskManager.Task::(_debug) Task a58bd9cd-70f3-40f5-b59c-054e43d28628: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:22:44,169::task::495::TaskManager.Task::(_debug) Task a58bd9cd-70f3-40f5-b59c-054e43d28628: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:22:44,170::task::495::TaskManager.Task::(_debug) Task a58bd9cd-70f3-40f5-b59c-054e43d28628: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:22:44,170::task::495::TaskManager.Task::(_debug) Task a58bd9cd-70f3-40f5-b59c-054e43d28628: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:22:44,171::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:22:44,171::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:22:44,172::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:22:44,173::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:22:44,173::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:22:44,174::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:22:44,174::task::495::TaskManager.Task::(_debug) Task a58bd9cd-70f3-40f5-b59c-054e43d28628: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:22:44,175::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:22:44,498::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0856625 s, 12.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:22:46,514::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:22:47,094::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0944997 s, 10.8 MB/s\n'; <rc> = 0 >Thread-15914100::INFO::2013-11-20 03:22:48,922::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914100::DEBUG::2013-11-20 03:22:48,923::task::495::TaskManager.Task::(_debug) Task 21d4d327-fb24-43b9-9714-7276d103e164: moving from state init -> state preparing >Thread-15914100::DEBUG::2013-11-20 03:22:48,924::task::495::TaskManager.Task::(_debug) Task 21d4d327-fb24-43b9-9714-7276d103e164: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000268936157227', 'lastCheck': 1384917760.459975, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00128889083862', 'lastCheck': 1384917766.757694, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0487239360809', 'lastCheck': 1384917759.081192, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0020170211792', 'lastCheck': 1384917759.0255761, 'valid': True, 'code': 0}} >Thread-15914100::DEBUG::2013-11-20 03:22:48,935::task::495::TaskManager.Task::(_debug) Task 21d4d327-fb24-43b9-9714-7276d103e164: moving from state preparing -> state finished >Thread-15914100::DEBUG::2013-11-20 03:22:48,936::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914100::DEBUG::2013-11-20 03:22:48,936::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914100::DEBUG::2013-11-20 03:22:48,937::task::495::TaskManager.Task::(_debug) Task 21d4d327-fb24-43b9-9714-7276d103e164: ref 0 aborting False >Thread-15914100::INFO::2013-11-20 03:22:48,938::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000268936157227', 'lastCheck': 1384917760.459975, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00128889083862', 'lastCheck': 1384917766.757694, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0487239360809', 'lastCheck': 1384917759.081192, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0020170211792', 'lastCheck': 1384917759.0255761, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:22:49,104::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 03:22:49,374::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:22:49,380::task::495::TaskManager.Task::(_debug) Task 937a09a4-d980-4159-ad4e-87f0e3338552: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:22:49,382::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`bdcb0c0f-f824-4e8a-a118-b4d96bc62bab`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:22:49,383::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:22:49,385::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914102::DEBUG::2013-11-20 03:22:49,387::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15912392::DEBUG::2013-11-20 03:22:49,387::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`bdcb0c0f-f824-4e8a-a118-b4d96bc62bab`::Granted request >Thread-15914102::INFO::2013-11-20 03:22:49,389::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914102::DEBUG::2013-11-20 03:22:49,390::task::495::TaskManager.Task::(_debug) Task cdf0cc85-5b06-4247-95e8-7b39338761d2: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:22:49,391::task::495::TaskManager.Task::(_debug) Task 937a09a4-d980-4159-ad4e-87f0e3338552: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914102::DEBUG::2013-11-20 03:22:49,392::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15912392::DEBUG::2013-11-20 03:22:49,392::task::495::TaskManager.Task::(_debug) Task 937a09a4-d980-4159-ad4e-87f0e3338552: ref 1 aborting False >Thread-15914102::DEBUG::2013-11-20 03:22:49,393::task::495::TaskManager.Task::(_debug) Task cdf0cc85-5b06-4247-95e8-7b39338761d2: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912392::DEBUG::2013-11-20 03:22:49,394::task::495::TaskManager.Task::(_debug) Task 937a09a4-d980-4159-ad4e-87f0e3338552: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15914102::DEBUG::2013-11-20 03:22:49,395::task::495::TaskManager.Task::(_debug) Task cdf0cc85-5b06-4247-95e8-7b39338761d2: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:22:49,396::task::495::TaskManager.Task::(_debug) Task 937a09a4-d980-4159-ad4e-87f0e3338552: moving from state preparing -> state finished >Thread-15914102::DEBUG::2013-11-20 03:22:49,396::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15912392::DEBUG::2013-11-20 03:22:49,397::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914102::DEBUG::2013-11-20 03:22:49,398::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:22:49,398::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914102::DEBUG::2013-11-20 03:22:49,399::task::495::TaskManager.Task::(_debug) Task cdf0cc85-5b06-4247-95e8-7b39338761d2: ref 0 aborting False >Thread-15912392::DEBUG::2013-11-20 03:22:49,400::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914102::INFO::2013-11-20 03:22:49,400::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15912392::DEBUG::2013-11-20 03:22:49,401::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:22:49,402::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:22:49,403::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:22:49,405::task::495::TaskManager.Task::(_debug) Task 937a09a4-d980-4159-ad4e-87f0e3338552: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:22:49,405::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15914103::DEBUG::2013-11-20 03:22:49,410::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914103::INFO::2013-11-20 03:22:49,411::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914103::DEBUG::2013-11-20 03:22:49,412::task::495::TaskManager.Task::(_debug) Task 3cfa4cb6-7118-47a2-98e8-a6e7fe76301a: moving from state init -> state preparing >Thread-15914103::DEBUG::2013-11-20 03:22:49,415::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a9f4f14b-c2de-4075-b0a3-2a3c6a87d812`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914103::DEBUG::2013-11-20 03:22:49,416::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914103::DEBUG::2013-11-20 03:22:49,418::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914103::DEBUG::2013-11-20 03:22:49,418::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a9f4f14b-c2de-4075-b0a3-2a3c6a87d812`::Granted request >Thread-15914103::DEBUG::2013-11-20 03:22:49,419::task::495::TaskManager.Task::(_debug) Task 3cfa4cb6-7118-47a2-98e8-a6e7fe76301a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914103::DEBUG::2013-11-20 03:22:49,420::task::495::TaskManager.Task::(_debug) Task 3cfa4cb6-7118-47a2-98e8-a6e7fe76301a: ref 1 aborting False >Thread-15914103::WARNING::2013-11-20 03:22:49,422::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914103::DEBUG::2013-11-20 03:22:49,424::task::495::TaskManager.Task::(_debug) Task 3cfa4cb6-7118-47a2-98e8-a6e7fe76301a: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914103::DEBUG::2013-11-20 03:22:49,429::task::495::TaskManager.Task::(_debug) Task 3cfa4cb6-7118-47a2-98e8-a6e7fe76301a: moving from state preparing -> state finished >Thread-15914103::DEBUG::2013-11-20 03:22:49,430::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914103::DEBUG::2013-11-20 03:22:49,432::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914103::DEBUG::2013-11-20 03:22:49,433::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914103::DEBUG::2013-11-20 03:22:49,434::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914103::DEBUG::2013-11-20 03:22:49,435::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914103::DEBUG::2013-11-20 03:22:49,435::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914103::DEBUG::2013-11-20 03:22:49,436::task::495::TaskManager.Task::(_debug) Task 3cfa4cb6-7118-47a2-98e8-a6e7fe76301a: ref 0 aborting False >Thread-15914103::INFO::2013-11-20 03:22:49,437::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:22:49,645::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0723585 s, 14.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:22:51,658::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:22:52,247::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0860252 s, 11.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:22:54,264::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:22:54,829::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0848384 s, 12.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:22:56,845::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:22:57,362::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0810217 s, 12.6 MB/s\n'; <rc> = 0 >Thread-15913518::INFO::2013-11-20 03:22:58,001::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:22:58,002::task::495::TaskManager.Task::(_debug) Task 1470228f-3684-482c-8162-3d75b9df53d0: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:22:58,003::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d13b05e2-affd-4b4f-b383-8f8d5be1ce66`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:22:58,004::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:22:58,005::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:22:58,006::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d13b05e2-affd-4b4f-b383-8f8d5be1ce66`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:22:58,007::task::495::TaskManager.Task::(_debug) Task 1470228f-3684-482c-8162-3d75b9df53d0: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:22:58,008::task::495::TaskManager.Task::(_debug) Task 1470228f-3684-482c-8162-3d75b9df53d0: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:22:58,008::task::495::TaskManager.Task::(_debug) Task 1470228f-3684-482c-8162-3d75b9df53d0: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:22:58,009::task::495::TaskManager.Task::(_debug) Task 1470228f-3684-482c-8162-3d75b9df53d0: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:22:58,010::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:22:58,011::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:22:58,012::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:22:58,012::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:22:58,013::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:22:58,013::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:22:58,014::task::495::TaskManager.Task::(_debug) Task 1470228f-3684-482c-8162-3d75b9df53d0: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:22:58,015::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:22:58,016::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:22:58,016::task::495::TaskManager.Task::(_debug) Task 43335147-a59a-4d36-bfab-9b52dad991aa: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:22:58,017::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e17d7e12-c7a8-4609-97fb-ac5016a9b44f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:22:58,018::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:22:58,019::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:22:58,020::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e17d7e12-c7a8-4609-97fb-ac5016a9b44f`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:22:58,021::task::495::TaskManager.Task::(_debug) Task 43335147-a59a-4d36-bfab-9b52dad991aa: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:22:58,021::task::495::TaskManager.Task::(_debug) Task 43335147-a59a-4d36-bfab-9b52dad991aa: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:22:58,022::task::495::TaskManager.Task::(_debug) Task 43335147-a59a-4d36-bfab-9b52dad991aa: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:22:58,023::task::495::TaskManager.Task::(_debug) Task 43335147-a59a-4d36-bfab-9b52dad991aa: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:22:58,023::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:22:58,024::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:22:58,026::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:22:58,027::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:22:58,027::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:22:58,028::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:22:58,029::task::495::TaskManager.Task::(_debug) Task 43335147-a59a-4d36-bfab-9b52dad991aa: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:22:58,029::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:22:59,375::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:22:59,950::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0774626 s, 13.2 MB/s\n'; <rc> = 0 >Thread-15914107::DEBUG::2013-11-20 03:23:00,301::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914107::INFO::2013-11-20 03:23:00,303::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914107::DEBUG::2013-11-20 03:23:00,303::task::495::TaskManager.Task::(_debug) Task aac98686-0d20-48b7-ade8-b1c680c5f0eb: moving from state init -> state preparing >Thread-15914107::DEBUG::2013-11-20 03:23:00,304::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914107::DEBUG::2013-11-20 03:23:00,305::task::495::TaskManager.Task::(_debug) Task aac98686-0d20-48b7-ade8-b1c680c5f0eb: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914107::DEBUG::2013-11-20 03:23:00,306::task::495::TaskManager.Task::(_debug) Task aac98686-0d20-48b7-ade8-b1c680c5f0eb: moving from state preparing -> state finished >Thread-15914107::DEBUG::2013-11-20 03:23:00,306::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914107::DEBUG::2013-11-20 03:23:00,307::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914107::DEBUG::2013-11-20 03:23:00,307::task::495::TaskManager.Task::(_debug) Task aac98686-0d20-48b7-ade8-b1c680c5f0eb: ref 0 aborting False >Thread-15914107::INFO::2013-11-20 03:23:00,308::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914108::DEBUG::2013-11-20 03:23:00,320::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914108::INFO::2013-11-20 03:23:00,321::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914108::DEBUG::2013-11-20 03:23:00,322::task::495::TaskManager.Task::(_debug) Task 70281b5d-dd39-412e-b713-2da9b580e6b7: moving from state init -> state preparing >Thread-15914108::DEBUG::2013-11-20 03:23:00,323::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bcf8cc59-043d-4e92-868e-10f87562fe4a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914108::DEBUG::2013-11-20 03:23:00,324::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914108::DEBUG::2013-11-20 03:23:00,325::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914108::DEBUG::2013-11-20 03:23:00,326::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`bcf8cc59-043d-4e92-868e-10f87562fe4a`::Granted request >Thread-15914108::DEBUG::2013-11-20 03:23:00,332::task::495::TaskManager.Task::(_debug) Task 70281b5d-dd39-412e-b713-2da9b580e6b7: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914108::DEBUG::2013-11-20 03:23:00,333::task::495::TaskManager.Task::(_debug) Task 70281b5d-dd39-412e-b713-2da9b580e6b7: ref 1 aborting False >Thread-15914108::WARNING::2013-11-20 03:23:00,336::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914108::DEBUG::2013-11-20 03:23:00,337::task::495::TaskManager.Task::(_debug) Task 70281b5d-dd39-412e-b713-2da9b580e6b7: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914108::DEBUG::2013-11-20 03:23:00,338::task::495::TaskManager.Task::(_debug) Task 70281b5d-dd39-412e-b713-2da9b580e6b7: moving from state preparing -> state finished >Thread-15914108::DEBUG::2013-11-20 03:23:00,339::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914108::DEBUG::2013-11-20 03:23:00,340::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914108::DEBUG::2013-11-20 03:23:00,340::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914108::DEBUG::2013-11-20 03:23:00,341::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914108::DEBUG::2013-11-20 03:23:00,342::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914108::DEBUG::2013-11-20 03:23:00,343::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914108::DEBUG::2013-11-20 03:23:00,343::task::495::TaskManager.Task::(_debug) Task 70281b5d-dd39-412e-b713-2da9b580e6b7: ref 0 aborting False >Thread-15914108::INFO::2013-11-20 03:23:00,344::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:23:01,961::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:23:02,484::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0711486 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:23:04,500::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914110::INFO::2013-11-20 03:23:04,740::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914110::DEBUG::2013-11-20 03:23:04,742::task::495::TaskManager.Task::(_debug) Task 9d20c0f7-00ba-471c-9289-2051b5078dc6: moving from state init -> state preparing >Thread-15914110::DEBUG::2013-11-20 03:23:04,743::task::495::TaskManager.Task::(_debug) Task 9d20c0f7-00ba-471c-9289-2051b5078dc6: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000328063964844', 'lastCheck': 1384917781.03455, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00263500213623', 'lastCheck': 1384917777.039649, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00180292129517', 'lastCheck': 1384917779.6180689, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00208306312561', 'lastCheck': 1384917779.607614, 'valid': True, 'code': 0}} >Thread-15914110::DEBUG::2013-11-20 03:23:04,745::task::495::TaskManager.Task::(_debug) Task 9d20c0f7-00ba-471c-9289-2051b5078dc6: moving from state preparing -> state finished >Thread-15914110::DEBUG::2013-11-20 03:23:04,758::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914110::DEBUG::2013-11-20 03:23:04,759::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914110::DEBUG::2013-11-20 03:23:04,760::task::495::TaskManager.Task::(_debug) Task 9d20c0f7-00ba-471c-9289-2051b5078dc6: ref 0 aborting False >Thread-15914110::INFO::2013-11-20 03:23:04,761::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000328063964844', 'lastCheck': 1384917781.03455, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00263500213623', 'lastCheck': 1384917777.039649, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00180292129517', 'lastCheck': 1384917779.6180689, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00208306312561', 'lastCheck': 1384917779.607614, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:23:05,033::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0550246 s, 18.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:23:07,043::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:23:07,608::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0759564 s, 13.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:23:09,628::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:23:10,117::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0630827 s, 16.2 MB/s\n'; <rc> = 0 >Thread-15914112::DEBUG::2013-11-20 03:23:10,573::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914112::INFO::2013-11-20 03:23:10,574::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914112::DEBUG::2013-11-20 03:23:10,575::task::495::TaskManager.Task::(_debug) Task 0838bb28-f1fb-42c5-a09c-680cc1651140: moving from state init -> state preparing >Thread-15914112::DEBUG::2013-11-20 03:23:10,575::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914112::DEBUG::2013-11-20 03:23:10,576::task::495::TaskManager.Task::(_debug) Task 0838bb28-f1fb-42c5-a09c-680cc1651140: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914112::DEBUG::2013-11-20 03:23:10,576::task::495::TaskManager.Task::(_debug) Task 0838bb28-f1fb-42c5-a09c-680cc1651140: moving from state preparing -> state finished >Thread-15914112::DEBUG::2013-11-20 03:23:10,576::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914112::DEBUG::2013-11-20 03:23:10,577::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914112::DEBUG::2013-11-20 03:23:10,577::task::495::TaskManager.Task::(_debug) Task 0838bb28-f1fb-42c5-a09c-680cc1651140: ref 0 aborting False >Thread-15914112::INFO::2013-11-20 03:23:10,577::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914113::DEBUG::2013-11-20 03:23:10,585::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914113::INFO::2013-11-20 03:23:10,586::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914113::DEBUG::2013-11-20 03:23:10,586::task::495::TaskManager.Task::(_debug) Task dad24318-3776-4303-a4b3-ef520aed7b93: moving from state init -> state preparing >Thread-15914113::DEBUG::2013-11-20 03:23:10,587::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`320566f4-0e84-4ae6-a52c-9683003f49ca`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914113::DEBUG::2013-11-20 03:23:10,588::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914113::DEBUG::2013-11-20 03:23:10,588::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914113::DEBUG::2013-11-20 03:23:10,588::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`320566f4-0e84-4ae6-a52c-9683003f49ca`::Granted request >Thread-15914113::DEBUG::2013-11-20 03:23:10,589::task::495::TaskManager.Task::(_debug) Task dad24318-3776-4303-a4b3-ef520aed7b93: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914113::DEBUG::2013-11-20 03:23:10,590::task::495::TaskManager.Task::(_debug) Task dad24318-3776-4303-a4b3-ef520aed7b93: ref 1 aborting False >Thread-15914113::WARNING::2013-11-20 03:23:10,591::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914113::DEBUG::2013-11-20 03:23:10,592::task::495::TaskManager.Task::(_debug) Task dad24318-3776-4303-a4b3-ef520aed7b93: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914113::DEBUG::2013-11-20 03:23:10,592::task::495::TaskManager.Task::(_debug) Task dad24318-3776-4303-a4b3-ef520aed7b93: moving from state preparing -> state finished >Thread-15914113::DEBUG::2013-11-20 03:23:10,593::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914113::DEBUG::2013-11-20 03:23:10,593::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914113::DEBUG::2013-11-20 03:23:10,596::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914113::DEBUG::2013-11-20 03:23:10,597::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914113::DEBUG::2013-11-20 03:23:10,597::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914113::DEBUG::2013-11-20 03:23:10,597::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914113::DEBUG::2013-11-20 03:23:10,598::task::495::TaskManager.Task::(_debug) Task dad24318-3776-4303-a4b3-ef520aed7b93: ref 0 aborting False >Thread-15914113::INFO::2013-11-20 03:23:10,598::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:23:12,128::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:23:12,639::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0683614 s, 15.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:23:14,649::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913546::INFO::2013-11-20 03:23:14,875::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:23:14,879::task::495::TaskManager.Task::(_debug) Task 94b86a29-4c51-4d66-95d7-f183b760ac34: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:23:14,881::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`2bf086c8-ba5d-4b09-8e2c-01fffb9986aa`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:23:14,883::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:23:14,884::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:23:14,885::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`2bf086c8-ba5d-4b09-8e2c-01fffb9986aa`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:23:14,886::task::495::TaskManager.Task::(_debug) Task 94b86a29-4c51-4d66-95d7-f183b760ac34: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:23:14,887::task::495::TaskManager.Task::(_debug) Task 94b86a29-4c51-4d66-95d7-f183b760ac34: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:23:14,888::task::495::TaskManager.Task::(_debug) Task 94b86a29-4c51-4d66-95d7-f183b760ac34: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:23:14,889::task::495::TaskManager.Task::(_debug) Task 94b86a29-4c51-4d66-95d7-f183b760ac34: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:23:14,890::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:23:14,890::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:23:14,891::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:23:14,892::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:23:14,893::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:23:14,893::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:23:14,894::task::495::TaskManager.Task::(_debug) Task 94b86a29-4c51-4d66-95d7-f183b760ac34: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:23:14,895::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:23:14,896::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:23:14,896::task::495::TaskManager.Task::(_debug) Task c5160395-314d-41a7-ac06-cbe61feaa8d5: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:23:14,897::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`e08ab628-d2ac-4be8-aae1-d108292bcb08`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:23:14,898::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:23:14,899::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:23:14,899::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`e08ab628-d2ac-4be8-aae1-d108292bcb08`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:23:14,900::task::495::TaskManager.Task::(_debug) Task c5160395-314d-41a7-ac06-cbe61feaa8d5: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:23:14,901::task::495::TaskManager.Task::(_debug) Task c5160395-314d-41a7-ac06-cbe61feaa8d5: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:23:14,902::task::495::TaskManager.Task::(_debug) Task c5160395-314d-41a7-ac06-cbe61feaa8d5: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:23:14,902::task::495::TaskManager.Task::(_debug) Task c5160395-314d-41a7-ac06-cbe61feaa8d5: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:23:14,903::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:23:14,904::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:23:14,904::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:23:14,905::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:23:14,905::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:23:14,906::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:23:14,907::task::495::TaskManager.Task::(_debug) Task c5160395-314d-41a7-ac06-cbe61feaa8d5: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:23:14,907::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:23:14,908::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:23:14,909::task::495::TaskManager.Task::(_debug) Task 8619e949-c4f5-4dda-8f44-11e61bb19962: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:23:14,910::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`64bcb84b-6a72-4f16-81fe-0d592cf5a5d5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:23:14,911::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:23:14,911::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:23:14,912::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`64bcb84b-6a72-4f16-81fe-0d592cf5a5d5`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:23:14,913::task::495::TaskManager.Task::(_debug) Task 8619e949-c4f5-4dda-8f44-11e61bb19962: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:23:14,914::task::495::TaskManager.Task::(_debug) Task 8619e949-c4f5-4dda-8f44-11e61bb19962: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:23:14,914::task::495::TaskManager.Task::(_debug) Task 8619e949-c4f5-4dda-8f44-11e61bb19962: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:23:14,915::task::495::TaskManager.Task::(_debug) Task 8619e949-c4f5-4dda-8f44-11e61bb19962: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:23:14,915::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:23:14,916::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:23:14,917::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:23:14,917::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:23:14,918::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:23:14,919::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:23:14,919::task::495::TaskManager.Task::(_debug) Task 8619e949-c4f5-4dda-8f44-11e61bb19962: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:23:14,920::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:23:14,921::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:23:14,921::task::495::TaskManager.Task::(_debug) Task c579a0c1-4edd-4e3c-9c77-1975edecb363: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:23:14,922::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`a226ad1c-a3d8-4150-8030-0ac300c7ac3a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:23:14,923::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:23:14,924::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:23:14,924::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`a226ad1c-a3d8-4150-8030-0ac300c7ac3a`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:23:14,925::task::495::TaskManager.Task::(_debug) Task c579a0c1-4edd-4e3c-9c77-1975edecb363: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:23:14,926::task::495::TaskManager.Task::(_debug) Task c579a0c1-4edd-4e3c-9c77-1975edecb363: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:23:14,927::task::495::TaskManager.Task::(_debug) Task c579a0c1-4edd-4e3c-9c77-1975edecb363: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:23:14,927::task::495::TaskManager.Task::(_debug) Task c579a0c1-4edd-4e3c-9c77-1975edecb363: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:23:14,928::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:23:14,928::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:23:14,929::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:23:14,930::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:23:14,930::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:23:14,931::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:23:14,932::task::495::TaskManager.Task::(_debug) Task c579a0c1-4edd-4e3c-9c77-1975edecb363: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:23:14,932::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:23:15,138::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0699892 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:23:17,150::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:23:17,545::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.049138 s, 20.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:23:19,556::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:23:20,081::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0759965 s, 13.5 MB/s\n'; <rc> = 0 >Thread-15914116::DEBUG::2013-11-20 03:23:20,856::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914116::INFO::2013-11-20 03:23:20,857::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914116::DEBUG::2013-11-20 03:23:20,858::task::495::TaskManager.Task::(_debug) Task 574b867e-05bf-40b8-be78-a8f11d9cc9ce: moving from state init -> state preparing >Thread-15914116::DEBUG::2013-11-20 03:23:20,858::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914116::DEBUG::2013-11-20 03:23:20,859::task::495::TaskManager.Task::(_debug) Task 574b867e-05bf-40b8-be78-a8f11d9cc9ce: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914116::DEBUG::2013-11-20 03:23:20,859::task::495::TaskManager.Task::(_debug) Task 574b867e-05bf-40b8-be78-a8f11d9cc9ce: moving from state preparing -> state finished >Thread-15914116::DEBUG::2013-11-20 03:23:20,860::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914116::DEBUG::2013-11-20 03:23:20,860::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914116::DEBUG::2013-11-20 03:23:20,860::task::495::TaskManager.Task::(_debug) Task 574b867e-05bf-40b8-be78-a8f11d9cc9ce: ref 0 aborting False >Thread-15914116::INFO::2013-11-20 03:23:20,861::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914117::DEBUG::2013-11-20 03:23:20,876::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914117::INFO::2013-11-20 03:23:20,877::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914117::DEBUG::2013-11-20 03:23:20,877::task::495::TaskManager.Task::(_debug) Task 6bf59448-ceb5-48a6-b850-3cfd38867e77: moving from state init -> state preparing >Thread-15914117::DEBUG::2013-11-20 03:23:20,878::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`57f39d9c-02c4-49d7-a371-7104c7c4b46d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914117::DEBUG::2013-11-20 03:23:20,878::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914117::DEBUG::2013-11-20 03:23:20,879::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914117::DEBUG::2013-11-20 03:23:20,879::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`57f39d9c-02c4-49d7-a371-7104c7c4b46d`::Granted request >Thread-15914117::DEBUG::2013-11-20 03:23:20,886::task::495::TaskManager.Task::(_debug) Task 6bf59448-ceb5-48a6-b850-3cfd38867e77: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914117::DEBUG::2013-11-20 03:23:20,886::task::495::TaskManager.Task::(_debug) Task 6bf59448-ceb5-48a6-b850-3cfd38867e77: ref 1 aborting False >Thread-15914117::WARNING::2013-11-20 03:23:20,888::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914117::DEBUG::2013-11-20 03:23:20,889::task::495::TaskManager.Task::(_debug) Task 6bf59448-ceb5-48a6-b850-3cfd38867e77: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914117::DEBUG::2013-11-20 03:23:20,889::task::495::TaskManager.Task::(_debug) Task 6bf59448-ceb5-48a6-b850-3cfd38867e77: moving from state preparing -> state finished >Thread-15914117::DEBUG::2013-11-20 03:23:20,890::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914117::DEBUG::2013-11-20 03:23:20,890::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914117::DEBUG::2013-11-20 03:23:20,890::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914117::DEBUG::2013-11-20 03:23:20,891::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914117::DEBUG::2013-11-20 03:23:20,891::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914117::DEBUG::2013-11-20 03:23:20,891::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914117::DEBUG::2013-11-20 03:23:20,892::task::495::TaskManager.Task::(_debug) Task 6bf59448-ceb5-48a6-b850-3cfd38867e77: ref 0 aborting False >Thread-15914117::INFO::2013-11-20 03:23:20,892::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:23:22,096::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:23:22,547::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0644397 s, 15.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:23:24,562::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913639::INFO::2013-11-20 03:23:24,892::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:23:24,894::task::495::TaskManager.Task::(_debug) Task b7227519-5aad-4d8a-ba62-a8af5274d98f: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:23:24,898::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`8c881f39-b188-4876-b535-d29ec12e171e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:23:24,901::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:23:24,904::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:23:24,906::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`8c881f39-b188-4876-b535-d29ec12e171e`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:23:24,910::task::495::TaskManager.Task::(_debug) Task b7227519-5aad-4d8a-ba62-a8af5274d98f: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:23:24,911::task::495::TaskManager.Task::(_debug) Task b7227519-5aad-4d8a-ba62-a8af5274d98f: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:23:24,912::task::495::TaskManager.Task::(_debug) Task b7227519-5aad-4d8a-ba62-a8af5274d98f: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:23:24,913::task::495::TaskManager.Task::(_debug) Task b7227519-5aad-4d8a-ba62-a8af5274d98f: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:23:24,914::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:23:24,914::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:23:24,915::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:23:24,916::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:23:24,916::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:23:24,917::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:23:24,918::task::495::TaskManager.Task::(_debug) Task b7227519-5aad-4d8a-ba62-a8af5274d98f: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:23:24,918::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:23:25,069::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0846258 s, 12.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:23:27,085::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914120::INFO::2013-11-20 03:23:27,250::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914120::DEBUG::2013-11-20 03:23:27,251::task::495::TaskManager.Task::(_debug) Task c324019f-b4e8-4d53-aec6-d06baafd5fc9: moving from state init -> state preparing >Thread-15914120::DEBUG::2013-11-20 03:23:27,252::task::495::TaskManager.Task::(_debug) Task c324019f-b4e8-4d53-aec6-d06baafd5fc9: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000267028808594', 'lastCheck': 1384917801.3506579, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00118207931519', 'lastCheck': 1384917797.3470709, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00159692764282', 'lastCheck': 1384917799.905041, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000741958618164', 'lastCheck': 1384917799.883111, 'valid': True, 'code': 0}} >Thread-15914120::DEBUG::2013-11-20 03:23:27,253::task::495::TaskManager.Task::(_debug) Task c324019f-b4e8-4d53-aec6-d06baafd5fc9: moving from state preparing -> state finished >Thread-15914120::DEBUG::2013-11-20 03:23:27,253::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914120::DEBUG::2013-11-20 03:23:27,254::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914120::DEBUG::2013-11-20 03:23:27,254::task::495::TaskManager.Task::(_debug) Task c324019f-b4e8-4d53-aec6-d06baafd5fc9: ref 0 aborting False >Thread-15914120::INFO::2013-11-20 03:23:27,254::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000267028808594', 'lastCheck': 1384917801.3506579, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00118207931519', 'lastCheck': 1384917797.3470709, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00159692764282', 'lastCheck': 1384917799.905041, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000741958618164', 'lastCheck': 1384917799.883111, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:23:27,534::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0791832 s, 12.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:23:29,548::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:23:29,981::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0595611 s, 17.2 MB/s\n'; <rc> = 0 >Thread-15914123::DEBUG::2013-11-20 03:23:31,386::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914123::INFO::2013-11-20 03:23:31,387::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914123::DEBUG::2013-11-20 03:23:31,388::task::495::TaskManager.Task::(_debug) Task a455b583-832c-4acf-9114-bb2000ad2d96: moving from state init -> state preparing >Thread-15914123::DEBUG::2013-11-20 03:23:31,389::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914123::DEBUG::2013-11-20 03:23:31,389::task::495::TaskManager.Task::(_debug) Task a455b583-832c-4acf-9114-bb2000ad2d96: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914123::DEBUG::2013-11-20 03:23:31,390::task::495::TaskManager.Task::(_debug) Task a455b583-832c-4acf-9114-bb2000ad2d96: moving from state preparing -> state finished >Thread-15914123::DEBUG::2013-11-20 03:23:31,391::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914123::DEBUG::2013-11-20 03:23:31,391::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914123::DEBUG::2013-11-20 03:23:31,392::task::495::TaskManager.Task::(_debug) Task a455b583-832c-4acf-9114-bb2000ad2d96: ref 0 aborting False >Thread-15914123::INFO::2013-11-20 03:23:31,392::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914124::DEBUG::2013-11-20 03:23:31,397::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914124::INFO::2013-11-20 03:23:31,398::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914124::DEBUG::2013-11-20 03:23:31,398::task::495::TaskManager.Task::(_debug) Task 59a06318-d81a-4c7c-b4f7-3b0d1dd3d429: moving from state init -> state preparing >Thread-15914124::DEBUG::2013-11-20 03:23:31,400::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`69c29eae-1f96-4795-a16c-254f36b64117`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914124::DEBUG::2013-11-20 03:23:31,401::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914124::DEBUG::2013-11-20 03:23:31,401::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914124::DEBUG::2013-11-20 03:23:31,402::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`69c29eae-1f96-4795-a16c-254f36b64117`::Granted request >Thread-15914124::DEBUG::2013-11-20 03:23:31,403::task::495::TaskManager.Task::(_debug) Task 59a06318-d81a-4c7c-b4f7-3b0d1dd3d429: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914124::DEBUG::2013-11-20 03:23:31,404::task::495::TaskManager.Task::(_debug) Task 59a06318-d81a-4c7c-b4f7-3b0d1dd3d429: ref 1 aborting False >Thread-15914124::WARNING::2013-11-20 03:23:31,406::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914124::DEBUG::2013-11-20 03:23:31,410::task::495::TaskManager.Task::(_debug) Task 59a06318-d81a-4c7c-b4f7-3b0d1dd3d429: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914124::DEBUG::2013-11-20 03:23:31,411::task::495::TaskManager.Task::(_debug) Task 59a06318-d81a-4c7c-b4f7-3b0d1dd3d429: moving from state preparing -> state finished >Thread-15914124::DEBUG::2013-11-20 03:23:31,412::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914124::DEBUG::2013-11-20 03:23:31,412::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914124::DEBUG::2013-11-20 03:23:31,413::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914124::DEBUG::2013-11-20 03:23:31,413::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914124::DEBUG::2013-11-20 03:23:31,414::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914124::DEBUG::2013-11-20 03:23:31,415::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914124::DEBUG::2013-11-20 03:23:31,415::task::495::TaskManager.Task::(_debug) Task 59a06318-d81a-4c7c-b4f7-3b0d1dd3d429: ref 0 aborting False >Thread-15914124::INFO::2013-11-20 03:23:31,416::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:23:31,992::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:23:32,471::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0562038 s, 18.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:23:34,481::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:23:35,044::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0917103 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:23:37,059::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:23:37,642::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.10011 s, 10.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:23:39,658::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:23:40,205::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0811716 s, 12.6 MB/s\n'; <rc> = 0 >Thread-15914128::DEBUG::2013-11-20 03:23:42,013::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914128::INFO::2013-11-20 03:23:42,015::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914128::DEBUG::2013-11-20 03:23:42,015::task::495::TaskManager.Task::(_debug) Task b9c97805-ad51-4727-bce9-d93b47e31d77: moving from state init -> state preparing >Thread-15914128::DEBUG::2013-11-20 03:23:42,016::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914128::DEBUG::2013-11-20 03:23:42,017::task::495::TaskManager.Task::(_debug) Task b9c97805-ad51-4727-bce9-d93b47e31d77: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914128::DEBUG::2013-11-20 03:23:42,017::task::495::TaskManager.Task::(_debug) Task b9c97805-ad51-4727-bce9-d93b47e31d77: moving from state preparing -> state finished >Thread-15914128::DEBUG::2013-11-20 03:23:42,018::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914128::DEBUG::2013-11-20 03:23:42,019::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914128::DEBUG::2013-11-20 03:23:42,019::task::495::TaskManager.Task::(_debug) Task b9c97805-ad51-4727-bce9-d93b47e31d77: ref 0 aborting False >Thread-15914128::INFO::2013-11-20 03:23:42,020::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914129::DEBUG::2013-11-20 03:23:42,027::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914129::INFO::2013-11-20 03:23:42,028::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914129::DEBUG::2013-11-20 03:23:42,029::task::495::TaskManager.Task::(_debug) Task fdafb356-ef9d-46d8-9818-901ccd46cdeb: moving from state init -> state preparing >Thread-15914129::DEBUG::2013-11-20 03:23:42,030::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6c898de0-88c5-465c-95d6-a6ffa113ef43`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914129::DEBUG::2013-11-20 03:23:42,031::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914129::DEBUG::2013-11-20 03:23:42,032::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914129::DEBUG::2013-11-20 03:23:42,033::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6c898de0-88c5-465c-95d6-a6ffa113ef43`::Granted request >Thread-15914129::DEBUG::2013-11-20 03:23:42,034::task::495::TaskManager.Task::(_debug) Task fdafb356-ef9d-46d8-9818-901ccd46cdeb: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914129::DEBUG::2013-11-20 03:23:42,035::task::495::TaskManager.Task::(_debug) Task fdafb356-ef9d-46d8-9818-901ccd46cdeb: ref 1 aborting False >Thread-15914129::WARNING::2013-11-20 03:23:42,038::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914129::DEBUG::2013-11-20 03:23:42,039::task::495::TaskManager.Task::(_debug) Task fdafb356-ef9d-46d8-9818-901ccd46cdeb: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914129::DEBUG::2013-11-20 03:23:42,040::task::495::TaskManager.Task::(_debug) Task fdafb356-ef9d-46d8-9818-901ccd46cdeb: moving from state preparing -> state finished >Thread-15914129::DEBUG::2013-11-20 03:23:42,040::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914129::DEBUG::2013-11-20 03:23:42,041::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914129::DEBUG::2013-11-20 03:23:42,042::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914129::DEBUG::2013-11-20 03:23:42,042::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914129::DEBUG::2013-11-20 03:23:42,043::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914129::DEBUG::2013-11-20 03:23:42,044::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914129::DEBUG::2013-11-20 03:23:42,044::task::495::TaskManager.Task::(_debug) Task fdafb356-ef9d-46d8-9818-901ccd46cdeb: ref 0 aborting False >Thread-15914129::INFO::2013-11-20 03:23:42,045::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:23:42,220::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913695::INFO::2013-11-20 03:23:42,463::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:23:42,464::task::495::TaskManager.Task::(_debug) Task bf5e7055-8096-482f-abde-95c4ef671803: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:23:42,465::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`39ea8d71-2bf5-4be3-9917-5ffe5e8158e6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:23:42,466::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:23:42,467::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:23:42,467::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`39ea8d71-2bf5-4be3-9917-5ffe5e8158e6`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:23:42,468::task::495::TaskManager.Task::(_debug) Task bf5e7055-8096-482f-abde-95c4ef671803: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:23:42,468::task::495::TaskManager.Task::(_debug) Task bf5e7055-8096-482f-abde-95c4ef671803: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:23:42,469::task::495::TaskManager.Task::(_debug) Task bf5e7055-8096-482f-abde-95c4ef671803: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:23:42,469::task::495::TaskManager.Task::(_debug) Task bf5e7055-8096-482f-abde-95c4ef671803: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:23:42,470::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:23:42,470::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:23:42,471::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:23:42,471::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:23:42,471::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:23:42,472::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:23:42,472::task::495::TaskManager.Task::(_debug) Task bf5e7055-8096-482f-abde-95c4ef671803: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:23:42,472::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:23:42,646::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0462369 s, 22.1 MB/s\n'; <rc> = 0 >Thread-15914130::INFO::2013-11-20 03:23:42,985::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914130::DEBUG::2013-11-20 03:23:42,986::task::495::TaskManager.Task::(_debug) Task 470cde72-2fba-4805-b411-4d61f2c96e30: moving from state init -> state preparing >Thread-15914130::DEBUG::2013-11-20 03:23:42,987::task::495::TaskManager.Task::(_debug) Task 470cde72-2fba-4805-b411-4d61f2c96e30: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000257015228271', 'lastCheck': 1384917821.3769021, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00767993927002', 'lastCheck': 1384917817.379355, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000957012176514', 'lastCheck': 1384917819.934777, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00175595283508', 'lastCheck': 1384917819.908145, 'valid': True, 'code': 0}} >Thread-15914130::DEBUG::2013-11-20 03:23:42,988::task::495::TaskManager.Task::(_debug) Task 470cde72-2fba-4805-b411-4d61f2c96e30: moving from state preparing -> state finished >Thread-15914130::DEBUG::2013-11-20 03:23:42,988::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914130::DEBUG::2013-11-20 03:23:42,989::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914130::DEBUG::2013-11-20 03:23:42,989::task::495::TaskManager.Task::(_debug) Task 470cde72-2fba-4805-b411-4d61f2c96e30: ref 0 aborting False >Thread-15914130::INFO::2013-11-20 03:23:42,990::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000257015228271', 'lastCheck': 1384917821.3769021, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00767993927002', 'lastCheck': 1384917817.379355, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000957012176514', 'lastCheck': 1384917819.934777, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00175595283508', 'lastCheck': 1384917819.908145, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:23:44,656::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913658::INFO::2013-11-20 03:23:44,916::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:23:44,917::task::495::TaskManager.Task::(_debug) Task 099b39d5-86d6-49bc-ba86-7884c1b038f5: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:23:44,919::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0bd2e0d5-ffaf-4f9d-860f-3ae17d8d1a7a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:23:44,920::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:23:44,921::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:23:44,922::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0bd2e0d5-ffaf-4f9d-860f-3ae17d8d1a7a`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:23:44,923::task::495::TaskManager.Task::(_debug) Task 099b39d5-86d6-49bc-ba86-7884c1b038f5: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:23:44,924::task::495::TaskManager.Task::(_debug) Task 099b39d5-86d6-49bc-ba86-7884c1b038f5: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:23:44,924::task::495::TaskManager.Task::(_debug) Task 099b39d5-86d6-49bc-ba86-7884c1b038f5: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:23:44,925::task::495::TaskManager.Task::(_debug) Task 099b39d5-86d6-49bc-ba86-7884c1b038f5: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:23:44,926::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:23:44,926::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:23:44,927::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:23:44,928::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:23:44,928::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:23:44,929::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:23:44,929::task::495::TaskManager.Task::(_debug) Task 099b39d5-86d6-49bc-ba86-7884c1b038f5: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:23:44,930::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:23:45,149::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0771664 s, 13.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:23:47,163::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:23:47,690::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.077777 s, 13.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:23:49,704::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15912392::INFO::2013-11-20 03:23:49,924::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:23:49,928::task::495::TaskManager.Task::(_debug) Task efed7532-8aa6-4f78-986c-79d63d03db9f: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:23:49,932::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`3d3c8c65-133b-4823-89e5-e3dae33d3bd2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:23:49,933::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:23:49,934::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:23:49,934::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`3d3c8c65-133b-4823-89e5-e3dae33d3bd2`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:23:49,936::task::495::TaskManager.Task::(_debug) Task efed7532-8aa6-4f78-986c-79d63d03db9f: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:23:49,936::task::495::TaskManager.Task::(_debug) Task efed7532-8aa6-4f78-986c-79d63d03db9f: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:23:49,937::task::495::TaskManager.Task::(_debug) Task efed7532-8aa6-4f78-986c-79d63d03db9f: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:23:49,937::task::495::TaskManager.Task::(_debug) Task efed7532-8aa6-4f78-986c-79d63d03db9f: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:23:49,938::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:23:49,939::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:23:49,939::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:23:49,940::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:23:49,940::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:23:49,941::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:23:49,943::task::495::TaskManager.Task::(_debug) Task efed7532-8aa6-4f78-986c-79d63d03db9f: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:23:49,945::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:23:50,226::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0830789 s, 12.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:23:52,241::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914134::DEBUG::2013-11-20 03:23:52,491::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914134::INFO::2013-11-20 03:23:52,493::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914134::DEBUG::2013-11-20 03:23:52,493::task::495::TaskManager.Task::(_debug) Task 16ce2804-63c4-45e5-99ac-ab7bd72ea92d: moving from state init -> state preparing >Thread-15914134::DEBUG::2013-11-20 03:23:52,494::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914134::DEBUG::2013-11-20 03:23:52,495::task::495::TaskManager.Task::(_debug) Task 16ce2804-63c4-45e5-99ac-ab7bd72ea92d: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914134::DEBUG::2013-11-20 03:23:52,495::task::495::TaskManager.Task::(_debug) Task 16ce2804-63c4-45e5-99ac-ab7bd72ea92d: moving from state preparing -> state finished >Thread-15914134::DEBUG::2013-11-20 03:23:52,496::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914134::DEBUG::2013-11-20 03:23:52,496::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914134::DEBUG::2013-11-20 03:23:52,497::task::495::TaskManager.Task::(_debug) Task 16ce2804-63c4-45e5-99ac-ab7bd72ea92d: ref 0 aborting False >Thread-15914134::INFO::2013-11-20 03:23:52,498::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914135::DEBUG::2013-11-20 03:23:52,502::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914135::INFO::2013-11-20 03:23:52,503::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914135::DEBUG::2013-11-20 03:23:52,504::task::495::TaskManager.Task::(_debug) Task 9a31086f-afee-4515-b2e8-760817bdfbb5: moving from state init -> state preparing >Thread-15914135::DEBUG::2013-11-20 03:23:52,505::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9c7691b1-4cf9-47b7-87a8-64a70a4c7f3a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914135::DEBUG::2013-11-20 03:23:52,505::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914135::DEBUG::2013-11-20 03:23:52,506::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914135::DEBUG::2013-11-20 03:23:52,506::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`9c7691b1-4cf9-47b7-87a8-64a70a4c7f3a`::Granted request >Thread-15914135::DEBUG::2013-11-20 03:23:52,507::task::495::TaskManager.Task::(_debug) Task 9a31086f-afee-4515-b2e8-760817bdfbb5: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914135::DEBUG::2013-11-20 03:23:52,507::task::495::TaskManager.Task::(_debug) Task 9a31086f-afee-4515-b2e8-760817bdfbb5: ref 1 aborting False >Thread-15914135::WARNING::2013-11-20 03:23:52,509::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914135::DEBUG::2013-11-20 03:23:52,509::task::495::TaskManager.Task::(_debug) Task 9a31086f-afee-4515-b2e8-760817bdfbb5: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914135::DEBUG::2013-11-20 03:23:52,510::task::495::TaskManager.Task::(_debug) Task 9a31086f-afee-4515-b2e8-760817bdfbb5: moving from state preparing -> state finished >Thread-15914135::DEBUG::2013-11-20 03:23:52,510::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914135::DEBUG::2013-11-20 03:23:52,510::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914135::DEBUG::2013-11-20 03:23:52,511::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914135::DEBUG::2013-11-20 03:23:52,511::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914135::DEBUG::2013-11-20 03:23:52,512::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914135::DEBUG::2013-11-20 03:23:52,512::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914135::DEBUG::2013-11-20 03:23:52,512::task::495::TaskManager.Task::(_debug) Task 9a31086f-afee-4515-b2e8-760817bdfbb5: ref 0 aborting False >Thread-15914135::INFO::2013-11-20 03:23:52,513::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:23:52,689::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0611704 s, 16.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:23:54,701::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:23:55,138::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.050262 s, 20.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:23:57,148::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:23:57,579::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0522138 s, 19.6 MB/s\n'; <rc> = 0 >Thread-15914138::INFO::2013-11-20 03:23:58,570::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914138::DEBUG::2013-11-20 03:23:58,571::task::495::TaskManager.Task::(_debug) Task eccb5990-3844-464e-b56e-f52905ad0f62: moving from state init -> state preparing >Thread-15914138::DEBUG::2013-11-20 03:23:58,572::task::495::TaskManager.Task::(_debug) Task eccb5990-3844-464e-b56e-f52905ad0f62: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000241994857788', 'lastCheck': 1384917831.4261451, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00191593170166', 'lastCheck': 1384917837.4539461, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00144600868225', 'lastCheck': 1384917829.949064, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00685596466064', 'lastCheck': 1384917829.928618, 'valid': True, 'code': 0}} >Thread-15914138::DEBUG::2013-11-20 03:23:58,572::task::495::TaskManager.Task::(_debug) Task eccb5990-3844-464e-b56e-f52905ad0f62: moving from state preparing -> state finished >Thread-15914138::DEBUG::2013-11-20 03:23:58,573::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914138::DEBUG::2013-11-20 03:23:58,574::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914138::DEBUG::2013-11-20 03:23:58,574::task::495::TaskManager.Task::(_debug) Task eccb5990-3844-464e-b56e-f52905ad0f62: ref 0 aborting False >Thread-15914138::INFO::2013-11-20 03:23:58,575::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000241994857788', 'lastCheck': 1384917831.4261451, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00191593170166', 'lastCheck': 1384917837.4539461, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00144600868225', 'lastCheck': 1384917829.949064, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00685596466064', 'lastCheck': 1384917829.928618, 'valid': True, 'code': 0}} >Thread-15913518::INFO::2013-11-20 03:23:58,991::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:23:58,992::task::495::TaskManager.Task::(_debug) Task 0c58d459-8815-4e03-baef-3526d9041e52: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:23:58,993::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`2110a916-4617-403a-b87b-1685900863f1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:23:58,994::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:23:58,994::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:23:58,994::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`2110a916-4617-403a-b87b-1685900863f1`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:23:58,995::task::495::TaskManager.Task::(_debug) Task 0c58d459-8815-4e03-baef-3526d9041e52: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:23:58,996::task::495::TaskManager.Task::(_debug) Task 0c58d459-8815-4e03-baef-3526d9041e52: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:23:58,996::task::495::TaskManager.Task::(_debug) Task 0c58d459-8815-4e03-baef-3526d9041e52: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:23:58,996::task::495::TaskManager.Task::(_debug) Task 0c58d459-8815-4e03-baef-3526d9041e52: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:23:58,997::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:23:58,997::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:23:58,998::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:23:58,998::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:23:58,998::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:23:58,999::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:23:58,999::task::495::TaskManager.Task::(_debug) Task 0c58d459-8815-4e03-baef-3526d9041e52: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:23:58,999::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:23:59,000::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:23:59,000::task::495::TaskManager.Task::(_debug) Task 18ea06da-a53c-4ad7-be89-38d5b98993cc: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:23:59,001::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a7c6fb6a-d818-4c0c-8a9a-6f0970d79d6f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:23:59,001::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:23:59,002::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:23:59,002::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a7c6fb6a-d818-4c0c-8a9a-6f0970d79d6f`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:23:59,005::task::495::TaskManager.Task::(_debug) Task 18ea06da-a53c-4ad7-be89-38d5b98993cc: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:23:59,006::task::495::TaskManager.Task::(_debug) Task 18ea06da-a53c-4ad7-be89-38d5b98993cc: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:23:59,006::task::495::TaskManager.Task::(_debug) Task 18ea06da-a53c-4ad7-be89-38d5b98993cc: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:23:59,007::task::495::TaskManager.Task::(_debug) Task 18ea06da-a53c-4ad7-be89-38d5b98993cc: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:23:59,007::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:23:59,007::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:23:59,008::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:23:59,008::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:23:59,008::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:23:59,009::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:23:59,009::task::495::TaskManager.Task::(_debug) Task 18ea06da-a53c-4ad7-be89-38d5b98993cc: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:23:59,010::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913547::DEBUG::2013-11-20 03:23:59,579::guestIF::97::vm.Vm::(_handleMessage) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::u'BRD-KLEE\\administrator' >Dummy-15912651::DEBUG::2013-11-20 03:23:59,588::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:24:00,141::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0858754 s, 11.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:24:02,158::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:24:02,674::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0806518 s, 12.7 MB/s\n'; <rc> = 0 >Thread-15914141::DEBUG::2013-11-20 03:24:02,711::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914141::INFO::2013-11-20 03:24:02,713::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914141::DEBUG::2013-11-20 03:24:02,714::task::495::TaskManager.Task::(_debug) Task 8eb97f56-75e9-4a66-8920-5ce3f6a6d9f4: moving from state init -> state preparing >Thread-15914141::DEBUG::2013-11-20 03:24:02,715::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914141::DEBUG::2013-11-20 03:24:02,716::task::495::TaskManager.Task::(_debug) Task 8eb97f56-75e9-4a66-8920-5ce3f6a6d9f4: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914141::DEBUG::2013-11-20 03:24:02,717::task::495::TaskManager.Task::(_debug) Task 8eb97f56-75e9-4a66-8920-5ce3f6a6d9f4: moving from state preparing -> state finished >Thread-15914141::DEBUG::2013-11-20 03:24:02,717::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914141::DEBUG::2013-11-20 03:24:02,718::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914141::DEBUG::2013-11-20 03:24:02,718::task::495::TaskManager.Task::(_debug) Task 8eb97f56-75e9-4a66-8920-5ce3f6a6d9f4: ref 0 aborting False >Thread-15914141::INFO::2013-11-20 03:24:02,719::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914142::DEBUG::2013-11-20 03:24:02,724::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914142::INFO::2013-11-20 03:24:02,725::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914142::DEBUG::2013-11-20 03:24:02,726::task::495::TaskManager.Task::(_debug) Task 17233ea1-ddda-4e49-9935-6c38d4657de0: moving from state init -> state preparing >Thread-15914142::DEBUG::2013-11-20 03:24:02,727::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`54fe37ee-ee2b-4041-8733-f85c3a320a61`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914142::DEBUG::2013-11-20 03:24:02,728::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914142::DEBUG::2013-11-20 03:24:02,729::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914142::DEBUG::2013-11-20 03:24:02,729::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`54fe37ee-ee2b-4041-8733-f85c3a320a61`::Granted request >Thread-15914142::DEBUG::2013-11-20 03:24:02,731::task::495::TaskManager.Task::(_debug) Task 17233ea1-ddda-4e49-9935-6c38d4657de0: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914142::DEBUG::2013-11-20 03:24:02,731::task::495::TaskManager.Task::(_debug) Task 17233ea1-ddda-4e49-9935-6c38d4657de0: ref 1 aborting False >Thread-15914142::WARNING::2013-11-20 03:24:02,734::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914142::DEBUG::2013-11-20 03:24:02,735::task::495::TaskManager.Task::(_debug) Task 17233ea1-ddda-4e49-9935-6c38d4657de0: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914142::DEBUG::2013-11-20 03:24:02,736::task::495::TaskManager.Task::(_debug) Task 17233ea1-ddda-4e49-9935-6c38d4657de0: moving from state preparing -> state finished >Thread-15914142::DEBUG::2013-11-20 03:24:02,736::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914142::DEBUG::2013-11-20 03:24:02,737::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914142::DEBUG::2013-11-20 03:24:02,737::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914142::DEBUG::2013-11-20 03:24:02,738::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914142::DEBUG::2013-11-20 03:24:02,739::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914142::DEBUG::2013-11-20 03:24:02,739::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914142::DEBUG::2013-11-20 03:24:02,740::task::495::TaskManager.Task::(_debug) Task 17233ea1-ddda-4e49-9935-6c38d4657de0: ref 0 aborting False >Thread-15914142::INFO::2013-11-20 03:24:02,741::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:24:04,690::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:24:05,115::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0589821 s, 17.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:24:07,136::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:24:07,660::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0784508 s, 13.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:24:09,676::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:24:10,192::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0790742 s, 12.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:24:12,208::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:24:12,701::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0883056 s, 11.6 MB/s\n'; <rc> = 0 >Thread-15914146::DEBUG::2013-11-20 03:24:12,976::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914146::INFO::2013-11-20 03:24:12,978::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914146::DEBUG::2013-11-20 03:24:12,978::task::495::TaskManager.Task::(_debug) Task 5b0b0838-2208-485d-b9ae-507fd5c0929c: moving from state init -> state preparing >Thread-15914146::DEBUG::2013-11-20 03:24:12,979::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914146::DEBUG::2013-11-20 03:24:12,979::task::495::TaskManager.Task::(_debug) Task 5b0b0838-2208-485d-b9ae-507fd5c0929c: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914146::DEBUG::2013-11-20 03:24:12,980::task::495::TaskManager.Task::(_debug) Task 5b0b0838-2208-485d-b9ae-507fd5c0929c: moving from state preparing -> state finished >Thread-15914146::DEBUG::2013-11-20 03:24:12,980::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914146::DEBUG::2013-11-20 03:24:12,980::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914146::DEBUG::2013-11-20 03:24:12,981::task::495::TaskManager.Task::(_debug) Task 5b0b0838-2208-485d-b9ae-507fd5c0929c: ref 0 aborting False >Thread-15914146::INFO::2013-11-20 03:24:12,981::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914147::DEBUG::2013-11-20 03:24:12,985::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914147::INFO::2013-11-20 03:24:12,985::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914147::DEBUG::2013-11-20 03:24:12,986::task::495::TaskManager.Task::(_debug) Task 8508f497-f2b2-4195-8573-f2294893c645: moving from state init -> state preparing >Thread-15914147::DEBUG::2013-11-20 03:24:12,986::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2027148c-98e6-4cdb-92f9-32bfb9c8f9d0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914147::DEBUG::2013-11-20 03:24:12,987::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914147::DEBUG::2013-11-20 03:24:12,987::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914147::DEBUG::2013-11-20 03:24:12,988::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2027148c-98e6-4cdb-92f9-32bfb9c8f9d0`::Granted request >Thread-15914147::DEBUG::2013-11-20 03:24:12,988::task::495::TaskManager.Task::(_debug) Task 8508f497-f2b2-4195-8573-f2294893c645: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914147::DEBUG::2013-11-20 03:24:12,989::task::495::TaskManager.Task::(_debug) Task 8508f497-f2b2-4195-8573-f2294893c645: ref 1 aborting False >Thread-15914147::WARNING::2013-11-20 03:24:12,990::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914147::DEBUG::2013-11-20 03:24:12,990::task::495::TaskManager.Task::(_debug) Task 8508f497-f2b2-4195-8573-f2294893c645: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914147::DEBUG::2013-11-20 03:24:12,991::task::495::TaskManager.Task::(_debug) Task 8508f497-f2b2-4195-8573-f2294893c645: moving from state preparing -> state finished >Thread-15914147::DEBUG::2013-11-20 03:24:12,991::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914147::DEBUG::2013-11-20 03:24:12,992::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914147::DEBUG::2013-11-20 03:24:12,992::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914147::DEBUG::2013-11-20 03:24:12,992::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914147::DEBUG::2013-11-20 03:24:12,993::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914147::DEBUG::2013-11-20 03:24:12,993::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914147::DEBUG::2013-11-20 03:24:12,993::task::495::TaskManager.Task::(_debug) Task 8508f497-f2b2-4195-8573-f2294893c645: ref 0 aborting False >Thread-15914147::INFO::2013-11-20 03:24:12,994::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914148::INFO::2013-11-20 03:24:13,987::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914148::DEBUG::2013-11-20 03:24:13,987::task::495::TaskManager.Task::(_debug) Task 4a121574-5c90-4143-8a51-7aff0353aa47: moving from state init -> state preparing >Thread-15914148::DEBUG::2013-11-20 03:24:13,988::task::495::TaskManager.Task::(_debug) Task 4a121574-5c90-4143-8a51-7aff0353aa47: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000250101089478', 'lastCheck': 1384917851.4937401, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0228769779205', 'lastCheck': 1384917847.521054, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00160479545593', 'lastCheck': 1384917850.0028369, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000396966934204', 'lastCheck': 1384917849.9499509, 'valid': True, 'code': 0}} >Thread-15914148::DEBUG::2013-11-20 03:24:13,988::task::495::TaskManager.Task::(_debug) Task 4a121574-5c90-4143-8a51-7aff0353aa47: moving from state preparing -> state finished >Thread-15914148::DEBUG::2013-11-20 03:24:13,988::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914148::DEBUG::2013-11-20 03:24:13,989::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914148::DEBUG::2013-11-20 03:24:13,989::task::495::TaskManager.Task::(_debug) Task 4a121574-5c90-4143-8a51-7aff0353aa47: ref 0 aborting False >Thread-15914148::INFO::2013-11-20 03:24:13,990::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000250101089478', 'lastCheck': 1384917851.4937401, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0228769779205', 'lastCheck': 1384917847.521054, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00160479545593', 'lastCheck': 1384917850.0028369, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000396966934204', 'lastCheck': 1384917849.9499509, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:24:14,716::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913546::INFO::2013-11-20 03:24:15,037::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:24:15,039::task::495::TaskManager.Task::(_debug) Task 03173789-5b61-4fd4-bb6c-6bee3f0ffb00: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:24:15,041::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`8a87f5a8-3820-4c23-ba08-58a9bb0c7987`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:24:15,042::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:24:15,044::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:24:15,045::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`8a87f5a8-3820-4c23-ba08-58a9bb0c7987`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:24:15,048::task::495::TaskManager.Task::(_debug) Task 03173789-5b61-4fd4-bb6c-6bee3f0ffb00: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:24:15,049::task::495::TaskManager.Task::(_debug) Task 03173789-5b61-4fd4-bb6c-6bee3f0ffb00: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:24:15,050::task::495::TaskManager.Task::(_debug) Task 03173789-5b61-4fd4-bb6c-6bee3f0ffb00: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:24:15,051::task::495::TaskManager.Task::(_debug) Task 03173789-5b61-4fd4-bb6c-6bee3f0ffb00: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:24:15,052::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:24:15,053::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:24:15,054::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:24:15,056::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:24:15,057::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:24:15,057::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:24:15,059::task::495::TaskManager.Task::(_debug) Task 03173789-5b61-4fd4-bb6c-6bee3f0ffb00: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:24:15,060::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:24:15,061::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:24:15,062::task::495::TaskManager.Task::(_debug) Task 1ae739bc-351f-4327-8f2a-6cec242ea64e: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:24:15,063::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`124fbf7b-0c6b-486d-9712-5e5137ea7e0f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:24:15,064::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:24:15,066::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:24:15,067::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`124fbf7b-0c6b-486d-9712-5e5137ea7e0f`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:24:15,069::task::495::TaskManager.Task::(_debug) Task 1ae739bc-351f-4327-8f2a-6cec242ea64e: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:24:15,070::task::495::TaskManager.Task::(_debug) Task 1ae739bc-351f-4327-8f2a-6cec242ea64e: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:24:15,071::task::495::TaskManager.Task::(_debug) Task 1ae739bc-351f-4327-8f2a-6cec242ea64e: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:24:15,072::task::495::TaskManager.Task::(_debug) Task 1ae739bc-351f-4327-8f2a-6cec242ea64e: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:24:15,073::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:24:15,074::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:24:15,075::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:24:15,076::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:24:15,077::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:24:15,078::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:24:15,079::task::495::TaskManager.Task::(_debug) Task 1ae739bc-351f-4327-8f2a-6cec242ea64e: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:24:15,080::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:24:15,081::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:24:15,081::task::495::TaskManager.Task::(_debug) Task 4c192c84-1898-4081-a378-302a012f10aa: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:24:15,083::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`349f1cdb-1200-4a79-9c7a-ecd333080015`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:24:15,084::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:24:15,085::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:24:15,085::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`349f1cdb-1200-4a79-9c7a-ecd333080015`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:24:15,087::task::495::TaskManager.Task::(_debug) Task 4c192c84-1898-4081-a378-302a012f10aa: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:24:15,088::task::495::TaskManager.Task::(_debug) Task 4c192c84-1898-4081-a378-302a012f10aa: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:24:15,089::task::495::TaskManager.Task::(_debug) Task 4c192c84-1898-4081-a378-302a012f10aa: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:24:15,089::task::495::TaskManager.Task::(_debug) Task 4c192c84-1898-4081-a378-302a012f10aa: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:24:15,090::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:24:15,091::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:24:15,092::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:24:15,093::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:24:15,094::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:24:15,095::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:24:15,096::task::495::TaskManager.Task::(_debug) Task 4c192c84-1898-4081-a378-302a012f10aa: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:24:15,096::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:24:15,097::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:24:15,098::task::495::TaskManager.Task::(_debug) Task 54743dc4-ceb4-4f86-89ba-84ee55e4171e: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:24:15,100::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`641e0c2c-7048-4f93-8720-3763b7835c49`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:24:15,100::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:24:15,101::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:24:15,102::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`641e0c2c-7048-4f93-8720-3763b7835c49`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:24:15,104::task::495::TaskManager.Task::(_debug) Task 54743dc4-ceb4-4f86-89ba-84ee55e4171e: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:24:15,104::task::495::TaskManager.Task::(_debug) Task 54743dc4-ceb4-4f86-89ba-84ee55e4171e: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:24:15,105::task::495::TaskManager.Task::(_debug) Task 54743dc4-ceb4-4f86-89ba-84ee55e4171e: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:24:15,106::task::495::TaskManager.Task::(_debug) Task 54743dc4-ceb4-4f86-89ba-84ee55e4171e: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:24:15,107::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:24:15,107::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:24:15,108::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:24:15,110::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:24:15,110::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:24:15,111::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:24:15,112::task::495::TaskManager.Task::(_debug) Task 54743dc4-ceb4-4f86-89ba-84ee55e4171e: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:24:15,113::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:24:15,230::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.128048 s, 8.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:24:17,245::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:24:17,690::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0749966 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:24:19,704::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-42840::DEBUG::2013-11-20 03:24:19,962::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:24:20,327::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.221923 s, 4.6 MB/s\n'; <rc> = 0 >Thread-42840::DEBUG::2013-11-20 03:24:20,719::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:24:22,344::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:24:22,753::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0470787 s, 21.8 MB/s\n'; <rc> = 0 >Thread-15914152::DEBUG::2013-11-20 03:24:23,185::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914152::INFO::2013-11-20 03:24:23,186::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914152::DEBUG::2013-11-20 03:24:23,186::task::495::TaskManager.Task::(_debug) Task 3a95c229-1a56-4f4e-9534-4ecd0c72c5b2: moving from state init -> state preparing >Thread-15914152::DEBUG::2013-11-20 03:24:23,193::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914152::DEBUG::2013-11-20 03:24:23,193::task::495::TaskManager.Task::(_debug) Task 3a95c229-1a56-4f4e-9534-4ecd0c72c5b2: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914152::DEBUG::2013-11-20 03:24:23,194::task::495::TaskManager.Task::(_debug) Task 3a95c229-1a56-4f4e-9534-4ecd0c72c5b2: moving from state preparing -> state finished >Thread-15914152::DEBUG::2013-11-20 03:24:23,194::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914152::DEBUG::2013-11-20 03:24:23,194::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914152::DEBUG::2013-11-20 03:24:23,195::task::495::TaskManager.Task::(_debug) Task 3a95c229-1a56-4f4e-9534-4ecd0c72c5b2: ref 0 aborting False >Thread-15914152::INFO::2013-11-20 03:24:23,195::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914153::DEBUG::2013-11-20 03:24:23,198::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914153::INFO::2013-11-20 03:24:23,199::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914153::DEBUG::2013-11-20 03:24:23,199::task::495::TaskManager.Task::(_debug) Task ab3fe7dc-2bab-42d5-aa40-ee0605e032eb: moving from state init -> state preparing >Thread-15914153::DEBUG::2013-11-20 03:24:23,200::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`092feff0-7e04-447e-b246-22e1a8815fd8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914153::DEBUG::2013-11-20 03:24:23,200::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914153::DEBUG::2013-11-20 03:24:23,201::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914153::DEBUG::2013-11-20 03:24:23,201::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`092feff0-7e04-447e-b246-22e1a8815fd8`::Granted request >Thread-15914153::DEBUG::2013-11-20 03:24:23,202::task::495::TaskManager.Task::(_debug) Task ab3fe7dc-2bab-42d5-aa40-ee0605e032eb: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914153::DEBUG::2013-11-20 03:24:23,202::task::495::TaskManager.Task::(_debug) Task ab3fe7dc-2bab-42d5-aa40-ee0605e032eb: ref 1 aborting False >Thread-15914153::WARNING::2013-11-20 03:24:23,203::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914153::DEBUG::2013-11-20 03:24:23,204::task::495::TaskManager.Task::(_debug) Task ab3fe7dc-2bab-42d5-aa40-ee0605e032eb: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914153::DEBUG::2013-11-20 03:24:23,204::task::495::TaskManager.Task::(_debug) Task ab3fe7dc-2bab-42d5-aa40-ee0605e032eb: moving from state preparing -> state finished >Thread-15914153::DEBUG::2013-11-20 03:24:23,205::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914153::DEBUG::2013-11-20 03:24:23,205::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914153::DEBUG::2013-11-20 03:24:23,205::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914153::DEBUG::2013-11-20 03:24:23,206::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914153::DEBUG::2013-11-20 03:24:23,206::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914153::DEBUG::2013-11-20 03:24:23,206::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914153::DEBUG::2013-11-20 03:24:23,207::task::495::TaskManager.Task::(_debug) Task ab3fe7dc-2bab-42d5-aa40-ee0605e032eb: ref 0 aborting False >Thread-15914153::INFO::2013-11-20 03:24:23,207::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:24:24,765::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913639::INFO::2013-11-20 03:24:25,175::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:24:25,179::task::495::TaskManager.Task::(_debug) Task eca30c8e-dcc9-4db1-8325-54663b3210e1: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:24:25,182::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`bd4414ef-d690-46a2-a335-5d392a1613fc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:24:25,183::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:24:25,185::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:24:25,186::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`bd4414ef-d690-46a2-a335-5d392a1613fc`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:24:25,188::task::495::TaskManager.Task::(_debug) Task eca30c8e-dcc9-4db1-8325-54663b3210e1: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:24:25,189::task::495::TaskManager.Task::(_debug) Task eca30c8e-dcc9-4db1-8325-54663b3210e1: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:24:25,190::task::495::TaskManager.Task::(_debug) Task eca30c8e-dcc9-4db1-8325-54663b3210e1: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:24:25,192::task::495::TaskManager.Task::(_debug) Task eca30c8e-dcc9-4db1-8325-54663b3210e1: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:24:25,194::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:24:25,195::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:24:25,196::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:24:25,197::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:24:25,198::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:24:25,199::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:24:25,200::task::495::TaskManager.Task::(_debug) Task eca30c8e-dcc9-4db1-8325-54663b3210e1: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:24:25,201::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:24:25,278::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0905736 s, 11.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:24:27,296::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:24:27,766::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0525131 s, 19.5 MB/s\n'; <rc> = 0 >Thread-15914156::INFO::2013-11-20 03:24:29,516::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914156::DEBUG::2013-11-20 03:24:29,517::task::495::TaskManager.Task::(_debug) Task b4521f84-aa53-43f2-9f8c-168df7346179: moving from state init -> state preparing >Thread-15914156::DEBUG::2013-11-20 03:24:29,525::task::495::TaskManager.Task::(_debug) Task b4521f84-aa53-43f2-9f8c-168df7346179: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000321865081787', 'lastCheck': 1384917861.5068901, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000999927520752', 'lastCheck': 1384917867.5552089, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00120806694031', 'lastCheck': 1384917860.1748381, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000303983688354', 'lastCheck': 1384917860.7216909, 'valid': True, 'code': 0}} >Thread-15914156::DEBUG::2013-11-20 03:24:29,526::task::495::TaskManager.Task::(_debug) Task b4521f84-aa53-43f2-9f8c-168df7346179: moving from state preparing -> state finished >Thread-15914156::DEBUG::2013-11-20 03:24:29,527::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914156::DEBUG::2013-11-20 03:24:29,528::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914156::DEBUG::2013-11-20 03:24:29,528::task::495::TaskManager.Task::(_debug) Task b4521f84-aa53-43f2-9f8c-168df7346179: ref 0 aborting False >Thread-15914156::INFO::2013-11-20 03:24:29,529::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000321865081787', 'lastCheck': 1384917861.5068901, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000999927520752', 'lastCheck': 1384917867.5552089, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00120806694031', 'lastCheck': 1384917860.1748381, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000303983688354', 'lastCheck': 1384917860.7216909, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:24:29,776::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:24:30,280::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911958 s, 11.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:24:32,295::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:24:32,695::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0465249 s, 22.0 MB/s\n'; <rc> = 0 >Thread-15914158::DEBUG::2013-11-20 03:24:33,392::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914158::INFO::2013-11-20 03:24:33,393::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914158::DEBUG::2013-11-20 03:24:33,394::task::495::TaskManager.Task::(_debug) Task 92998aa9-6428-4ea4-b498-550e7e68810b: moving from state init -> state preparing >Thread-15914158::DEBUG::2013-11-20 03:24:33,394::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914158::DEBUG::2013-11-20 03:24:33,395::task::495::TaskManager.Task::(_debug) Task 92998aa9-6428-4ea4-b498-550e7e68810b: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914158::DEBUG::2013-11-20 03:24:33,395::task::495::TaskManager.Task::(_debug) Task 92998aa9-6428-4ea4-b498-550e7e68810b: moving from state preparing -> state finished >Thread-15914158::DEBUG::2013-11-20 03:24:33,395::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914158::DEBUG::2013-11-20 03:24:33,396::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914158::DEBUG::2013-11-20 03:24:33,396::task::495::TaskManager.Task::(_debug) Task 92998aa9-6428-4ea4-b498-550e7e68810b: ref 0 aborting False >Thread-15914158::INFO::2013-11-20 03:24:33,396::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914159::DEBUG::2013-11-20 03:24:33,400::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914159::INFO::2013-11-20 03:24:33,400::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914159::DEBUG::2013-11-20 03:24:33,401::task::495::TaskManager.Task::(_debug) Task 11117f9e-fd9a-49b8-835d-d73c08fd0f79: moving from state init -> state preparing >Thread-15914159::DEBUG::2013-11-20 03:24:33,401::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`62bca08b-ec85-4e32-86ff-d19b8258c44d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914159::DEBUG::2013-11-20 03:24:33,402::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914159::DEBUG::2013-11-20 03:24:33,402::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914159::DEBUG::2013-11-20 03:24:33,403::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`62bca08b-ec85-4e32-86ff-d19b8258c44d`::Granted request >Thread-15914159::DEBUG::2013-11-20 03:24:33,409::task::495::TaskManager.Task::(_debug) Task 11117f9e-fd9a-49b8-835d-d73c08fd0f79: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914159::DEBUG::2013-11-20 03:24:33,409::task::495::TaskManager.Task::(_debug) Task 11117f9e-fd9a-49b8-835d-d73c08fd0f79: ref 1 aborting False >Thread-15914159::WARNING::2013-11-20 03:24:33,411::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914159::DEBUG::2013-11-20 03:24:33,411::task::495::TaskManager.Task::(_debug) Task 11117f9e-fd9a-49b8-835d-d73c08fd0f79: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914159::DEBUG::2013-11-20 03:24:33,412::task::495::TaskManager.Task::(_debug) Task 11117f9e-fd9a-49b8-835d-d73c08fd0f79: moving from state preparing -> state finished >Thread-15914159::DEBUG::2013-11-20 03:24:33,412::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914159::DEBUG::2013-11-20 03:24:33,412::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914159::DEBUG::2013-11-20 03:24:33,413::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914159::DEBUG::2013-11-20 03:24:33,413::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914159::DEBUG::2013-11-20 03:24:33,413::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914159::DEBUG::2013-11-20 03:24:33,414::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914159::DEBUG::2013-11-20 03:24:33,414::task::495::TaskManager.Task::(_debug) Task 11117f9e-fd9a-49b8-835d-d73c08fd0f79: ref 0 aborting False >Thread-15914159::INFO::2013-11-20 03:24:33,415::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:24:34,709::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:24:35,127::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0541877 s, 18.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:24:37,138::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:24:37,673::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.125659 s, 8.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:24:39,688::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:24:40,137::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0618561 s, 16.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:24:42,149::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:24:42,546::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0469905 s, 21.8 MB/s\n'; <rc> = 0 >Thread-15913695::INFO::2013-11-20 03:24:42,901::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:24:42,901::task::495::TaskManager.Task::(_debug) Task e297488f-5d86-4ee7-8cb2-8c5cc3eecf3f: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:24:42,902::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`bb3e02e4-983d-4a7b-99ff-31f30cbecf77`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:24:42,903::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:24:42,903::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:24:42,904::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`bb3e02e4-983d-4a7b-99ff-31f30cbecf77`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:24:42,904::task::495::TaskManager.Task::(_debug) Task e297488f-5d86-4ee7-8cb2-8c5cc3eecf3f: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:24:42,905::task::495::TaskManager.Task::(_debug) Task e297488f-5d86-4ee7-8cb2-8c5cc3eecf3f: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:24:42,905::task::495::TaskManager.Task::(_debug) Task e297488f-5d86-4ee7-8cb2-8c5cc3eecf3f: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:24:42,906::task::495::TaskManager.Task::(_debug) Task e297488f-5d86-4ee7-8cb2-8c5cc3eecf3f: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:24:42,906::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:24:42,907::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:24:42,907::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:24:42,907::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:24:42,908::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:24:42,908::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:24:42,908::task::495::TaskManager.Task::(_debug) Task e297488f-5d86-4ee7-8cb2-8c5cc3eecf3f: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:24:42,909::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15914164::DEBUG::2013-11-20 03:24:44,071::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914164::INFO::2013-11-20 03:24:44,072::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914164::DEBUG::2013-11-20 03:24:44,073::task::495::TaskManager.Task::(_debug) Task 87558e6e-79db-494f-898a-fb1f80968e1d: moving from state init -> state preparing >Thread-15914164::DEBUG::2013-11-20 03:24:44,074::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914164::DEBUG::2013-11-20 03:24:44,075::task::495::TaskManager.Task::(_debug) Task 87558e6e-79db-494f-898a-fb1f80968e1d: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914164::DEBUG::2013-11-20 03:24:44,075::task::495::TaskManager.Task::(_debug) Task 87558e6e-79db-494f-898a-fb1f80968e1d: moving from state preparing -> state finished >Thread-15914164::DEBUG::2013-11-20 03:24:44,076::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914164::DEBUG::2013-11-20 03:24:44,077::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914164::DEBUG::2013-11-20 03:24:44,077::task::495::TaskManager.Task::(_debug) Task 87558e6e-79db-494f-898a-fb1f80968e1d: ref 0 aborting False >Thread-15914164::INFO::2013-11-20 03:24:44,078::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914165::DEBUG::2013-11-20 03:24:44,083::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914165::INFO::2013-11-20 03:24:44,084::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914165::DEBUG::2013-11-20 03:24:44,084::task::495::TaskManager.Task::(_debug) Task 7d9f1892-c089-4400-931e-a59421baa245: moving from state init -> state preparing >Thread-15914165::DEBUG::2013-11-20 03:24:44,086::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`09a776cc-88d7-4587-8129-b2bb8fc3c407`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914165::DEBUG::2013-11-20 03:24:44,086::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914165::DEBUG::2013-11-20 03:24:44,087::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914165::DEBUG::2013-11-20 03:24:44,088::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`09a776cc-88d7-4587-8129-b2bb8fc3c407`::Granted request >Thread-15914165::DEBUG::2013-11-20 03:24:44,089::task::495::TaskManager.Task::(_debug) Task 7d9f1892-c089-4400-931e-a59421baa245: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914165::DEBUG::2013-11-20 03:24:44,089::task::495::TaskManager.Task::(_debug) Task 7d9f1892-c089-4400-931e-a59421baa245: ref 1 aborting False >Thread-15914165::WARNING::2013-11-20 03:24:44,092::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914165::DEBUG::2013-11-20 03:24:44,093::task::495::TaskManager.Task::(_debug) Task 7d9f1892-c089-4400-931e-a59421baa245: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914165::DEBUG::2013-11-20 03:24:44,094::task::495::TaskManager.Task::(_debug) Task 7d9f1892-c089-4400-931e-a59421baa245: moving from state preparing -> state finished >Thread-15914165::DEBUG::2013-11-20 03:24:44,094::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914165::DEBUG::2013-11-20 03:24:44,095::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914165::DEBUG::2013-11-20 03:24:44,096::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914165::DEBUG::2013-11-20 03:24:44,096::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914165::DEBUG::2013-11-20 03:24:44,097::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914165::DEBUG::2013-11-20 03:24:44,098::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914165::DEBUG::2013-11-20 03:24:44,098::task::495::TaskManager.Task::(_debug) Task 7d9f1892-c089-4400-931e-a59421baa245: ref 0 aborting False >Thread-15914165::INFO::2013-11-20 03:24:44,099::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:24:44,559::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:24:44,991::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0511572 s, 20.0 MB/s\n'; <rc> = 0 >Thread-15913658::INFO::2013-11-20 03:24:45,320::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:24:45,321::task::495::TaskManager.Task::(_debug) Task d0cb5065-b888-4712-a5a2-75dc882b7cba: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:24:45,324::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`687b363d-e713-43e8-8e4b-d67c9391c4b5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:24:45,325::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:24:45,326::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:24:45,326::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`687b363d-e713-43e8-8e4b-d67c9391c4b5`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:24:45,327::task::495::TaskManager.Task::(_debug) Task d0cb5065-b888-4712-a5a2-75dc882b7cba: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:24:45,328::task::495::TaskManager.Task::(_debug) Task d0cb5065-b888-4712-a5a2-75dc882b7cba: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:24:45,329::task::495::TaskManager.Task::(_debug) Task d0cb5065-b888-4712-a5a2-75dc882b7cba: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:24:45,329::task::495::TaskManager.Task::(_debug) Task d0cb5065-b888-4712-a5a2-75dc882b7cba: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:24:45,330::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:24:45,331::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:24:45,331::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:24:45,332::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:24:45,333::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:24:45,333::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:24:45,334::task::495::TaskManager.Task::(_debug) Task d0cb5065-b888-4712-a5a2-75dc882b7cba: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:24:45,334::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15914166::INFO::2013-11-20 03:24:45,717::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914166::DEBUG::2013-11-20 03:24:45,718::task::495::TaskManager.Task::(_debug) Task 83afa418-4d3d-4e1d-b9c4-b034f89546d7: moving from state init -> state preparing >Thread-15914166::DEBUG::2013-11-20 03:24:45,719::task::495::TaskManager.Task::(_debug) Task 83afa418-4d3d-4e1d-b9c4-b034f89546d7: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000272035598755', 'lastCheck': 1384917881.54861, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00165200233459', 'lastCheck': 1384917877.5721259, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00133800506592', 'lastCheck': 1384917880.3223679, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000862121582031', 'lastCheck': 1384917880.876127, 'valid': True, 'code': 0}} >Thread-15914166::DEBUG::2013-11-20 03:24:45,720::task::495::TaskManager.Task::(_debug) Task 83afa418-4d3d-4e1d-b9c4-b034f89546d7: moving from state preparing -> state finished >Thread-15914166::DEBUG::2013-11-20 03:24:45,720::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914166::DEBUG::2013-11-20 03:24:45,721::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914166::DEBUG::2013-11-20 03:24:45,721::task::495::TaskManager.Task::(_debug) Task 83afa418-4d3d-4e1d-b9c4-b034f89546d7: ref 0 aborting False >Thread-15914166::INFO::2013-11-20 03:24:45,722::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000272035598755', 'lastCheck': 1384917881.54861, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00165200233459', 'lastCheck': 1384917877.5721259, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00133800506592', 'lastCheck': 1384917880.3223679, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000862121582031', 'lastCheck': 1384917880.876127, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:24:47,002::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:24:47,552::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0765975 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:24:49,567::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:24:50,098::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0742785 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15912392::INFO::2013-11-20 03:24:50,316::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:24:50,317::task::495::TaskManager.Task::(_debug) Task 45690f3b-9d12-4407-be73-b6033b4dff36: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:24:50,320::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`2db53c16-fb83-4866-8ca9-3ffa9c9128cb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:24:50,320::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:24:50,321::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:24:50,322::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`2db53c16-fb83-4866-8ca9-3ffa9c9128cb`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:24:50,324::task::495::TaskManager.Task::(_debug) Task 45690f3b-9d12-4407-be73-b6033b4dff36: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:24:50,325::task::495::TaskManager.Task::(_debug) Task 45690f3b-9d12-4407-be73-b6033b4dff36: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:24:50,326::task::495::TaskManager.Task::(_debug) Task 45690f3b-9d12-4407-be73-b6033b4dff36: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:24:50,327::task::495::TaskManager.Task::(_debug) Task 45690f3b-9d12-4407-be73-b6033b4dff36: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:24:50,328::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:24:50,329::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:24:50,331::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:24:50,331::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:24:50,332::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:24:50,334::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:24:50,335::task::495::TaskManager.Task::(_debug) Task 45690f3b-9d12-4407-be73-b6033b4dff36: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:24:50,336::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:24:52,112::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:24:52,619::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0631525 s, 16.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:24:54,634::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914170::DEBUG::2013-11-20 03:24:54,857::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914170::INFO::2013-11-20 03:24:54,860::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914170::DEBUG::2013-11-20 03:24:54,868::task::495::TaskManager.Task::(_debug) Task 2fb80eed-a863-4f0a-8e2d-1df19017e76d: moving from state init -> state preparing >Thread-15914170::DEBUG::2013-11-20 03:24:54,870::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914170::DEBUG::2013-11-20 03:24:54,870::task::495::TaskManager.Task::(_debug) Task 2fb80eed-a863-4f0a-8e2d-1df19017e76d: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914170::DEBUG::2013-11-20 03:24:54,871::task::495::TaskManager.Task::(_debug) Task 2fb80eed-a863-4f0a-8e2d-1df19017e76d: moving from state preparing -> state finished >Thread-15914170::DEBUG::2013-11-20 03:24:54,872::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914170::DEBUG::2013-11-20 03:24:54,872::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914170::DEBUG::2013-11-20 03:24:54,873::task::495::TaskManager.Task::(_debug) Task 2fb80eed-a863-4f0a-8e2d-1df19017e76d: ref 0 aborting False >Thread-15914170::INFO::2013-11-20 03:24:54,873::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914171::DEBUG::2013-11-20 03:24:54,879::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914171::INFO::2013-11-20 03:24:54,880::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914171::DEBUG::2013-11-20 03:24:54,880::task::495::TaskManager.Task::(_debug) Task 96e79e1e-5a02-4f84-b606-851f12a7b524: moving from state init -> state preparing >Thread-15914171::DEBUG::2013-11-20 03:24:54,882::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a3fbd7df-37f4-424a-a73f-442a77ce5fe0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914171::DEBUG::2013-11-20 03:24:54,882::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914171::DEBUG::2013-11-20 03:24:54,883::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914171::DEBUG::2013-11-20 03:24:54,883::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a3fbd7df-37f4-424a-a73f-442a77ce5fe0`::Granted request >Thread-15914171::DEBUG::2013-11-20 03:24:54,884::task::495::TaskManager.Task::(_debug) Task 96e79e1e-5a02-4f84-b606-851f12a7b524: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914171::DEBUG::2013-11-20 03:24:54,885::task::495::TaskManager.Task::(_debug) Task 96e79e1e-5a02-4f84-b606-851f12a7b524: ref 1 aborting False >Thread-15914171::WARNING::2013-11-20 03:24:54,887::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914171::DEBUG::2013-11-20 03:24:54,888::task::495::TaskManager.Task::(_debug) Task 96e79e1e-5a02-4f84-b606-851f12a7b524: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914171::DEBUG::2013-11-20 03:24:54,888::task::495::TaskManager.Task::(_debug) Task 96e79e1e-5a02-4f84-b606-851f12a7b524: moving from state preparing -> state finished >Thread-15914171::DEBUG::2013-11-20 03:24:54,889::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914171::DEBUG::2013-11-20 03:24:54,889::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914171::DEBUG::2013-11-20 03:24:54,890::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914171::DEBUG::2013-11-20 03:24:54,891::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914171::DEBUG::2013-11-20 03:24:54,891::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914171::DEBUG::2013-11-20 03:24:54,891::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914171::DEBUG::2013-11-20 03:24:54,892::task::495::TaskManager.Task::(_debug) Task 96e79e1e-5a02-4f84-b606-851f12a7b524: ref 0 aborting False >Thread-15914171::INFO::2013-11-20 03:24:54,892::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:24:55,163::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0730583 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:24:57,178::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:24:57,703::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0869889 s, 11.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:24:59,720::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913518::INFO::2013-11-20 03:24:59,961::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:24:59,962::task::495::TaskManager.Task::(_debug) Task 3fe6af1a-280f-4951-abae-174aef09b7a0: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:24:59,963::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`89bbb32b-5cee-4586-8309-4e1805713f4d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:24:59,964::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:24:59,964::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:24:59,965::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`89bbb32b-5cee-4586-8309-4e1805713f4d`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:24:59,966::task::495::TaskManager.Task::(_debug) Task 3fe6af1a-280f-4951-abae-174aef09b7a0: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:24:59,966::task::495::TaskManager.Task::(_debug) Task 3fe6af1a-280f-4951-abae-174aef09b7a0: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:24:59,967::task::495::TaskManager.Task::(_debug) Task 3fe6af1a-280f-4951-abae-174aef09b7a0: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:24:59,967::task::495::TaskManager.Task::(_debug) Task 3fe6af1a-280f-4951-abae-174aef09b7a0: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:24:59,968::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:24:59,968::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:24:59,968::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:24:59,969::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:24:59,969::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:24:59,970::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:24:59,970::task::495::TaskManager.Task::(_debug) Task 3fe6af1a-280f-4951-abae-174aef09b7a0: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:24:59,970::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:24:59,971::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:24:59,971::task::495::TaskManager.Task::(_debug) Task 4fe37c6f-5bcf-40b0-8493-e376052e9066: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:24:59,972::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`78f84e39-ebf5-4c57-a727-aafe4403c94e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:24:59,972::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:24:59,973::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:24:59,973::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`78f84e39-ebf5-4c57-a727-aafe4403c94e`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:24:59,974::task::495::TaskManager.Task::(_debug) Task 4fe37c6f-5bcf-40b0-8493-e376052e9066: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:24:59,974::task::495::TaskManager.Task::(_debug) Task 4fe37c6f-5bcf-40b0-8493-e376052e9066: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:24:59,975::task::495::TaskManager.Task::(_debug) Task 4fe37c6f-5bcf-40b0-8493-e376052e9066: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:24:59,975::task::495::TaskManager.Task::(_debug) Task 4fe37c6f-5bcf-40b0-8493-e376052e9066: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:24:59,975::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:24:59,976::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:24:59,976::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:24:59,976::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:24:59,977::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:24:59,977::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:24:59,977::task::495::TaskManager.Task::(_debug) Task 4fe37c6f-5bcf-40b0-8493-e376052e9066: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:24:59,978::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:25:00,245::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0799752 s, 12.8 MB/s\n'; <rc> = 0 >Thread-15914174::INFO::2013-11-20 03:25:02,034::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914174::DEBUG::2013-11-20 03:25:02,035::task::495::TaskManager.Task::(_debug) Task ba8cc424-71d5-4a3e-8dfc-ff922cbe6b28: moving from state init -> state preparing >Thread-15914174::DEBUG::2013-11-20 03:25:02,035::task::495::TaskManager.Task::(_debug) Task ba8cc424-71d5-4a3e-8dfc-ff922cbe6b28: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000346183776855', 'lastCheck': 1384917901.581207, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0186901092529', 'lastCheck': 1384917897.8769, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00669407844543', 'lastCheck': 1384917900.3870571, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0002760887146', 'lastCheck': 1384917900.915432, 'valid': True, 'code': 0}} >Thread-15914174::DEBUG::2013-11-20 03:25:02,036::task::495::TaskManager.Task::(_debug) Task ba8cc424-71d5-4a3e-8dfc-ff922cbe6b28: moving from state preparing -> state finished >Thread-15914174::DEBUG::2013-11-20 03:25:02,036::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914174::DEBUG::2013-11-20 03:25:02,037::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914174::DEBUG::2013-11-20 03:25:02,037::task::495::TaskManager.Task::(_debug) Task ba8cc424-71d5-4a3e-8dfc-ff922cbe6b28: ref 0 aborting False >Thread-15914174::INFO::2013-11-20 03:25:02,037::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000346183776855', 'lastCheck': 1384917901.581207, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0186901092529', 'lastCheck': 1384917897.8769, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00669407844543', 'lastCheck': 1384917900.3870571, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0002760887146', 'lastCheck': 1384917900.915432, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:25:02,258::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:02,680::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0728409 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:25:04,694::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:05,189::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0459419 s, 22.3 MB/s\n'; <rc> = 0 >Thread-15914176::DEBUG::2013-11-20 03:25:05,349::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914176::INFO::2013-11-20 03:25:05,350::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914176::DEBUG::2013-11-20 03:25:05,351::task::495::TaskManager.Task::(_debug) Task 0204fe53-6a8b-4e20-91e7-7f010e5d60c1: moving from state init -> state preparing >Thread-15914176::DEBUG::2013-11-20 03:25:05,351::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914176::DEBUG::2013-11-20 03:25:05,351::task::495::TaskManager.Task::(_debug) Task 0204fe53-6a8b-4e20-91e7-7f010e5d60c1: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914176::DEBUG::2013-11-20 03:25:05,352::task::495::TaskManager.Task::(_debug) Task 0204fe53-6a8b-4e20-91e7-7f010e5d60c1: moving from state preparing -> state finished >Thread-15914176::DEBUG::2013-11-20 03:25:05,352::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914176::DEBUG::2013-11-20 03:25:05,353::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914176::DEBUG::2013-11-20 03:25:05,353::task::495::TaskManager.Task::(_debug) Task 0204fe53-6a8b-4e20-91e7-7f010e5d60c1: ref 0 aborting False >Thread-15914176::INFO::2013-11-20 03:25:05,353::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914177::DEBUG::2013-11-20 03:25:05,357::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914177::INFO::2013-11-20 03:25:05,358::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914177::DEBUG::2013-11-20 03:25:05,359::task::495::TaskManager.Task::(_debug) Task 26782f95-b7ec-4087-b026-32e8b79e023f: moving from state init -> state preparing >Thread-15914177::DEBUG::2013-11-20 03:25:05,360::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b5b70b9f-4003-42ab-819a-c03649f4acc7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914177::DEBUG::2013-11-20 03:25:05,361::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914177::DEBUG::2013-11-20 03:25:05,362::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914177::DEBUG::2013-11-20 03:25:05,362::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b5b70b9f-4003-42ab-819a-c03649f4acc7`::Granted request >Thread-15914177::DEBUG::2013-11-20 03:25:05,363::task::495::TaskManager.Task::(_debug) Task 26782f95-b7ec-4087-b026-32e8b79e023f: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914177::DEBUG::2013-11-20 03:25:05,364::task::495::TaskManager.Task::(_debug) Task 26782f95-b7ec-4087-b026-32e8b79e023f: ref 1 aborting False >Thread-15914177::WARNING::2013-11-20 03:25:05,366::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914177::DEBUG::2013-11-20 03:25:05,367::task::495::TaskManager.Task::(_debug) Task 26782f95-b7ec-4087-b026-32e8b79e023f: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914177::DEBUG::2013-11-20 03:25:05,368::task::495::TaskManager.Task::(_debug) Task 26782f95-b7ec-4087-b026-32e8b79e023f: moving from state preparing -> state finished >Thread-15914177::DEBUG::2013-11-20 03:25:05,368::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914177::DEBUG::2013-11-20 03:25:05,369::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914177::DEBUG::2013-11-20 03:25:05,369::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914177::DEBUG::2013-11-20 03:25:05,370::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914177::DEBUG::2013-11-20 03:25:05,370::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914177::DEBUG::2013-11-20 03:25:05,371::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914177::DEBUG::2013-11-20 03:25:05,372::task::495::TaskManager.Task::(_debug) Task 26782f95-b7ec-4087-b026-32e8b79e023f: ref 0 aborting False >Thread-15914177::INFO::2013-11-20 03:25:05,372::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:25:07,201::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:07,676::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.050299 s, 20.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:25:09,684::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:10,155::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0599094 s, 17.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:25:12,169::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:12,632::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0852778 s, 12.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:25:14,646::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:15,190::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.088593 s, 11.6 MB/s\n'; <rc> = 0 >Thread-15913546::INFO::2013-11-20 03:25:15,394::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:25:15,396::task::495::TaskManager.Task::(_debug) Task 72fa7c61-4f17-4fd0-8a63-0c12737af6f2: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:25:15,398::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fd31316e-cbf2-43cf-a4fd-a447e751137b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:25:15,399::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:25:15,400::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:25:15,401::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fd31316e-cbf2-43cf-a4fd-a447e751137b`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:25:15,402::task::495::TaskManager.Task::(_debug) Task 72fa7c61-4f17-4fd0-8a63-0c12737af6f2: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:25:15,402::task::495::TaskManager.Task::(_debug) Task 72fa7c61-4f17-4fd0-8a63-0c12737af6f2: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:25:15,403::task::495::TaskManager.Task::(_debug) Task 72fa7c61-4f17-4fd0-8a63-0c12737af6f2: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:25:15,404::task::495::TaskManager.Task::(_debug) Task 72fa7c61-4f17-4fd0-8a63-0c12737af6f2: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:25:15,404::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:25:15,405::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:25:15,405::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:25:15,406::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:25:15,407::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:25:15,407::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:25:15,408::task::495::TaskManager.Task::(_debug) Task 72fa7c61-4f17-4fd0-8a63-0c12737af6f2: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:25:15,408::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:25:15,409::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:25:15,410::task::495::TaskManager.Task::(_debug) Task 0bf396c1-8beb-4d76-9ed0-cce926a7b63c: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:25:15,411::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`756cfb85-5835-495b-8eab-7e00cde7d716`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:25:15,411::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:25:15,412::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:25:15,412::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`756cfb85-5835-495b-8eab-7e00cde7d716`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:25:15,413::task::495::TaskManager.Task::(_debug) Task 0bf396c1-8beb-4d76-9ed0-cce926a7b63c: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:25:15,414::task::495::TaskManager.Task::(_debug) Task 0bf396c1-8beb-4d76-9ed0-cce926a7b63c: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:25:15,414::task::495::TaskManager.Task::(_debug) Task 0bf396c1-8beb-4d76-9ed0-cce926a7b63c: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:25:15,415::task::495::TaskManager.Task::(_debug) Task 0bf396c1-8beb-4d76-9ed0-cce926a7b63c: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:25:15,415::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:25:15,416::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:25:15,417::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:25:15,417::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:25:15,418::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:25:15,418::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:25:15,419::task::495::TaskManager.Task::(_debug) Task 0bf396c1-8beb-4d76-9ed0-cce926a7b63c: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:25:15,419::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:25:15,420::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:25:15,421::task::495::TaskManager.Task::(_debug) Task e189f7a8-0ac9-4ea2-863a-fc1270340661: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:25:15,422::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`95bff181-5f02-4c1e-aaa1-57d24d272a65`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:25:15,422::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:25:15,423::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:25:15,423::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`95bff181-5f02-4c1e-aaa1-57d24d272a65`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:25:15,424::task::495::TaskManager.Task::(_debug) Task e189f7a8-0ac9-4ea2-863a-fc1270340661: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:25:15,425::task::495::TaskManager.Task::(_debug) Task e189f7a8-0ac9-4ea2-863a-fc1270340661: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:25:15,425::task::495::TaskManager.Task::(_debug) Task e189f7a8-0ac9-4ea2-863a-fc1270340661: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:25:15,426::task::495::TaskManager.Task::(_debug) Task e189f7a8-0ac9-4ea2-863a-fc1270340661: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:25:15,426::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:25:15,427::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:25:15,428::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:25:15,428::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:25:15,429::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:25:15,429::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:25:15,430::task::495::TaskManager.Task::(_debug) Task e189f7a8-0ac9-4ea2-863a-fc1270340661: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:25:15,431::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:25:15,432::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:25:15,432::task::495::TaskManager.Task::(_debug) Task 8ba31eac-9ae6-4a72-b290-b806b3666bc3: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:25:15,433::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`9d33dd12-ea3d-4293-9857-d29668ba2f45`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:25:15,434::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:25:15,435::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:25:15,435::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`9d33dd12-ea3d-4293-9857-d29668ba2f45`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:25:15,436::task::495::TaskManager.Task::(_debug) Task 8ba31eac-9ae6-4a72-b290-b806b3666bc3: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:25:15,436::task::495::TaskManager.Task::(_debug) Task 8ba31eac-9ae6-4a72-b290-b806b3666bc3: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:25:15,437::task::495::TaskManager.Task::(_debug) Task 8ba31eac-9ae6-4a72-b290-b806b3666bc3: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:25:15,438::task::495::TaskManager.Task::(_debug) Task 8ba31eac-9ae6-4a72-b290-b806b3666bc3: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:25:15,438::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:25:15,439::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:25:15,439::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:25:15,440::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:25:15,440::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:25:15,441::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:25:15,441::task::495::TaskManager.Task::(_debug) Task 8ba31eac-9ae6-4a72-b290-b806b3666bc3: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:25:15,442::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15914182::DEBUG::2013-11-20 03:25:16,100::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914182::INFO::2013-11-20 03:25:16,101::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914182::DEBUG::2013-11-20 03:25:16,102::task::495::TaskManager.Task::(_debug) Task f30dbafe-bf59-4032-bb72-ea9a45eac8cc: moving from state init -> state preparing >Thread-15914182::DEBUG::2013-11-20 03:25:16,103::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914182::DEBUG::2013-11-20 03:25:16,103::task::495::TaskManager.Task::(_debug) Task f30dbafe-bf59-4032-bb72-ea9a45eac8cc: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914182::DEBUG::2013-11-20 03:25:16,104::task::495::TaskManager.Task::(_debug) Task f30dbafe-bf59-4032-bb72-ea9a45eac8cc: moving from state preparing -> state finished >Thread-15914182::DEBUG::2013-11-20 03:25:16,105::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914182::DEBUG::2013-11-20 03:25:16,105::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914182::DEBUG::2013-11-20 03:25:16,106::task::495::TaskManager.Task::(_debug) Task f30dbafe-bf59-4032-bb72-ea9a45eac8cc: ref 0 aborting False >Thread-15914182::INFO::2013-11-20 03:25:16,107::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914183::DEBUG::2013-11-20 03:25:16,111::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914183::INFO::2013-11-20 03:25:16,111::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914183::DEBUG::2013-11-20 03:25:16,112::task::495::TaskManager.Task::(_debug) Task 803d5f8e-2d47-4040-b02a-e0a14ecebc15: moving from state init -> state preparing >Thread-15914183::DEBUG::2013-11-20 03:25:16,112::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`849d1137-1d75-459b-b248-ce459f74760d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914183::DEBUG::2013-11-20 03:25:16,113::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914183::DEBUG::2013-11-20 03:25:16,113::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914183::DEBUG::2013-11-20 03:25:16,114::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`849d1137-1d75-459b-b248-ce459f74760d`::Granted request >Thread-15914183::DEBUG::2013-11-20 03:25:16,114::task::495::TaskManager.Task::(_debug) Task 803d5f8e-2d47-4040-b02a-e0a14ecebc15: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914183::DEBUG::2013-11-20 03:25:16,114::task::495::TaskManager.Task::(_debug) Task 803d5f8e-2d47-4040-b02a-e0a14ecebc15: ref 1 aborting False >Thread-15914183::WARNING::2013-11-20 03:25:16,116::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914183::DEBUG::2013-11-20 03:25:16,116::task::495::TaskManager.Task::(_debug) Task 803d5f8e-2d47-4040-b02a-e0a14ecebc15: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914183::DEBUG::2013-11-20 03:25:16,117::task::495::TaskManager.Task::(_debug) Task 803d5f8e-2d47-4040-b02a-e0a14ecebc15: moving from state preparing -> state finished >Thread-15914183::DEBUG::2013-11-20 03:25:16,117::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914183::DEBUG::2013-11-20 03:25:16,118::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914183::DEBUG::2013-11-20 03:25:16,118::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914183::DEBUG::2013-11-20 03:25:16,119::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914183::DEBUG::2013-11-20 03:25:16,119::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914183::DEBUG::2013-11-20 03:25:16,119::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914183::DEBUG::2013-11-20 03:25:16,120::task::495::TaskManager.Task::(_debug) Task 803d5f8e-2d47-4040-b02a-e0a14ecebc15: ref 0 aborting False >Thread-15914183::INFO::2013-11-20 03:25:16,120::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:25:17,206::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:17,590::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0447946 s, 22.9 MB/s\n'; <rc> = 0 >Thread-15914184::INFO::2013-11-20 03:25:17,956::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914184::DEBUG::2013-11-20 03:25:17,956::task::495::TaskManager.Task::(_debug) Task f2d91abe-211e-477c-9682-a617508f80a5: moving from state init -> state preparing >Thread-15914184::DEBUG::2013-11-20 03:25:17,957::task::495::TaskManager.Task::(_debug) Task f2d91abe-211e-477c-9682-a617508f80a5: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000247955322266', 'lastCheck': 1384917911.5935581, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00114488601685', 'lastCheck': 1384917917.9459331, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00110197067261', 'lastCheck': 1384917910.429177, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00131106376648', 'lastCheck': 1384917910.9349611, 'valid': True, 'code': 0}} >Thread-15914184::DEBUG::2013-11-20 03:25:17,958::task::495::TaskManager.Task::(_debug) Task f2d91abe-211e-477c-9682-a617508f80a5: moving from state preparing -> state finished >Thread-15914184::DEBUG::2013-11-20 03:25:17,958::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914184::DEBUG::2013-11-20 03:25:17,958::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914184::DEBUG::2013-11-20 03:25:17,959::task::495::TaskManager.Task::(_debug) Task f2d91abe-211e-477c-9682-a617508f80a5: ref 0 aborting False >Thread-15914184::INFO::2013-11-20 03:25:17,959::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000247955322266', 'lastCheck': 1384917911.5935581, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00114488601685', 'lastCheck': 1384917917.9459331, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00110197067261', 'lastCheck': 1384917910.429177, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00131106376648', 'lastCheck': 1384917910.9349611, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:25:19,601::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:20,024::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.084817 s, 12.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:25:22,034::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:22,420::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0488885 s, 20.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:25:24,431::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:24,925::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0874095 s, 11.7 MB/s\n'; <rc> = 0 >Thread-15913639::INFO::2013-11-20 03:25:25,405::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:25:25,406::task::495::TaskManager.Task::(_debug) Task c6eb6704-faf0-4ad5-9c1f-66b67c5bcb31: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:25:25,408::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`6922a98f-58c2-4165-a149-450c7a9601d2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:25:25,409::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:25:25,410::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:25:25,411::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`6922a98f-58c2-4165-a149-450c7a9601d2`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:25:25,412::task::495::TaskManager.Task::(_debug) Task c6eb6704-faf0-4ad5-9c1f-66b67c5bcb31: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:25:25,413::task::495::TaskManager.Task::(_debug) Task c6eb6704-faf0-4ad5-9c1f-66b67c5bcb31: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:25:25,414::task::495::TaskManager.Task::(_debug) Task c6eb6704-faf0-4ad5-9c1f-66b67c5bcb31: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:25:25,415::task::495::TaskManager.Task::(_debug) Task c6eb6704-faf0-4ad5-9c1f-66b67c5bcb31: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:25:25,415::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:25:25,416::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:25:25,417::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:25:25,418::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:25:25,418::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:25:25,419::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:25:25,420::task::495::TaskManager.Task::(_debug) Task c6eb6704-faf0-4ad5-9c1f-66b67c5bcb31: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:25:25,420::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15914188::DEBUG::2013-11-20 03:25:26,769::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914188::INFO::2013-11-20 03:25:26,770::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914188::DEBUG::2013-11-20 03:25:26,770::task::495::TaskManager.Task::(_debug) Task 87f2daf2-b02c-48bc-898f-39523ba97365: moving from state init -> state preparing >Thread-15914188::DEBUG::2013-11-20 03:25:26,771::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914188::DEBUG::2013-11-20 03:25:26,771::task::495::TaskManager.Task::(_debug) Task 87f2daf2-b02c-48bc-898f-39523ba97365: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914188::DEBUG::2013-11-20 03:25:26,772::task::495::TaskManager.Task::(_debug) Task 87f2daf2-b02c-48bc-898f-39523ba97365: moving from state preparing -> state finished >Thread-15914188::DEBUG::2013-11-20 03:25:26,772::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914188::DEBUG::2013-11-20 03:25:26,772::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914188::DEBUG::2013-11-20 03:25:26,773::task::495::TaskManager.Task::(_debug) Task 87f2daf2-b02c-48bc-898f-39523ba97365: ref 0 aborting False >Thread-15914188::INFO::2013-11-20 03:25:26,773::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914189::DEBUG::2013-11-20 03:25:26,776::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914189::INFO::2013-11-20 03:25:26,777::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914189::DEBUG::2013-11-20 03:25:26,777::task::495::TaskManager.Task::(_debug) Task 7bf15b61-8265-4084-b2d6-44384af07a0c: moving from state init -> state preparing >Thread-15914189::DEBUG::2013-11-20 03:25:26,778::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`17070512-edee-4218-ae83-d3f1f91e81c0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914189::DEBUG::2013-11-20 03:25:26,778::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914189::DEBUG::2013-11-20 03:25:26,778::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914189::DEBUG::2013-11-20 03:25:26,779::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`17070512-edee-4218-ae83-d3f1f91e81c0`::Granted request >Thread-15914189::DEBUG::2013-11-20 03:25:26,779::task::495::TaskManager.Task::(_debug) Task 7bf15b61-8265-4084-b2d6-44384af07a0c: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914189::DEBUG::2013-11-20 03:25:26,780::task::495::TaskManager.Task::(_debug) Task 7bf15b61-8265-4084-b2d6-44384af07a0c: ref 1 aborting False >Thread-15914189::WARNING::2013-11-20 03:25:26,781::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914189::DEBUG::2013-11-20 03:25:26,781::task::495::TaskManager.Task::(_debug) Task 7bf15b61-8265-4084-b2d6-44384af07a0c: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914189::DEBUG::2013-11-20 03:25:26,782::task::495::TaskManager.Task::(_debug) Task 7bf15b61-8265-4084-b2d6-44384af07a0c: moving from state preparing -> state finished >Thread-15914189::DEBUG::2013-11-20 03:25:26,782::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914189::DEBUG::2013-11-20 03:25:26,782::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914189::DEBUG::2013-11-20 03:25:26,783::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914189::DEBUG::2013-11-20 03:25:26,783::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914189::DEBUG::2013-11-20 03:25:26,783::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914189::DEBUG::2013-11-20 03:25:26,784::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914189::DEBUG::2013-11-20 03:25:26,784::task::495::TaskManager.Task::(_debug) Task 7bf15b61-8265-4084-b2d6-44384af07a0c: ref 0 aborting False >Thread-15914189::INFO::2013-11-20 03:25:26,785::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:25:26,941::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:27,462::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0792325 s, 12.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:25:29,474::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:29,962::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0702964 s, 14.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:25:31,976::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:32,420::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0713112 s, 14.4 MB/s\n'; <rc> = 0 >Thread-15914192::INFO::2013-11-20 03:25:33,881::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914192::DEBUG::2013-11-20 03:25:33,882::task::495::TaskManager.Task::(_debug) Task 8db60b99-0bc1-48f4-813a-539d01dcc3e8: moving from state init -> state preparing >Thread-15914192::DEBUG::2013-11-20 03:25:33,883::task::495::TaskManager.Task::(_debug) Task 8db60b99-0bc1-48f4-813a-539d01dcc3e8: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000260829925537', 'lastCheck': 1384917931.6539071, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00118589401245', 'lastCheck': 1384917928.1423759, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.001051902771', 'lastCheck': 1384917930.6570599, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000253915786743', 'lastCheck': 1384917931.1412001, 'valid': True, 'code': 0}} >Thread-15914192::DEBUG::2013-11-20 03:25:33,884::task::495::TaskManager.Task::(_debug) Task 8db60b99-0bc1-48f4-813a-539d01dcc3e8: moving from state preparing -> state finished >Thread-15914192::DEBUG::2013-11-20 03:25:33,885::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914192::DEBUG::2013-11-20 03:25:33,885::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914192::DEBUG::2013-11-20 03:25:33,886::task::495::TaskManager.Task::(_debug) Task 8db60b99-0bc1-48f4-813a-539d01dcc3e8: ref 0 aborting False >Thread-15914192::INFO::2013-11-20 03:25:33,886::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000260829925537', 'lastCheck': 1384917931.6539071, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00118589401245', 'lastCheck': 1384917928.1423759, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.001051902771', 'lastCheck': 1384917930.6570599, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000253915786743', 'lastCheck': 1384917931.1412001, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:25:34,435::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:34,877::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.045375 s, 22.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:25:36,888::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914194::DEBUG::2013-11-20 03:25:37,072::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914194::INFO::2013-11-20 03:25:37,074::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914194::DEBUG::2013-11-20 03:25:37,075::task::495::TaskManager.Task::(_debug) Task 534df733-84c4-43da-969a-623012dd8104: moving from state init -> state preparing >Thread-15914194::DEBUG::2013-11-20 03:25:37,076::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914194::DEBUG::2013-11-20 03:25:37,076::task::495::TaskManager.Task::(_debug) Task 534df733-84c4-43da-969a-623012dd8104: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914194::DEBUG::2013-11-20 03:25:37,077::task::495::TaskManager.Task::(_debug) Task 534df733-84c4-43da-969a-623012dd8104: moving from state preparing -> state finished >Thread-15914194::DEBUG::2013-11-20 03:25:37,078::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914194::DEBUG::2013-11-20 03:25:37,078::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914194::DEBUG::2013-11-20 03:25:37,079::task::495::TaskManager.Task::(_debug) Task 534df733-84c4-43da-969a-623012dd8104: ref 0 aborting False >Thread-15914194::INFO::2013-11-20 03:25:37,079::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914195::DEBUG::2013-11-20 03:25:37,083::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914195::INFO::2013-11-20 03:25:37,084::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914195::DEBUG::2013-11-20 03:25:37,084::task::495::TaskManager.Task::(_debug) Task 4423a692-8f49-494b-83bb-bdd4742ade9c: moving from state init -> state preparing >Thread-15914195::DEBUG::2013-11-20 03:25:37,085::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`68102de6-27a2-426d-83c9-477f80074f8b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914195::DEBUG::2013-11-20 03:25:37,085::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914195::DEBUG::2013-11-20 03:25:37,086::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914195::DEBUG::2013-11-20 03:25:37,086::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`68102de6-27a2-426d-83c9-477f80074f8b`::Granted request >Thread-15914195::DEBUG::2013-11-20 03:25:37,087::task::495::TaskManager.Task::(_debug) Task 4423a692-8f49-494b-83bb-bdd4742ade9c: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914195::DEBUG::2013-11-20 03:25:37,088::task::495::TaskManager.Task::(_debug) Task 4423a692-8f49-494b-83bb-bdd4742ade9c: ref 1 aborting False >Thread-15914195::WARNING::2013-11-20 03:25:37,089::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914195::DEBUG::2013-11-20 03:25:37,090::task::495::TaskManager.Task::(_debug) Task 4423a692-8f49-494b-83bb-bdd4742ade9c: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914195::DEBUG::2013-11-20 03:25:37,091::task::495::TaskManager.Task::(_debug) Task 4423a692-8f49-494b-83bb-bdd4742ade9c: moving from state preparing -> state finished >Thread-15914195::DEBUG::2013-11-20 03:25:37,092::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914195::DEBUG::2013-11-20 03:25:37,092::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914195::DEBUG::2013-11-20 03:25:37,093::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914195::DEBUG::2013-11-20 03:25:37,094::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914195::DEBUG::2013-11-20 03:25:37,094::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914195::DEBUG::2013-11-20 03:25:37,095::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914195::DEBUG::2013-11-20 03:25:37,095::task::495::TaskManager.Task::(_debug) Task 4423a692-8f49-494b-83bb-bdd4742ade9c: ref 0 aborting False >Thread-15914195::INFO::2013-11-20 03:25:37,096::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:25:37,365::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0657327 s, 15.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:25:39,377::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:39,942::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0887424 s, 11.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:25:41,956::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:42,500::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0598083 s, 17.1 MB/s\n'; <rc> = 0 >Thread-15913695::INFO::2013-11-20 03:25:43,211::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:25:43,211::task::495::TaskManager.Task::(_debug) Task ecea87f6-1929-493d-91c9-6a1b97c0d6ea: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:25:43,213::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0a260628-0b3d-4a9f-b435-3cfbe99e6e9d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:25:43,214::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:25:43,215::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:25:43,215::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0a260628-0b3d-4a9f-b435-3cfbe99e6e9d`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:25:43,216::task::495::TaskManager.Task::(_debug) Task ecea87f6-1929-493d-91c9-6a1b97c0d6ea: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:25:43,217::task::495::TaskManager.Task::(_debug) Task ecea87f6-1929-493d-91c9-6a1b97c0d6ea: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:25:43,218::task::495::TaskManager.Task::(_debug) Task ecea87f6-1929-493d-91c9-6a1b97c0d6ea: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:25:43,218::task::495::TaskManager.Task::(_debug) Task ecea87f6-1929-493d-91c9-6a1b97c0d6ea: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:25:43,219::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:25:43,220::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:25:43,220::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:25:43,221::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:25:43,222::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:25:43,222::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:25:43,223::task::495::TaskManager.Task::(_debug) Task ecea87f6-1929-493d-91c9-6a1b97c0d6ea: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:25:43,224::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:25:44,512::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:45,099::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0798212 s, 12.8 MB/s\n'; <rc> = 0 >Thread-15913658::INFO::2013-11-20 03:25:45,606::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:25:45,607::task::495::TaskManager.Task::(_debug) Task 0546d03e-f502-4262-96f6-3a5dcb5ef185: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:25:45,609::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`fa2e8d4a-fbff-4fd8-abe0-0887b04ae618`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:25:45,610::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:25:45,611::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:25:45,612::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`fa2e8d4a-fbff-4fd8-abe0-0887b04ae618`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:25:45,614::task::495::TaskManager.Task::(_debug) Task 0546d03e-f502-4262-96f6-3a5dcb5ef185: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:25:45,614::task::495::TaskManager.Task::(_debug) Task 0546d03e-f502-4262-96f6-3a5dcb5ef185: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:25:45,615::task::495::TaskManager.Task::(_debug) Task 0546d03e-f502-4262-96f6-3a5dcb5ef185: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:25:45,616::task::495::TaskManager.Task::(_debug) Task 0546d03e-f502-4262-96f6-3a5dcb5ef185: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:25:45,617::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:25:45,617::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:25:45,618::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:25:45,619::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:25:45,619::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:25:45,620::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:25:45,620::task::495::TaskManager.Task::(_debug) Task 0546d03e-f502-4262-96f6-3a5dcb5ef185: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:25:45,621::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:25:47,117::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:47,536::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0573256 s, 17.9 MB/s\n'; <rc> = 0 >Thread-15914200::DEBUG::2013-11-20 03:25:48,507::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914200::INFO::2013-11-20 03:25:48,508::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914200::DEBUG::2013-11-20 03:25:48,509::task::495::TaskManager.Task::(_debug) Task f8c3d8b2-9c58-47b6-924d-f99648f487a1: moving from state init -> state preparing >Thread-15914200::DEBUG::2013-11-20 03:25:48,510::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914200::DEBUG::2013-11-20 03:25:48,510::task::495::TaskManager.Task::(_debug) Task f8c3d8b2-9c58-47b6-924d-f99648f487a1: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914200::DEBUG::2013-11-20 03:25:48,511::task::495::TaskManager.Task::(_debug) Task f8c3d8b2-9c58-47b6-924d-f99648f487a1: moving from state preparing -> state finished >Thread-15914200::DEBUG::2013-11-20 03:25:48,511::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914200::DEBUG::2013-11-20 03:25:48,512::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914200::DEBUG::2013-11-20 03:25:48,513::task::495::TaskManager.Task::(_debug) Task f8c3d8b2-9c58-47b6-924d-f99648f487a1: ref 0 aborting False >Thread-15914200::INFO::2013-11-20 03:25:48,513::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914201::DEBUG::2013-11-20 03:25:48,517::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914201::INFO::2013-11-20 03:25:48,518::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914201::DEBUG::2013-11-20 03:25:48,519::task::495::TaskManager.Task::(_debug) Task c112046d-0995-4876-ae39-57400cc29044: moving from state init -> state preparing >Thread-15914201::DEBUG::2013-11-20 03:25:48,520::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`49518d10-e827-4bbc-b130-8ee1aba0189c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914201::DEBUG::2013-11-20 03:25:48,521::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914201::DEBUG::2013-11-20 03:25:48,522::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914201::DEBUG::2013-11-20 03:25:48,522::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`49518d10-e827-4bbc-b130-8ee1aba0189c`::Granted request >Thread-15914201::DEBUG::2013-11-20 03:25:48,523::task::495::TaskManager.Task::(_debug) Task c112046d-0995-4876-ae39-57400cc29044: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914201::DEBUG::2013-11-20 03:25:48,524::task::495::TaskManager.Task::(_debug) Task c112046d-0995-4876-ae39-57400cc29044: ref 1 aborting False >Thread-15914201::WARNING::2013-11-20 03:25:48,526::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914201::DEBUG::2013-11-20 03:25:48,527::task::495::TaskManager.Task::(_debug) Task c112046d-0995-4876-ae39-57400cc29044: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914201::DEBUG::2013-11-20 03:25:48,528::task::495::TaskManager.Task::(_debug) Task c112046d-0995-4876-ae39-57400cc29044: moving from state preparing -> state finished >Thread-15914201::DEBUG::2013-11-20 03:25:48,529::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914201::DEBUG::2013-11-20 03:25:48,529::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914201::DEBUG::2013-11-20 03:25:48,530::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914201::DEBUG::2013-11-20 03:25:48,530::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914201::DEBUG::2013-11-20 03:25:48,531::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914201::DEBUG::2013-11-20 03:25:48,531::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914201::DEBUG::2013-11-20 03:25:48,532::task::495::TaskManager.Task::(_debug) Task c112046d-0995-4876-ae39-57400cc29044: ref 0 aborting False >Thread-15914201::INFO::2013-11-20 03:25:48,532::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:25:49,547::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914202::INFO::2013-11-20 03:25:49,761::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914202::DEBUG::2013-11-20 03:25:49,762::task::495::TaskManager.Task::(_debug) Task ddf2d631-ad0d-4707-87aa-7db96f8b2bef: moving from state init -> state preparing >Thread-15914202::DEBUG::2013-11-20 03:25:49,762::task::495::TaskManager.Task::(_debug) Task ddf2d631-ad0d-4707-87aa-7db96f8b2bef: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000285863876343', 'lastCheck': 1384917941.668704, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00100803375244', 'lastCheck': 1384917948.2951529, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00147485733032', 'lastCheck': 1384917940.6751621, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000445127487183', 'lastCheck': 1384917941.1554401, 'valid': True, 'code': 0}} >Thread-15914202::DEBUG::2013-11-20 03:25:49,763::task::495::TaskManager.Task::(_debug) Task ddf2d631-ad0d-4707-87aa-7db96f8b2bef: moving from state preparing -> state finished >Thread-15914202::DEBUG::2013-11-20 03:25:49,763::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914202::DEBUG::2013-11-20 03:25:49,764::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914202::DEBUG::2013-11-20 03:25:49,764::task::495::TaskManager.Task::(_debug) Task ddf2d631-ad0d-4707-87aa-7db96f8b2bef: ref 0 aborting False >Thread-15914202::INFO::2013-11-20 03:25:49,765::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000285863876343', 'lastCheck': 1384917941.668704, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00100803375244', 'lastCheck': 1384917948.2951529, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00147485733032', 'lastCheck': 1384917940.6751621, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000445127487183', 'lastCheck': 1384917941.1554401, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:25:49,984::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.048866 s, 21.0 MB/s\n'; <rc> = 0 >Thread-15912392::INFO::2013-11-20 03:25:50,734::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:25:50,735::task::495::TaskManager.Task::(_debug) Task 3ae79593-f7d1-4054-9107-957e645984a2: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:25:50,736::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f5b33a9e-1418-4b65-88b8-ebd2db18169f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:25:50,736::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:25:50,736::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:25:50,737::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f5b33a9e-1418-4b65-88b8-ebd2db18169f`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:25:50,737::task::495::TaskManager.Task::(_debug) Task 3ae79593-f7d1-4054-9107-957e645984a2: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:25:50,738::task::495::TaskManager.Task::(_debug) Task 3ae79593-f7d1-4054-9107-957e645984a2: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:25:50,738::task::495::TaskManager.Task::(_debug) Task 3ae79593-f7d1-4054-9107-957e645984a2: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:25:50,738::task::495::TaskManager.Task::(_debug) Task 3ae79593-f7d1-4054-9107-957e645984a2: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:25:50,739::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:25:50,739::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:25:50,739::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:25:50,740::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:25:50,741::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:25:50,742::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:25:50,742::task::495::TaskManager.Task::(_debug) Task 3ae79593-f7d1-4054-9107-957e645984a2: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:25:50,743::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:25:51,994::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:52,591::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0859061 s, 11.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:25:54,600::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:55,040::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0478088 s, 21.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:25:57,050::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:57,524::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0854156 s, 12.0 MB/s\n'; <rc> = 0 >Thread-15914206::DEBUG::2013-11-20 03:25:58,820::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914206::INFO::2013-11-20 03:25:58,821::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914206::DEBUG::2013-11-20 03:25:58,821::task::495::TaskManager.Task::(_debug) Task 693ec483-1cac-464f-a324-7f82e3ab7204: moving from state init -> state preparing >Thread-15914206::DEBUG::2013-11-20 03:25:58,822::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914206::DEBUG::2013-11-20 03:25:58,822::task::495::TaskManager.Task::(_debug) Task 693ec483-1cac-464f-a324-7f82e3ab7204: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914206::DEBUG::2013-11-20 03:25:58,823::task::495::TaskManager.Task::(_debug) Task 693ec483-1cac-464f-a324-7f82e3ab7204: moving from state preparing -> state finished >Thread-15914206::DEBUG::2013-11-20 03:25:58,823::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914206::DEBUG::2013-11-20 03:25:58,823::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914206::DEBUG::2013-11-20 03:25:58,824::task::495::TaskManager.Task::(_debug) Task 693ec483-1cac-464f-a324-7f82e3ab7204: ref 0 aborting False >Thread-15914206::INFO::2013-11-20 03:25:58,824::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914207::DEBUG::2013-11-20 03:25:58,828::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914207::INFO::2013-11-20 03:25:58,828::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914207::DEBUG::2013-11-20 03:25:58,829::task::495::TaskManager.Task::(_debug) Task ea4bfaf8-c00d-4ed6-ad93-4f38d7e17f51: moving from state init -> state preparing >Thread-15914207::DEBUG::2013-11-20 03:25:58,829::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`abf7bcc5-8d0c-4194-ba23-d930b972cb93`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914207::DEBUG::2013-11-20 03:25:58,830::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914207::DEBUG::2013-11-20 03:25:58,830::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914207::DEBUG::2013-11-20 03:25:58,831::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`abf7bcc5-8d0c-4194-ba23-d930b972cb93`::Granted request >Thread-15914207::DEBUG::2013-11-20 03:25:58,831::task::495::TaskManager.Task::(_debug) Task ea4bfaf8-c00d-4ed6-ad93-4f38d7e17f51: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914207::DEBUG::2013-11-20 03:25:58,832::task::495::TaskManager.Task::(_debug) Task ea4bfaf8-c00d-4ed6-ad93-4f38d7e17f51: ref 1 aborting False >Thread-15914207::WARNING::2013-11-20 03:25:58,833::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914207::DEBUG::2013-11-20 03:25:58,833::task::495::TaskManager.Task::(_debug) Task ea4bfaf8-c00d-4ed6-ad93-4f38d7e17f51: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914207::DEBUG::2013-11-20 03:25:58,834::task::495::TaskManager.Task::(_debug) Task ea4bfaf8-c00d-4ed6-ad93-4f38d7e17f51: moving from state preparing -> state finished >Thread-15914207::DEBUG::2013-11-20 03:25:58,834::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914207::DEBUG::2013-11-20 03:25:58,834::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914207::DEBUG::2013-11-20 03:25:58,835::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914207::DEBUG::2013-11-20 03:25:58,835::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914207::DEBUG::2013-11-20 03:25:58,836::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914207::DEBUG::2013-11-20 03:25:58,836::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914207::DEBUG::2013-11-20 03:25:58,836::task::495::TaskManager.Task::(_debug) Task ea4bfaf8-c00d-4ed6-ad93-4f38d7e17f51: ref 0 aborting False >Thread-15914207::INFO::2013-11-20 03:25:58,837::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:25:59,540::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:25:59,907::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0460046 s, 22.3 MB/s\n'; <rc> = 0 >Thread-15913518::INFO::2013-11-20 03:26:00,581::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:26:00,581::task::495::TaskManager.Task::(_debug) Task 6e0efeb8-d144-4d57-8446-100400e18fda: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:26:00,583::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0f269f2b-b10e-4fab-a051-6a345e3d7490`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:26:00,583::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:26:00,584::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:26:00,584::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0f269f2b-b10e-4fab-a051-6a345e3d7490`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:26:00,585::task::495::TaskManager.Task::(_debug) Task 6e0efeb8-d144-4d57-8446-100400e18fda: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:26:00,585::task::495::TaskManager.Task::(_debug) Task 6e0efeb8-d144-4d57-8446-100400e18fda: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:26:00,586::task::495::TaskManager.Task::(_debug) Task 6e0efeb8-d144-4d57-8446-100400e18fda: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:26:00,586::task::495::TaskManager.Task::(_debug) Task 6e0efeb8-d144-4d57-8446-100400e18fda: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:26:00,586::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:26:00,587::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:26:00,587::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:26:00,588::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:26:00,588::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:26:00,588::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:26:00,589::task::495::TaskManager.Task::(_debug) Task 6e0efeb8-d144-4d57-8446-100400e18fda: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:26:00,589::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:26:00,590::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:26:00,590::task::495::TaskManager.Task::(_debug) Task e35924e3-acbc-4530-985a-e4b3df1ce9d9: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:26:00,591::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0d8fbc5b-bd75-42db-b99e-a692ef625fe6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:26:00,592::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:26:00,592::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:26:00,593::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0d8fbc5b-bd75-42db-b99e-a692ef625fe6`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:26:00,593::task::495::TaskManager.Task::(_debug) Task e35924e3-acbc-4530-985a-e4b3df1ce9d9: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:26:00,594::task::495::TaskManager.Task::(_debug) Task e35924e3-acbc-4530-985a-e4b3df1ce9d9: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:26:00,594::task::495::TaskManager.Task::(_debug) Task e35924e3-acbc-4530-985a-e4b3df1ce9d9: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:26:00,595::task::495::TaskManager.Task::(_debug) Task e35924e3-acbc-4530-985a-e4b3df1ce9d9: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:26:00,595::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:26:00,595::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:26:00,596::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:26:00,596::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:26:00,597::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:26:00,597::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:26:00,598::task::495::TaskManager.Task::(_debug) Task e35924e3-acbc-4530-985a-e4b3df1ce9d9: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:26:00,598::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:26:01,917::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:26:02,323::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0521208 s, 19.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:26:04,337::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:26:04,727::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0549529 s, 18.6 MB/s\n'; <rc> = 0 >Thread-15914210::INFO::2013-11-20 03:26:05,200::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914210::DEBUG::2013-11-20 03:26:05,201::task::495::TaskManager.Task::(_debug) Task 096ae64f-e37f-42d2-be8a-3c1bb644fe91: moving from state init -> state preparing >Thread-15914210::DEBUG::2013-11-20 03:26:05,201::task::495::TaskManager.Task::(_debug) Task 096ae64f-e37f-42d2-be8a-3c1bb644fe91: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000313997268677', 'lastCheck': 1384917961.813659, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0109629631042', 'lastCheck': 1384917958.319757, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00120091438293', 'lastCheck': 1384917960.8119359, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000341892242432', 'lastCheck': 1384917961.301254, 'valid': True, 'code': 0}} >Thread-15914210::DEBUG::2013-11-20 03:26:05,202::task::495::TaskManager.Task::(_debug) Task 096ae64f-e37f-42d2-be8a-3c1bb644fe91: moving from state preparing -> state finished >Thread-15914210::DEBUG::2013-11-20 03:26:05,202::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914210::DEBUG::2013-11-20 03:26:05,202::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914210::DEBUG::2013-11-20 03:26:05,203::task::495::TaskManager.Task::(_debug) Task 096ae64f-e37f-42d2-be8a-3c1bb644fe91: ref 0 aborting False >Thread-15914210::INFO::2013-11-20 03:26:05,203::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000313997268677', 'lastCheck': 1384917961.813659, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0109629631042', 'lastCheck': 1384917958.319757, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00120091438293', 'lastCheck': 1384917960.8119359, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000341892242432', 'lastCheck': 1384917961.301254, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:26:06,743::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:26:07,235::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0646625 s, 15.8 MB/s\n'; <rc> = 0 >Thread-15914213::DEBUG::2013-11-20 03:26:09,084::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914213::INFO::2013-11-20 03:26:09,085::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914213::DEBUG::2013-11-20 03:26:09,085::task::495::TaskManager.Task::(_debug) Task b6abbe36-4eca-42f5-966b-9d9b6fa3ac89: moving from state init -> state preparing >Thread-15914213::DEBUG::2013-11-20 03:26:09,086::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914213::DEBUG::2013-11-20 03:26:09,086::task::495::TaskManager.Task::(_debug) Task b6abbe36-4eca-42f5-966b-9d9b6fa3ac89: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914213::DEBUG::2013-11-20 03:26:09,087::task::495::TaskManager.Task::(_debug) Task b6abbe36-4eca-42f5-966b-9d9b6fa3ac89: moving from state preparing -> state finished >Thread-15914213::DEBUG::2013-11-20 03:26:09,087::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914213::DEBUG::2013-11-20 03:26:09,087::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914213::DEBUG::2013-11-20 03:26:09,088::task::495::TaskManager.Task::(_debug) Task b6abbe36-4eca-42f5-966b-9d9b6fa3ac89: ref 0 aborting False >Thread-15914213::INFO::2013-11-20 03:26:09,088::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914214::DEBUG::2013-11-20 03:26:09,092::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914214::INFO::2013-11-20 03:26:09,093::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914214::DEBUG::2013-11-20 03:26:09,094::task::495::TaskManager.Task::(_debug) Task 3c4fe439-58b0-4721-b549-1e3bac9f34dd: moving from state init -> state preparing >Thread-15914214::DEBUG::2013-11-20 03:26:09,095::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`668e7355-ee53-48d9-a346-cea9a928ae0e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914214::DEBUG::2013-11-20 03:26:09,095::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914214::DEBUG::2013-11-20 03:26:09,096::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914214::DEBUG::2013-11-20 03:26:09,097::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`668e7355-ee53-48d9-a346-cea9a928ae0e`::Granted request >Thread-15914214::DEBUG::2013-11-20 03:26:09,098::task::495::TaskManager.Task::(_debug) Task 3c4fe439-58b0-4721-b549-1e3bac9f34dd: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914214::DEBUG::2013-11-20 03:26:09,098::task::495::TaskManager.Task::(_debug) Task 3c4fe439-58b0-4721-b549-1e3bac9f34dd: ref 1 aborting False >Thread-15914214::WARNING::2013-11-20 03:26:09,100::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914214::DEBUG::2013-11-20 03:26:09,101::task::495::TaskManager.Task::(_debug) Task 3c4fe439-58b0-4721-b549-1e3bac9f34dd: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914214::DEBUG::2013-11-20 03:26:09,102::task::495::TaskManager.Task::(_debug) Task 3c4fe439-58b0-4721-b549-1e3bac9f34dd: moving from state preparing -> state finished >Thread-15914214::DEBUG::2013-11-20 03:26:09,102::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914214::DEBUG::2013-11-20 03:26:09,103::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914214::DEBUG::2013-11-20 03:26:09,103::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914214::DEBUG::2013-11-20 03:26:09,104::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914214::DEBUG::2013-11-20 03:26:09,105::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914214::DEBUG::2013-11-20 03:26:09,105::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914214::DEBUG::2013-11-20 03:26:09,106::task::495::TaskManager.Task::(_debug) Task 3c4fe439-58b0-4721-b549-1e3bac9f34dd: ref 0 aborting False >Thread-15914214::INFO::2013-11-20 03:26:09,106::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:26:09,249::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:26:09,745::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0847992 s, 12.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:26:11,758::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:26:12,266::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0764994 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:26:14,282::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:26:14,823::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0817977 s, 12.5 MB/s\n'; <rc> = 0 >Thread-15913546::INFO::2013-11-20 03:26:15,964::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:26:15,965::task::495::TaskManager.Task::(_debug) Task e362fa8a-298b-4186-a1de-ee3272445db5: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:26:15,968::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`9527fbd8-1363-4ef6-ac4f-0e1c2615690e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:26:15,968::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:26:15,969::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:26:15,970::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`9527fbd8-1363-4ef6-ac4f-0e1c2615690e`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:26:15,971::task::495::TaskManager.Task::(_debug) Task e362fa8a-298b-4186-a1de-ee3272445db5: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:26:15,972::task::495::TaskManager.Task::(_debug) Task e362fa8a-298b-4186-a1de-ee3272445db5: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:26:15,973::task::495::TaskManager.Task::(_debug) Task e362fa8a-298b-4186-a1de-ee3272445db5: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:26:15,973::task::495::TaskManager.Task::(_debug) Task e362fa8a-298b-4186-a1de-ee3272445db5: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:26:15,974::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:26:15,974::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:26:15,975::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:26:15,976::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:26:15,976::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:26:15,977::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:26:15,978::task::495::TaskManager.Task::(_debug) Task e362fa8a-298b-4186-a1de-ee3272445db5: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:26:15,978::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:26:15,979::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:26:15,980::task::495::TaskManager.Task::(_debug) Task 32cc8b85-8e09-4b56-8209-e10893b8e494: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:26:15,981::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`3289d277-10c6-409d-a879-29519daf4ec2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:26:15,981::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:26:15,982::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:26:15,983::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`3289d277-10c6-409d-a879-29519daf4ec2`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:26:15,984::task::495::TaskManager.Task::(_debug) Task 32cc8b85-8e09-4b56-8209-e10893b8e494: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:26:15,984::task::495::TaskManager.Task::(_debug) Task 32cc8b85-8e09-4b56-8209-e10893b8e494: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:26:15,985::task::495::TaskManager.Task::(_debug) Task 32cc8b85-8e09-4b56-8209-e10893b8e494: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:26:15,986::task::495::TaskManager.Task::(_debug) Task 32cc8b85-8e09-4b56-8209-e10893b8e494: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:26:15,986::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:26:15,987::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:26:15,988::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:26:15,988::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:26:15,989::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:26:15,989::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:26:15,990::task::495::TaskManager.Task::(_debug) Task 32cc8b85-8e09-4b56-8209-e10893b8e494: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:26:15,991::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:26:15,991::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:26:15,992::task::495::TaskManager.Task::(_debug) Task 1a8e739f-d9b7-4af3-8ccd-0eb351e61164: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:26:15,993::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`7b8abf4b-5a0f-4ce6-a2bc-d583adac623d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:26:15,994::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:26:15,994::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:26:15,995::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`7b8abf4b-5a0f-4ce6-a2bc-d583adac623d`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:26:15,996::task::495::TaskManager.Task::(_debug) Task 1a8e739f-d9b7-4af3-8ccd-0eb351e61164: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:26:15,997::task::495::TaskManager.Task::(_debug) Task 1a8e739f-d9b7-4af3-8ccd-0eb351e61164: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:26:15,997::task::495::TaskManager.Task::(_debug) Task 1a8e739f-d9b7-4af3-8ccd-0eb351e61164: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:26:15,998::task::495::TaskManager.Task::(_debug) Task 1a8e739f-d9b7-4af3-8ccd-0eb351e61164: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:26:15,998::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:26:15,999::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:26:16,000::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:26:16,000::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:26:16,001::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:26:16,001::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:26:16,002::task::495::TaskManager.Task::(_debug) Task 1a8e739f-d9b7-4af3-8ccd-0eb351e61164: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:26:16,002::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:26:16,003::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:26:16,004::task::495::TaskManager.Task::(_debug) Task db3181ee-d388-4103-a00c-2f2d19dfca76: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:26:16,005::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`0985cdb4-42f0-4fcc-84ac-29fb1e59f3bf`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:26:16,006::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:26:16,006::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:26:16,007::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`0985cdb4-42f0-4fcc-84ac-29fb1e59f3bf`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:26:16,008::task::495::TaskManager.Task::(_debug) Task db3181ee-d388-4103-a00c-2f2d19dfca76: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:26:16,008::task::495::TaskManager.Task::(_debug) Task db3181ee-d388-4103-a00c-2f2d19dfca76: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:26:16,009::task::495::TaskManager.Task::(_debug) Task db3181ee-d388-4103-a00c-2f2d19dfca76: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:26:16,010::task::495::TaskManager.Task::(_debug) Task db3181ee-d388-4103-a00c-2f2d19dfca76: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:26:16,010::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:26:16,011::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:26:16,011::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:26:16,012::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:26:16,012::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:26:16,013::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:26:16,014::task::495::TaskManager.Task::(_debug) Task db3181ee-d388-4103-a00c-2f2d19dfca76: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:26:16,014::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:26:16,838::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:26:17,280::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0538265 s, 19.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:26:19,290::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914218::DEBUG::2013-11-20 03:26:19,514::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914218::INFO::2013-11-20 03:26:19,517::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914218::DEBUG::2013-11-20 03:26:19,518::task::495::TaskManager.Task::(_debug) Task 081fb30c-d435-4db0-9675-d2a6660e3b7c: moving from state init -> state preparing >Thread-15914218::DEBUG::2013-11-20 03:26:19,519::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914218::DEBUG::2013-11-20 03:26:19,520::task::495::TaskManager.Task::(_debug) Task 081fb30c-d435-4db0-9675-d2a6660e3b7c: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914218::DEBUG::2013-11-20 03:26:19,521::task::495::TaskManager.Task::(_debug) Task 081fb30c-d435-4db0-9675-d2a6660e3b7c: moving from state preparing -> state finished >Thread-15914218::DEBUG::2013-11-20 03:26:19,522::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914218::DEBUG::2013-11-20 03:26:19,522::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914218::DEBUG::2013-11-20 03:26:19,523::task::495::TaskManager.Task::(_debug) Task 081fb30c-d435-4db0-9675-d2a6660e3b7c: ref 0 aborting False >Thread-15914218::INFO::2013-11-20 03:26:19,524::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914219::DEBUG::2013-11-20 03:26:19,529::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914219::INFO::2013-11-20 03:26:19,530::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914219::DEBUG::2013-11-20 03:26:19,530::task::495::TaskManager.Task::(_debug) Task 5acea737-22cd-4bf6-bf58-3d6c84e224ac: moving from state init -> state preparing >Thread-15914219::DEBUG::2013-11-20 03:26:19,531::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`0d38fa27-2126-4084-ab9c-12aed4f70e06`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914219::DEBUG::2013-11-20 03:26:19,532::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914219::DEBUG::2013-11-20 03:26:19,532::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914219::DEBUG::2013-11-20 03:26:19,533::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`0d38fa27-2126-4084-ab9c-12aed4f70e06`::Granted request >Thread-15914219::DEBUG::2013-11-20 03:26:19,534::task::495::TaskManager.Task::(_debug) Task 5acea737-22cd-4bf6-bf58-3d6c84e224ac: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914219::DEBUG::2013-11-20 03:26:19,534::task::495::TaskManager.Task::(_debug) Task 5acea737-22cd-4bf6-bf58-3d6c84e224ac: ref 1 aborting False >Thread-15914219::WARNING::2013-11-20 03:26:19,536::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914219::DEBUG::2013-11-20 03:26:19,537::task::495::TaskManager.Task::(_debug) Task 5acea737-22cd-4bf6-bf58-3d6c84e224ac: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914219::DEBUG::2013-11-20 03:26:19,537::task::495::TaskManager.Task::(_debug) Task 5acea737-22cd-4bf6-bf58-3d6c84e224ac: moving from state preparing -> state finished >Thread-15914219::DEBUG::2013-11-20 03:26:19,538::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914219::DEBUG::2013-11-20 03:26:19,538::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914219::DEBUG::2013-11-20 03:26:19,538::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914219::DEBUG::2013-11-20 03:26:19,539::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914219::DEBUG::2013-11-20 03:26:19,539::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914219::DEBUG::2013-11-20 03:26:19,539::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914219::DEBUG::2013-11-20 03:26:19,540::task::495::TaskManager.Task::(_debug) Task 5acea737-22cd-4bf6-bf58-3d6c84e224ac: ref 0 aborting False >Thread-15914219::INFO::2013-11-20 03:26:19,540::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:26:19,771::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0504943 s, 20.3 MB/s\n'; <rc> = 0 >Thread-15914220::INFO::2013-11-20 03:26:20,646::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914220::DEBUG::2013-11-20 03:26:20,647::task::495::TaskManager.Task::(_debug) Task a9957d1d-28e8-44f1-a7c4-328930e58236: moving from state init -> state preparing >Thread-15914220::DEBUG::2013-11-20 03:26:20,647::task::495::TaskManager.Task::(_debug) Task a9957d1d-28e8-44f1-a7c4-328930e58236: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000365018844604', 'lastCheck': 1384917971.9667611, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0106649398804', 'lastCheck': 1384917978.5083621, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0129110813141', 'lastCheck': 1384917970.9620841, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000373125076294', 'lastCheck': 1384917971.423285, 'valid': True, 'code': 0}} >Thread-15914220::DEBUG::2013-11-20 03:26:20,648::task::495::TaskManager.Task::(_debug) Task a9957d1d-28e8-44f1-a7c4-328930e58236: moving from state preparing -> state finished >Thread-15914220::DEBUG::2013-11-20 03:26:20,648::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914220::DEBUG::2013-11-20 03:26:20,648::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914220::DEBUG::2013-11-20 03:26:20,649::task::495::TaskManager.Task::(_debug) Task a9957d1d-28e8-44f1-a7c4-328930e58236: ref 0 aborting False >Thread-15914220::INFO::2013-11-20 03:26:20,649::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000365018844604', 'lastCheck': 1384917971.9667611, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0106649398804', 'lastCheck': 1384917978.5083621, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0129110813141', 'lastCheck': 1384917970.9620841, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000373125076294', 'lastCheck': 1384917971.423285, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:26:21,789::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:26:22,246::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0664918 s, 15.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:26:24,259::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:26:24,770::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0725052 s, 14.1 MB/s\n'; <rc> = 0 >Thread-15913639::INFO::2013-11-20 03:26:25,999::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:26:26,000::task::495::TaskManager.Task::(_debug) Task 8336ef99-987f-4ceb-a645-9fa9940f5685: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:26:26,002::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`fadbab66-666b-4f5a-a40a-d90923d1e231`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:26:26,002::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:26:26,003::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:26:26,003::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`fadbab66-666b-4f5a-a40a-d90923d1e231`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:26:26,004::task::495::TaskManager.Task::(_debug) Task 8336ef99-987f-4ceb-a645-9fa9940f5685: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:26:26,005::task::495::TaskManager.Task::(_debug) Task 8336ef99-987f-4ceb-a645-9fa9940f5685: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:26:26,005::task::495::TaskManager.Task::(_debug) Task 8336ef99-987f-4ceb-a645-9fa9940f5685: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:26:26,006::task::495::TaskManager.Task::(_debug) Task 8336ef99-987f-4ceb-a645-9fa9940f5685: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:26:26,006::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:26:26,006::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:26:26,007::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:26:26,007::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:26:26,008::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:26:26,008::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:26:26,008::task::495::TaskManager.Task::(_debug) Task 8336ef99-987f-4ceb-a645-9fa9940f5685: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:26:26,009::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:26:26,784::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:26:27,333::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0802852 s, 12.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:26:29,346::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914224::DEBUG::2013-11-20 03:26:29,748::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914224::INFO::2013-11-20 03:26:29,751::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914224::DEBUG::2013-11-20 03:26:29,751::task::495::TaskManager.Task::(_debug) Task 7fca5584-106f-4e96-88a2-40eb33c92871: moving from state init -> state preparing >Thread-15914224::DEBUG::2013-11-20 03:26:29,753::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914224::DEBUG::2013-11-20 03:26:29,754::task::495::TaskManager.Task::(_debug) Task 7fca5584-106f-4e96-88a2-40eb33c92871: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914224::DEBUG::2013-11-20 03:26:29,754::task::495::TaskManager.Task::(_debug) Task 7fca5584-106f-4e96-88a2-40eb33c92871: moving from state preparing -> state finished >Thread-15914224::DEBUG::2013-11-20 03:26:29,755::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914224::DEBUG::2013-11-20 03:26:29,756::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914224::DEBUG::2013-11-20 03:26:29,757::task::495::TaskManager.Task::(_debug) Task 7fca5584-106f-4e96-88a2-40eb33c92871: ref 0 aborting False >Thread-15914224::INFO::2013-11-20 03:26:29,758::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914225::DEBUG::2013-11-20 03:26:29,769::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914225::INFO::2013-11-20 03:26:29,771::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914225::DEBUG::2013-11-20 03:26:29,771::task::495::TaskManager.Task::(_debug) Task f64894ca-4c8d-44ab-a26e-d92ff0559cbf: moving from state init -> state preparing >Thread-15914225::DEBUG::2013-11-20 03:26:29,786::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`aa957722-7762-490b-9d24-eb270280794e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914225::DEBUG::2013-11-20 03:26:29,787::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914225::DEBUG::2013-11-20 03:26:29,789::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914225::DEBUG::2013-11-20 03:26:29,790::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`aa957722-7762-490b-9d24-eb270280794e`::Granted request >Thread-15914225::DEBUG::2013-11-20 03:26:29,792::task::495::TaskManager.Task::(_debug) Task f64894ca-4c8d-44ab-a26e-d92ff0559cbf: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914225::DEBUG::2013-11-20 03:26:29,793::task::495::TaskManager.Task::(_debug) Task f64894ca-4c8d-44ab-a26e-d92ff0559cbf: ref 1 aborting False >Thread-15914225::WARNING::2013-11-20 03:26:29,796::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914225::DEBUG::2013-11-20 03:26:29,797::task::495::TaskManager.Task::(_debug) Task f64894ca-4c8d-44ab-a26e-d92ff0559cbf: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914225::DEBUG::2013-11-20 03:26:29,798::task::495::TaskManager.Task::(_debug) Task f64894ca-4c8d-44ab-a26e-d92ff0559cbf: moving from state preparing -> state finished >Thread-15914225::DEBUG::2013-11-20 03:26:29,799::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914225::DEBUG::2013-11-20 03:26:29,800::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914225::DEBUG::2013-11-20 03:26:29,801::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914225::DEBUG::2013-11-20 03:26:29,802::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914225::DEBUG::2013-11-20 03:26:29,803::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914225::DEBUG::2013-11-20 03:26:29,804::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914225::DEBUG::2013-11-20 03:26:29,805::task::495::TaskManager.Task::(_debug) Task f64894ca-4c8d-44ab-a26e-d92ff0559cbf: ref 0 aborting False >Thread-15914225::INFO::2013-11-20 03:26:29,806::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:26:29,911::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.126573 s, 8.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:26:31,926::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:26:32,465::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0780003 s, 13.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:26:34,476::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:26:35,009::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0721548 s, 14.2 MB/s\n'; <rc> = 0 >Thread-15914228::INFO::2013-11-20 03:26:36,148::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914228::DEBUG::2013-11-20 03:26:36,149::task::495::TaskManager.Task::(_debug) Task 957f2ca4-e131-49a6-b47c-fc578b734fdf: moving from state init -> state preparing >Thread-15914228::DEBUG::2013-11-20 03:26:36,150::task::495::TaskManager.Task::(_debug) Task 957f2ca4-e131-49a6-b47c-fc578b734fdf: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000535011291504', 'lastCheck': 1384917992.1937079, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150990486145', 'lastCheck': 1384917988.524219, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.010822057724', 'lastCheck': 1384917991.059525, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000527858734131', 'lastCheck': 1384917991.545583, 'valid': True, 'code': 0}} >Thread-15914228::DEBUG::2013-11-20 03:26:36,150::task::495::TaskManager.Task::(_debug) Task 957f2ca4-e131-49a6-b47c-fc578b734fdf: moving from state preparing -> state finished >Thread-15914228::DEBUG::2013-11-20 03:26:36,150::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914228::DEBUG::2013-11-20 03:26:36,151::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914228::DEBUG::2013-11-20 03:26:36,151::task::495::TaskManager.Task::(_debug) Task 957f2ca4-e131-49a6-b47c-fc578b734fdf: ref 0 aborting False >Thread-15914228::INFO::2013-11-20 03:26:36,152::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000535011291504', 'lastCheck': 1384917992.1937079, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150990486145', 'lastCheck': 1384917988.524219, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.010822057724', 'lastCheck': 1384917991.059525, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000527858734131', 'lastCheck': 1384917991.545583, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:26:37,025::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:26:37,565::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0675388 s, 15.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:26:39,582::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:26:40,120::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0913017 s, 11.2 MB/s\n'; <rc> = 0 >Thread-15914231::DEBUG::2013-11-20 03:26:41,164::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914231::INFO::2013-11-20 03:26:41,166::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914231::DEBUG::2013-11-20 03:26:41,166::task::495::TaskManager.Task::(_debug) Task e681d02a-4a4b-4eb0-8ff8-074c3bef4d87: moving from state init -> state preparing >Thread-15914231::DEBUG::2013-11-20 03:26:41,169::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914231::DEBUG::2013-11-20 03:26:41,170::task::495::TaskManager.Task::(_debug) Task e681d02a-4a4b-4eb0-8ff8-074c3bef4d87: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914231::DEBUG::2013-11-20 03:26:41,174::task::495::TaskManager.Task::(_debug) Task e681d02a-4a4b-4eb0-8ff8-074c3bef4d87: moving from state preparing -> state finished >Thread-15914231::DEBUG::2013-11-20 03:26:41,178::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914231::DEBUG::2013-11-20 03:26:41,181::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914231::DEBUG::2013-11-20 03:26:41,186::task::495::TaskManager.Task::(_debug) Task e681d02a-4a4b-4eb0-8ff8-074c3bef4d87: ref 0 aborting False >Thread-15914231::INFO::2013-11-20 03:26:41,191::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914232::DEBUG::2013-11-20 03:26:41,211::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914232::INFO::2013-11-20 03:26:41,214::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914232::DEBUG::2013-11-20 03:26:41,216::task::495::TaskManager.Task::(_debug) Task 024d8dea-83da-4b5b-a500-bc5b5500d73b: moving from state init -> state preparing >Thread-15914232::DEBUG::2013-11-20 03:26:41,217::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`8a40209f-2430-469c-9d4b-bef172fb4f18`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914232::DEBUG::2013-11-20 03:26:41,218::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914232::DEBUG::2013-11-20 03:26:41,218::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914232::DEBUG::2013-11-20 03:26:41,219::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`8a40209f-2430-469c-9d4b-bef172fb4f18`::Granted request >Thread-15914232::DEBUG::2013-11-20 03:26:41,224::task::495::TaskManager.Task::(_debug) Task 024d8dea-83da-4b5b-a500-bc5b5500d73b: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914232::DEBUG::2013-11-20 03:26:41,225::task::495::TaskManager.Task::(_debug) Task 024d8dea-83da-4b5b-a500-bc5b5500d73b: ref 1 aborting False >Thread-15914232::WARNING::2013-11-20 03:26:41,227::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914232::DEBUG::2013-11-20 03:26:41,232::task::495::TaskManager.Task::(_debug) Task 024d8dea-83da-4b5b-a500-bc5b5500d73b: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914232::DEBUG::2013-11-20 03:26:41,233::task::495::TaskManager.Task::(_debug) Task 024d8dea-83da-4b5b-a500-bc5b5500d73b: moving from state preparing -> state finished >Thread-15914232::DEBUG::2013-11-20 03:26:41,234::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914232::DEBUG::2013-11-20 03:26:41,235::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914232::DEBUG::2013-11-20 03:26:41,235::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914232::DEBUG::2013-11-20 03:26:41,236::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914232::DEBUG::2013-11-20 03:26:41,237::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914232::DEBUG::2013-11-20 03:26:41,237::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914232::DEBUG::2013-11-20 03:26:41,238::task::495::TaskManager.Task::(_debug) Task 024d8dea-83da-4b5b-a500-bc5b5500d73b: ref 0 aborting False >Thread-15914232::INFO::2013-11-20 03:26:41,239::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:26:42,135::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:26:42,653::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0908515 s, 11.3 MB/s\n'; <rc> = 0 >Thread-15913695::INFO::2013-11-20 03:26:43,808::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:26:43,809::task::495::TaskManager.Task::(_debug) Task 355f2756-a364-4b7e-b569-fda450e96f5d: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:26:43,811::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a1c73c2f-3309-4249-b719-297544b533ce`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:26:43,811::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:26:43,812::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:26:43,813::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a1c73c2f-3309-4249-b719-297544b533ce`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:26:43,814::task::495::TaskManager.Task::(_debug) Task 355f2756-a364-4b7e-b569-fda450e96f5d: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:26:43,815::task::495::TaskManager.Task::(_debug) Task 355f2756-a364-4b7e-b569-fda450e96f5d: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:26:43,815::task::495::TaskManager.Task::(_debug) Task 355f2756-a364-4b7e-b569-fda450e96f5d: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:26:43,816::task::495::TaskManager.Task::(_debug) Task 355f2756-a364-4b7e-b569-fda450e96f5d: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:26:43,817::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:26:43,817::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:26:43,818::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:26:43,819::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:26:43,819::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:26:43,820::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:26:43,820::task::495::TaskManager.Task::(_debug) Task 355f2756-a364-4b7e-b569-fda450e96f5d: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:26:43,821::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:26:44,668::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:26:45,106::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0746779 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15913658::INFO::2013-11-20 03:26:46,327::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:26:46,328::task::495::TaskManager.Task::(_debug) Task 99602d19-04eb-49ee-b4e6-30612cd50599: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:26:46,330::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`2b571e01-0007-42db-ad88-cd23b47d32ef`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:26:46,330::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:26:46,331::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:26:46,332::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`2b571e01-0007-42db-ad88-cd23b47d32ef`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:26:46,333::task::495::TaskManager.Task::(_debug) Task 99602d19-04eb-49ee-b4e6-30612cd50599: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:26:46,333::task::495::TaskManager.Task::(_debug) Task 99602d19-04eb-49ee-b4e6-30612cd50599: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:26:46,334::task::495::TaskManager.Task::(_debug) Task 99602d19-04eb-49ee-b4e6-30612cd50599: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:26:46,335::task::495::TaskManager.Task::(_debug) Task 99602d19-04eb-49ee-b4e6-30612cd50599: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:26:46,335::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:26:46,336::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:26:46,336::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:26:46,337::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:26:46,338::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:26:46,338::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:26:46,339::task::495::TaskManager.Task::(_debug) Task 99602d19-04eb-49ee-b4e6-30612cd50599: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:26:46,339::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:26:47,122::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:26:47,616::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0856837 s, 12.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:26:49,632::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:26:50,206::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0815278 s, 12.6 MB/s\n'; <rc> = 0 >Thread-15912392::INFO::2013-11-20 03:26:51,302::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:26:51,303::task::495::TaskManager.Task::(_debug) Task 8ce32823-bfe2-48ba-a6de-29d01b811f96: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:26:51,305::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`918d91b4-e576-44bc-9750-fb9a50706dfa`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:26:51,306::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:26:51,307::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:26:51,308::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`918d91b4-e576-44bc-9750-fb9a50706dfa`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:26:51,309::task::495::TaskManager.Task::(_debug) Task 8ce32823-bfe2-48ba-a6de-29d01b811f96: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:26:51,310::task::495::TaskManager.Task::(_debug) Task 8ce32823-bfe2-48ba-a6de-29d01b811f96: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:26:51,311::task::495::TaskManager.Task::(_debug) Task 8ce32823-bfe2-48ba-a6de-29d01b811f96: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:26:51,312::task::495::TaskManager.Task::(_debug) Task 8ce32823-bfe2-48ba-a6de-29d01b811f96: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:26:51,313::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:26:51,313::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:26:51,314::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:26:51,315::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:26:51,316::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:26:51,316::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:26:51,317::task::495::TaskManager.Task::(_debug) Task 8ce32823-bfe2-48ba-a6de-29d01b811f96: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:26:51,318::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15914236::DEBUG::2013-11-20 03:26:51,439::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914236::INFO::2013-11-20 03:26:51,440::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914236::DEBUG::2013-11-20 03:26:51,441::task::495::TaskManager.Task::(_debug) Task 88cfff6f-fe18-4812-90a7-cdd4ba7dd695: moving from state init -> state preparing >Thread-15914236::DEBUG::2013-11-20 03:26:51,442::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914236::DEBUG::2013-11-20 03:26:51,442::task::495::TaskManager.Task::(_debug) Task 88cfff6f-fe18-4812-90a7-cdd4ba7dd695: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914236::DEBUG::2013-11-20 03:26:51,443::task::495::TaskManager.Task::(_debug) Task 88cfff6f-fe18-4812-90a7-cdd4ba7dd695: moving from state preparing -> state finished >Thread-15914236::DEBUG::2013-11-20 03:26:51,443::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914236::DEBUG::2013-11-20 03:26:51,444::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914236::DEBUG::2013-11-20 03:26:51,445::task::495::TaskManager.Task::(_debug) Task 88cfff6f-fe18-4812-90a7-cdd4ba7dd695: ref 0 aborting False >Thread-15914236::INFO::2013-11-20 03:26:51,445::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914237::DEBUG::2013-11-20 03:26:51,456::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914237::INFO::2013-11-20 03:26:51,457::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914237::DEBUG::2013-11-20 03:26:51,458::task::495::TaskManager.Task::(_debug) Task 26fab461-7df8-438a-a9d0-19c5416cd5de: moving from state init -> state preparing >Thread-15914237::DEBUG::2013-11-20 03:26:51,466::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`74b6d504-21ea-4c53-9dc7-ba49ad5dd3d2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914237::DEBUG::2013-11-20 03:26:51,467::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914237::DEBUG::2013-11-20 03:26:51,468::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914237::DEBUG::2013-11-20 03:26:51,468::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`74b6d504-21ea-4c53-9dc7-ba49ad5dd3d2`::Granted request >Thread-15914237::DEBUG::2013-11-20 03:26:51,471::task::495::TaskManager.Task::(_debug) Task 26fab461-7df8-438a-a9d0-19c5416cd5de: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914237::DEBUG::2013-11-20 03:26:51,472::task::495::TaskManager.Task::(_debug) Task 26fab461-7df8-438a-a9d0-19c5416cd5de: ref 1 aborting False >Thread-15914237::WARNING::2013-11-20 03:26:51,474::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914237::DEBUG::2013-11-20 03:26:51,475::task::495::TaskManager.Task::(_debug) Task 26fab461-7df8-438a-a9d0-19c5416cd5de: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914237::DEBUG::2013-11-20 03:26:51,476::task::495::TaskManager.Task::(_debug) Task 26fab461-7df8-438a-a9d0-19c5416cd5de: moving from state preparing -> state finished >Thread-15914237::DEBUG::2013-11-20 03:26:51,476::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914237::DEBUG::2013-11-20 03:26:51,477::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914237::DEBUG::2013-11-20 03:26:51,477::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914237::DEBUG::2013-11-20 03:26:51,478::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914237::DEBUG::2013-11-20 03:26:51,478::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914237::DEBUG::2013-11-20 03:26:51,479::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914237::DEBUG::2013-11-20 03:26:51,480::task::495::TaskManager.Task::(_debug) Task 26fab461-7df8-438a-a9d0-19c5416cd5de: ref 0 aborting False >Thread-15914237::INFO::2013-11-20 03:26:51,480::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914238::INFO::2013-11-20 03:26:51,615::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914238::DEBUG::2013-11-20 03:26:51,616::task::495::TaskManager.Task::(_debug) Task b82fc932-a9d2-43ba-9cf1-e1e5b6ef19a0: moving from state init -> state preparing >Thread-15914238::DEBUG::2013-11-20 03:26:51,616::task::495::TaskManager.Task::(_debug) Task b82fc932-a9d2-43ba-9cf1-e1e5b6ef19a0: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000475883483887', 'lastCheck': 1384918002.3308311, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0128321647644', 'lastCheck': 1384918008.8547599, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00153589248657', 'lastCheck': 1384918011.3427219, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000518798828125', 'lastCheck': 1384918001.7824221, 'valid': True, 'code': 0}} >Thread-15914238::DEBUG::2013-11-20 03:26:51,617::task::495::TaskManager.Task::(_debug) Task b82fc932-a9d2-43ba-9cf1-e1e5b6ef19a0: moving from state preparing -> state finished >Thread-15914238::DEBUG::2013-11-20 03:26:51,617::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914238::DEBUG::2013-11-20 03:26:51,617::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914238::DEBUG::2013-11-20 03:26:51,618::task::495::TaskManager.Task::(_debug) Task b82fc932-a9d2-43ba-9cf1-e1e5b6ef19a0: ref 0 aborting False >Thread-15914238::INFO::2013-11-20 03:26:51,618::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000475883483887', 'lastCheck': 1384918002.3308311, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0128321647644', 'lastCheck': 1384918008.8547599, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00153589248657', 'lastCheck': 1384918011.3427219, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000518798828125', 'lastCheck': 1384918001.7824221, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:26:52,220::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-42838::DEBUG::2013-11-20 03:26:52,409::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:26:52,690::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0513323 s, 19.9 MB/s\n'; <rc> = 0 >Thread-42838::DEBUG::2013-11-20 03:26:53,111::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:26:54,701::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:26:55,158::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0664979 s, 15.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:26:57,170::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:26:57,591::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0520295 s, 19.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:26:59,601::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:27:00,062::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0594011 s, 17.2 MB/s\n'; <rc> = 0 >Thread-15913518::INFO::2013-11-20 03:27:01,490::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:27:01,490::task::495::TaskManager.Task::(_debug) Task 38347dbb-cf0a-4d46-8d03-f3f7ef0a7cb6: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:27:01,492::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`25bcf7f0-5eeb-4e2c-860c-2d700a7dab53`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:27:01,493::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:27:01,494::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:27:01,494::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`25bcf7f0-5eeb-4e2c-860c-2d700a7dab53`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:27:01,495::task::495::TaskManager.Task::(_debug) Task 38347dbb-cf0a-4d46-8d03-f3f7ef0a7cb6: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:27:01,496::task::495::TaskManager.Task::(_debug) Task 38347dbb-cf0a-4d46-8d03-f3f7ef0a7cb6: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:27:01,496::task::495::TaskManager.Task::(_debug) Task 38347dbb-cf0a-4d46-8d03-f3f7ef0a7cb6: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:27:01,497::task::495::TaskManager.Task::(_debug) Task 38347dbb-cf0a-4d46-8d03-f3f7ef0a7cb6: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:27:01,498::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:27:01,498::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:27:01,499::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:27:01,499::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:27:01,500::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:27:01,500::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:27:01,501::task::495::TaskManager.Task::(_debug) Task 38347dbb-cf0a-4d46-8d03-f3f7ef0a7cb6: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:27:01,502::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:27:01,503::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:27:01,503::task::495::TaskManager.Task::(_debug) Task cdb4dc89-d5ed-460b-8fbe-10b1402606c3: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:27:01,504::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`9e5f3493-77cf-40b3-9ab2-3df39bd295c1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:27:01,505::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:27:01,505::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:27:01,506::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`9e5f3493-77cf-40b3-9ab2-3df39bd295c1`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:27:01,507::task::495::TaskManager.Task::(_debug) Task cdb4dc89-d5ed-460b-8fbe-10b1402606c3: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:27:01,508::task::495::TaskManager.Task::(_debug) Task cdb4dc89-d5ed-460b-8fbe-10b1402606c3: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:27:01,508::task::495::TaskManager.Task::(_debug) Task cdb4dc89-d5ed-460b-8fbe-10b1402606c3: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:27:01,509::task::495::TaskManager.Task::(_debug) Task cdb4dc89-d5ed-460b-8fbe-10b1402606c3: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:27:01,509::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:27:01,510::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:27:01,511::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:27:01,511::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:27:01,512::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:27:01,512::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:27:01,513::task::495::TaskManager.Task::(_debug) Task cdb4dc89-d5ed-460b-8fbe-10b1402606c3: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:27:01,514::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15914243::DEBUG::2013-11-20 03:27:01,976::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914243::INFO::2013-11-20 03:27:01,977::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914243::DEBUG::2013-11-20 03:27:01,978::task::495::TaskManager.Task::(_debug) Task e5bd68d3-d462-4ea4-b0e2-242a551def1c: moving from state init -> state preparing >Thread-15914243::DEBUG::2013-11-20 03:27:01,979::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914243::DEBUG::2013-11-20 03:27:01,979::task::495::TaskManager.Task::(_debug) Task e5bd68d3-d462-4ea4-b0e2-242a551def1c: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914243::DEBUG::2013-11-20 03:27:01,980::task::495::TaskManager.Task::(_debug) Task e5bd68d3-d462-4ea4-b0e2-242a551def1c: moving from state preparing -> state finished >Thread-15914243::DEBUG::2013-11-20 03:27:01,980::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914243::DEBUG::2013-11-20 03:27:01,981::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914243::DEBUG::2013-11-20 03:27:01,981::task::495::TaskManager.Task::(_debug) Task e5bd68d3-d462-4ea4-b0e2-242a551def1c: ref 0 aborting False >Thread-15914243::INFO::2013-11-20 03:27:01,982::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914244::DEBUG::2013-11-20 03:27:01,986::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914244::INFO::2013-11-20 03:27:01,987::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914244::DEBUG::2013-11-20 03:27:01,987::task::495::TaskManager.Task::(_debug) Task ca8f4ae7-4e18-4e74-8111-77bfbfb16821: moving from state init -> state preparing >Thread-15914244::DEBUG::2013-11-20 03:27:01,994::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3215e5f8-5a9e-4f52-8927-5422be031a6e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914244::DEBUG::2013-11-20 03:27:01,995::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914244::DEBUG::2013-11-20 03:27:02,001::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914244::DEBUG::2013-11-20 03:27:02,002::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3215e5f8-5a9e-4f52-8927-5422be031a6e`::Granted request >Thread-15914244::DEBUG::2013-11-20 03:27:02,003::task::495::TaskManager.Task::(_debug) Task ca8f4ae7-4e18-4e74-8111-77bfbfb16821: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914244::DEBUG::2013-11-20 03:27:02,003::task::495::TaskManager.Task::(_debug) Task ca8f4ae7-4e18-4e74-8111-77bfbfb16821: ref 1 aborting False >Thread-15914244::WARNING::2013-11-20 03:27:02,005::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914244::DEBUG::2013-11-20 03:27:02,006::task::495::TaskManager.Task::(_debug) Task ca8f4ae7-4e18-4e74-8111-77bfbfb16821: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914244::DEBUG::2013-11-20 03:27:02,007::task::495::TaskManager.Task::(_debug) Task ca8f4ae7-4e18-4e74-8111-77bfbfb16821: moving from state preparing -> state finished >Thread-15914244::DEBUG::2013-11-20 03:27:02,007::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914244::DEBUG::2013-11-20 03:27:02,008::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914244::DEBUG::2013-11-20 03:27:02,008::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914244::DEBUG::2013-11-20 03:27:02,009::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914244::DEBUG::2013-11-20 03:27:02,009::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914244::DEBUG::2013-11-20 03:27:02,010::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914244::DEBUG::2013-11-20 03:27:02,011::task::495::TaskManager.Task::(_debug) Task ca8f4ae7-4e18-4e74-8111-77bfbfb16821: ref 0 aborting False >Thread-15914244::INFO::2013-11-20 03:27:02,011::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:27:02,075::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:27:02,551::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0448105 s, 22.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:27:04,563::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:27:04,986::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0513742 s, 19.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:27:06,997::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914246::INFO::2013-11-20 03:27:07,364::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914246::DEBUG::2013-11-20 03:27:07,366::task::495::TaskManager.Task::(_debug) Task 77a76421-c0ae-4093-a54a-c33b39c257be: moving from state init -> state preparing >Thread-15914246::DEBUG::2013-11-20 03:27:07,367::task::495::TaskManager.Task::(_debug) Task 77a76421-c0ae-4093-a54a-c33b39c257be: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000359058380127', 'lastCheck': 1384918023.2802761, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0606088638306', 'lastCheck': 1384918018.953573, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00896286964417', 'lastCheck': 1384918021.430521, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00142407417297', 'lastCheck': 1384918021.883749, 'valid': True, 'code': 0}} >Thread-15914246::DEBUG::2013-11-20 03:27:07,376::task::495::TaskManager.Task::(_debug) Task 77a76421-c0ae-4093-a54a-c33b39c257be: moving from state preparing -> state finished >Thread-15914246::DEBUG::2013-11-20 03:27:07,377::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914246::DEBUG::2013-11-20 03:27:07,377::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914246::DEBUG::2013-11-20 03:27:07,378::task::495::TaskManager.Task::(_debug) Task 77a76421-c0ae-4093-a54a-c33b39c257be: ref 0 aborting False >Thread-15914246::INFO::2013-11-20 03:27:07,379::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000359058380127', 'lastCheck': 1384918023.2802761, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0606088638306', 'lastCheck': 1384918018.953573, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00896286964417', 'lastCheck': 1384918021.430521, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00142407417297', 'lastCheck': 1384918021.883749, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:27:07,545::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0971099 s, 10.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:27:09,560::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:27:10,028::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.056394 s, 18.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:27:12,042::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914249::DEBUG::2013-11-20 03:27:12,247::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914249::INFO::2013-11-20 03:27:12,250::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914249::DEBUG::2013-11-20 03:27:12,250::task::495::TaskManager.Task::(_debug) Task 3d322897-ea93-4f51-a12f-b245126a9470: moving from state init -> state preparing >Thread-15914249::DEBUG::2013-11-20 03:27:12,251::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914249::DEBUG::2013-11-20 03:27:12,252::task::495::TaskManager.Task::(_debug) Task 3d322897-ea93-4f51-a12f-b245126a9470: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914249::DEBUG::2013-11-20 03:27:12,253::task::495::TaskManager.Task::(_debug) Task 3d322897-ea93-4f51-a12f-b245126a9470: moving from state preparing -> state finished >Thread-15914249::DEBUG::2013-11-20 03:27:12,253::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914249::DEBUG::2013-11-20 03:27:12,254::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914249::DEBUG::2013-11-20 03:27:12,254::task::495::TaskManager.Task::(_debug) Task 3d322897-ea93-4f51-a12f-b245126a9470: ref 0 aborting False >Thread-15914249::INFO::2013-11-20 03:27:12,255::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914250::DEBUG::2013-11-20 03:27:12,260::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914250::INFO::2013-11-20 03:27:12,261::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914250::DEBUG::2013-11-20 03:27:12,262::task::495::TaskManager.Task::(_debug) Task 43475944-106b-4498-a140-91c466473cdf: moving from state init -> state preparing >Thread-15914250::DEBUG::2013-11-20 03:27:12,264::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`70c71b13-2992-4339-84c9-62cdf97ddfa7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914250::DEBUG::2013-11-20 03:27:12,265::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914250::DEBUG::2013-11-20 03:27:12,266::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914250::DEBUG::2013-11-20 03:27:12,266::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`70c71b13-2992-4339-84c9-62cdf97ddfa7`::Granted request >Thread-15914250::DEBUG::2013-11-20 03:27:12,271::task::495::TaskManager.Task::(_debug) Task 43475944-106b-4498-a140-91c466473cdf: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914250::DEBUG::2013-11-20 03:27:12,272::task::495::TaskManager.Task::(_debug) Task 43475944-106b-4498-a140-91c466473cdf: ref 1 aborting False >Thread-15914250::WARNING::2013-11-20 03:27:12,274::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914250::DEBUG::2013-11-20 03:27:12,280::task::495::TaskManager.Task::(_debug) Task 43475944-106b-4498-a140-91c466473cdf: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914250::DEBUG::2013-11-20 03:27:12,281::task::495::TaskManager.Task::(_debug) Task 43475944-106b-4498-a140-91c466473cdf: moving from state preparing -> state finished >Thread-15914250::DEBUG::2013-11-20 03:27:12,281::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914250::DEBUG::2013-11-20 03:27:12,282::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914250::DEBUG::2013-11-20 03:27:12,283::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914250::DEBUG::2013-11-20 03:27:12,283::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914250::DEBUG::2013-11-20 03:27:12,284::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914250::DEBUG::2013-11-20 03:27:12,284::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914250::DEBUG::2013-11-20 03:27:12,285::task::495::TaskManager.Task::(_debug) Task 43475944-106b-4498-a140-91c466473cdf: ref 0 aborting False >Thread-15914250::INFO::2013-11-20 03:27:12,285::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:27:12,496::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0626639 s, 16.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:27:14,507::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:27:14,988::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0745252 s, 13.7 MB/s\n'; <rc> = 0 >Thread-15913546::INFO::2013-11-20 03:27:16,455::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:27:16,465::task::495::TaskManager.Task::(_debug) Task dea10702-dda1-4473-b405-9dd99bac5ddc: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:27:16,466::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fe8ee39e-6d07-41c5-bc17-4c1bbcdf51b4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:27:16,467::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:27:16,469::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:27:16,469::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fe8ee39e-6d07-41c5-bc17-4c1bbcdf51b4`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:27:16,470::task::495::TaskManager.Task::(_debug) Task dea10702-dda1-4473-b405-9dd99bac5ddc: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:27:16,471::task::495::TaskManager.Task::(_debug) Task dea10702-dda1-4473-b405-9dd99bac5ddc: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:27:16,471::task::495::TaskManager.Task::(_debug) Task dea10702-dda1-4473-b405-9dd99bac5ddc: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:27:16,471::task::495::TaskManager.Task::(_debug) Task dea10702-dda1-4473-b405-9dd99bac5ddc: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:27:16,472::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:27:16,472::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:27:16,473::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:27:16,473::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:27:16,474::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:27:16,474::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:27:16,474::task::495::TaskManager.Task::(_debug) Task dea10702-dda1-4473-b405-9dd99bac5ddc: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:27:16,475::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:27:16,475::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:27:16,476::task::495::TaskManager.Task::(_debug) Task 0d9128e3-f300-4ae1-a96e-af0aa4e4d74c: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:27:16,476::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`2ece3f99-8ecd-4fee-a227-c607ef20d352`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:27:16,477::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:27:16,477::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:27:16,478::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`2ece3f99-8ecd-4fee-a227-c607ef20d352`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:27:16,478::task::495::TaskManager.Task::(_debug) Task 0d9128e3-f300-4ae1-a96e-af0aa4e4d74c: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:27:16,479::task::495::TaskManager.Task::(_debug) Task 0d9128e3-f300-4ae1-a96e-af0aa4e4d74c: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:27:16,479::task::495::TaskManager.Task::(_debug) Task 0d9128e3-f300-4ae1-a96e-af0aa4e4d74c: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:27:16,479::task::495::TaskManager.Task::(_debug) Task 0d9128e3-f300-4ae1-a96e-af0aa4e4d74c: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:27:16,480::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:27:16,480::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:27:16,480::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:27:16,481::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:27:16,481::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:27:16,481::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:27:16,482::task::495::TaskManager.Task::(_debug) Task 0d9128e3-f300-4ae1-a96e-af0aa4e4d74c: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:27:16,482::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:27:16,483::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:27:16,483::task::495::TaskManager.Task::(_debug) Task d07e0ee8-4b51-4b9c-8bae-cffe3abd7f5a: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:27:16,484::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`d4f85e67-c76b-4ef4-afd3-4755178f0878`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:27:16,484::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:27:16,485::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:27:16,485::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`d4f85e67-c76b-4ef4-afd3-4755178f0878`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:27:16,486::task::495::TaskManager.Task::(_debug) Task d07e0ee8-4b51-4b9c-8bae-cffe3abd7f5a: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:27:16,487::task::495::TaskManager.Task::(_debug) Task d07e0ee8-4b51-4b9c-8bae-cffe3abd7f5a: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:27:16,488::task::495::TaskManager.Task::(_debug) Task d07e0ee8-4b51-4b9c-8bae-cffe3abd7f5a: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:27:16,488::task::495::TaskManager.Task::(_debug) Task d07e0ee8-4b51-4b9c-8bae-cffe3abd7f5a: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:27:16,489::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:27:16,489::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:27:16,490::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:27:16,491::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:27:16,491::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:27:16,492::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:27:16,492::task::495::TaskManager.Task::(_debug) Task d07e0ee8-4b51-4b9c-8bae-cffe3abd7f5a: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:27:16,493::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:27:16,494::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:27:16,494::task::495::TaskManager.Task::(_debug) Task 8b8f8245-7f5a-4490-8387-0711478cc6e1: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:27:16,496::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`009288a2-4dd3-46d8-aa85-ea5eaebac1c9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:27:16,496::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:27:16,497::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:27:16,497::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`009288a2-4dd3-46d8-aa85-ea5eaebac1c9`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:27:16,498::task::495::TaskManager.Task::(_debug) Task 8b8f8245-7f5a-4490-8387-0711478cc6e1: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:27:16,499::task::495::TaskManager.Task::(_debug) Task 8b8f8245-7f5a-4490-8387-0711478cc6e1: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:27:16,499::task::495::TaskManager.Task::(_debug) Task 8b8f8245-7f5a-4490-8387-0711478cc6e1: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:27:16,499::task::495::TaskManager.Task::(_debug) Task 8b8f8245-7f5a-4490-8387-0711478cc6e1: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:27:16,500::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:27:16,500::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:27:16,500::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:27:16,501::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:27:16,501::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:27:16,501::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:27:16,502::task::495::TaskManager.Task::(_debug) Task 8b8f8245-7f5a-4490-8387-0711478cc6e1: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:27:16,502::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:27:17,003::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:27:17,524::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0823539 s, 12.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:27:19,540::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:27:19,999::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0463075 s, 22.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:27:22,011::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:27:22,417::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0510312 s, 20.1 MB/s\n'; <rc> = 0 >Thread-15914254::DEBUG::2013-11-20 03:27:22,744::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914254::INFO::2013-11-20 03:27:22,745::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914254::DEBUG::2013-11-20 03:27:22,746::task::495::TaskManager.Task::(_debug) Task 131fa2de-7a90-4ce4-916b-418b9a711b8e: moving from state init -> state preparing >Thread-15914254::DEBUG::2013-11-20 03:27:22,747::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914254::DEBUG::2013-11-20 03:27:22,748::task::495::TaskManager.Task::(_debug) Task 131fa2de-7a90-4ce4-916b-418b9a711b8e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914254::DEBUG::2013-11-20 03:27:22,748::task::495::TaskManager.Task::(_debug) Task 131fa2de-7a90-4ce4-916b-418b9a711b8e: moving from state preparing -> state finished >Thread-15914254::DEBUG::2013-11-20 03:27:22,749::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914254::DEBUG::2013-11-20 03:27:22,750::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914254::DEBUG::2013-11-20 03:27:22,750::task::495::TaskManager.Task::(_debug) Task 131fa2de-7a90-4ce4-916b-418b9a711b8e: ref 0 aborting False >Thread-15914254::INFO::2013-11-20 03:27:22,751::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914255::DEBUG::2013-11-20 03:27:22,772::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914255::INFO::2013-11-20 03:27:22,773::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914255::DEBUG::2013-11-20 03:27:22,774::task::495::TaskManager.Task::(_debug) Task 03dcaf63-079e-4eb4-8dfc-53cd7a089905: moving from state init -> state preparing >Thread-15914255::DEBUG::2013-11-20 03:27:22,785::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`8527e6ff-772a-4c51-82c6-61fac0fb2fb1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914255::DEBUG::2013-11-20 03:27:22,785::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914255::DEBUG::2013-11-20 03:27:22,786::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914255::DEBUG::2013-11-20 03:27:22,786::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`8527e6ff-772a-4c51-82c6-61fac0fb2fb1`::Granted request >Thread-15914255::DEBUG::2013-11-20 03:27:22,787::task::495::TaskManager.Task::(_debug) Task 03dcaf63-079e-4eb4-8dfc-53cd7a089905: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914255::DEBUG::2013-11-20 03:27:22,788::task::495::TaskManager.Task::(_debug) Task 03dcaf63-079e-4eb4-8dfc-53cd7a089905: ref 1 aborting False >Thread-15914255::WARNING::2013-11-20 03:27:22,789::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914255::DEBUG::2013-11-20 03:27:22,790::task::495::TaskManager.Task::(_debug) Task 03dcaf63-079e-4eb4-8dfc-53cd7a089905: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914255::DEBUG::2013-11-20 03:27:22,790::task::495::TaskManager.Task::(_debug) Task 03dcaf63-079e-4eb4-8dfc-53cd7a089905: moving from state preparing -> state finished >Thread-15914255::DEBUG::2013-11-20 03:27:22,791::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914255::DEBUG::2013-11-20 03:27:22,791::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914255::DEBUG::2013-11-20 03:27:22,791::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914255::DEBUG::2013-11-20 03:27:22,792::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914255::DEBUG::2013-11-20 03:27:22,792::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914255::DEBUG::2013-11-20 03:27:22,793::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914255::DEBUG::2013-11-20 03:27:22,793::task::495::TaskManager.Task::(_debug) Task 03dcaf63-079e-4eb4-8dfc-53cd7a089905: ref 0 aborting False >Thread-15914255::INFO::2013-11-20 03:27:22,793::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914256::INFO::2013-11-20 03:27:22,872::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914256::DEBUG::2013-11-20 03:27:22,873::task::495::TaskManager.Task::(_debug) Task ea9be4c0-8ee9-4420-8fa9-2e657482e03b: moving from state init -> state preparing >Thread-15914256::DEBUG::2013-11-20 03:27:22,874::task::495::TaskManager.Task::(_debug) Task ea9be4c0-8ee9-4420-8fa9-2e657482e03b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00036096572876', 'lastCheck': 1384918033.2950549, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0151832103729', 'lastCheck': 1384918039.0007169, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00879907608032', 'lastCheck': 1384918041.4687941, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000380992889404', 'lastCheck': 1384918041.9038639, 'valid': True, 'code': 0}} >Thread-15914256::DEBUG::2013-11-20 03:27:22,886::task::495::TaskManager.Task::(_debug) Task ea9be4c0-8ee9-4420-8fa9-2e657482e03b: moving from state preparing -> state finished >Thread-15914256::DEBUG::2013-11-20 03:27:22,889::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914256::DEBUG::2013-11-20 03:27:22,892::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914256::DEBUG::2013-11-20 03:27:22,894::task::495::TaskManager.Task::(_debug) Task ea9be4c0-8ee9-4420-8fa9-2e657482e03b: ref 0 aborting False >Thread-15914256::INFO::2013-11-20 03:27:22,897::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00036096572876', 'lastCheck': 1384918033.2950549, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0151832103729', 'lastCheck': 1384918039.0007169, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00879907608032', 'lastCheck': 1384918041.4687941, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000380992889404', 'lastCheck': 1384918041.9038639, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:27:24,438::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:27:24,986::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0826078 s, 12.4 MB/s\n'; <rc> = 0 >Thread-15913639::INFO::2013-11-20 03:27:26,658::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:27:26,663::task::495::TaskManager.Task::(_debug) Task a922e5e9-3ef7-406e-89ad-6e58fe94ddf9: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:27:26,665::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a14601e3-c7e1-4355-8f73-5462e75b9149`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:27:26,666::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:27:26,667::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:27:26,668::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a14601e3-c7e1-4355-8f73-5462e75b9149`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:27:26,668::task::495::TaskManager.Task::(_debug) Task a922e5e9-3ef7-406e-89ad-6e58fe94ddf9: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:27:26,669::task::495::TaskManager.Task::(_debug) Task a922e5e9-3ef7-406e-89ad-6e58fe94ddf9: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:27:26,670::task::495::TaskManager.Task::(_debug) Task a922e5e9-3ef7-406e-89ad-6e58fe94ddf9: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:27:26,670::task::495::TaskManager.Task::(_debug) Task a922e5e9-3ef7-406e-89ad-6e58fe94ddf9: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:27:26,671::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:27:26,671::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:27:26,672::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:27:26,673::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:27:26,673::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:27:26,673::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:27:26,674::task::495::TaskManager.Task::(_debug) Task a922e5e9-3ef7-406e-89ad-6e58fe94ddf9: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:27:26,675::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:27:27,000::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:27:27,450::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0496348 s, 20.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:27:29,460::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:27:29,889::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0529633 s, 19.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:27:31,898::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:27:32,291::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0477116 s, 21.5 MB/s\n'; <rc> = 0 >Thread-15914261::DEBUG::2013-11-20 03:27:33,503::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914261::INFO::2013-11-20 03:27:33,504::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914261::DEBUG::2013-11-20 03:27:33,505::task::495::TaskManager.Task::(_debug) Task 5799558c-3b9b-4f4f-bc3a-62b76600b2c9: moving from state init -> state preparing >Thread-15914261::DEBUG::2013-11-20 03:27:33,505::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914261::DEBUG::2013-11-20 03:27:33,506::task::495::TaskManager.Task::(_debug) Task 5799558c-3b9b-4f4f-bc3a-62b76600b2c9: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914261::DEBUG::2013-11-20 03:27:33,506::task::495::TaskManager.Task::(_debug) Task 5799558c-3b9b-4f4f-bc3a-62b76600b2c9: moving from state preparing -> state finished >Thread-15914261::DEBUG::2013-11-20 03:27:33,507::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914261::DEBUG::2013-11-20 03:27:33,507::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914261::DEBUG::2013-11-20 03:27:33,507::task::495::TaskManager.Task::(_debug) Task 5799558c-3b9b-4f4f-bc3a-62b76600b2c9: ref 0 aborting False >Thread-15914261::INFO::2013-11-20 03:27:33,508::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914262::DEBUG::2013-11-20 03:27:33,516::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914262::INFO::2013-11-20 03:27:33,516::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914262::DEBUG::2013-11-20 03:27:33,517::task::495::TaskManager.Task::(_debug) Task 0f35ffeb-85ee-4df2-99f8-c7405ed802fc: moving from state init -> state preparing >Thread-15914262::DEBUG::2013-11-20 03:27:33,528::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6884954d-828a-4f13-bf38-d9d19c554849`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914262::DEBUG::2013-11-20 03:27:33,529::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914262::DEBUG::2013-11-20 03:27:33,529::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914262::DEBUG::2013-11-20 03:27:33,530::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6884954d-828a-4f13-bf38-d9d19c554849`::Granted request >Thread-15914262::DEBUG::2013-11-20 03:27:33,530::task::495::TaskManager.Task::(_debug) Task 0f35ffeb-85ee-4df2-99f8-c7405ed802fc: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914262::DEBUG::2013-11-20 03:27:33,531::task::495::TaskManager.Task::(_debug) Task 0f35ffeb-85ee-4df2-99f8-c7405ed802fc: ref 1 aborting False >Thread-15914262::WARNING::2013-11-20 03:27:33,532::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914262::DEBUG::2013-11-20 03:27:33,538::task::495::TaskManager.Task::(_debug) Task 0f35ffeb-85ee-4df2-99f8-c7405ed802fc: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914262::DEBUG::2013-11-20 03:27:33,538::task::495::TaskManager.Task::(_debug) Task 0f35ffeb-85ee-4df2-99f8-c7405ed802fc: moving from state preparing -> state finished >Thread-15914262::DEBUG::2013-11-20 03:27:33,539::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914262::DEBUG::2013-11-20 03:27:33,539::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914262::DEBUG::2013-11-20 03:27:33,540::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914262::DEBUG::2013-11-20 03:27:33,540::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914262::DEBUG::2013-11-20 03:27:33,541::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914262::DEBUG::2013-11-20 03:27:33,541::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914262::DEBUG::2013-11-20 03:27:33,541::task::495::TaskManager.Task::(_debug) Task 0f35ffeb-85ee-4df2-99f8-c7405ed802fc: ref 0 aborting False >Thread-15914262::INFO::2013-11-20 03:27:33,542::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:27:34,306::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:27:34,778::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0811539 s, 12.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:27:36,793::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:27:37,174::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0543953 s, 18.8 MB/s\n'; <rc> = 0 >Thread-15914264::INFO::2013-11-20 03:27:38,779::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914264::DEBUG::2013-11-20 03:27:38,779::task::495::TaskManager.Task::(_debug) Task a3b79a09-44c8-412f-b6b0-6dd05b59e383: moving from state init -> state preparing >Thread-15914264::DEBUG::2013-11-20 03:27:38,780::task::495::TaskManager.Task::(_debug) Task a3b79a09-44c8-412f-b6b0-6dd05b59e383: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000218868255615', 'lastCheck': 1384918053.321713, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000968933105469', 'lastCheck': 1384918049.203701, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00130891799927', 'lastCheck': 1384918051.6666851, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00103712081909', 'lastCheck': 1384918052.061547, 'valid': True, 'code': 0}} >Thread-15914264::DEBUG::2013-11-20 03:27:38,791::task::495::TaskManager.Task::(_debug) Task a3b79a09-44c8-412f-b6b0-6dd05b59e383: moving from state preparing -> state finished >Thread-15914264::DEBUG::2013-11-20 03:27:38,792::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914264::DEBUG::2013-11-20 03:27:38,792::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914264::DEBUG::2013-11-20 03:27:38,792::task::495::TaskManager.Task::(_debug) Task a3b79a09-44c8-412f-b6b0-6dd05b59e383: ref 0 aborting False >Thread-15914264::INFO::2013-11-20 03:27:38,793::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000218868255615', 'lastCheck': 1384918053.321713, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000968933105469', 'lastCheck': 1384918049.203701, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00130891799927', 'lastCheck': 1384918051.6666851, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00103712081909', 'lastCheck': 1384918052.061547, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:27:39,187::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:27:39,709::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0826891 s, 12.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:27:41,722::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:27:42,241::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0848291 s, 12.1 MB/s\n'; <rc> = 0 >Thread-15914267::DEBUG::2013-11-20 03:27:43,825::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914267::INFO::2013-11-20 03:27:43,827::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914267::DEBUG::2013-11-20 03:27:43,827::task::495::TaskManager.Task::(_debug) Task d313a45a-66a9-4b56-bfac-6b02fb5a579d: moving from state init -> state preparing >Thread-15914267::DEBUG::2013-11-20 03:27:43,828::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914267::DEBUG::2013-11-20 03:27:43,829::task::495::TaskManager.Task::(_debug) Task d313a45a-66a9-4b56-bfac-6b02fb5a579d: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914267::DEBUG::2013-11-20 03:27:43,830::task::495::TaskManager.Task::(_debug) Task d313a45a-66a9-4b56-bfac-6b02fb5a579d: moving from state preparing -> state finished >Thread-15914267::DEBUG::2013-11-20 03:27:43,830::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914267::DEBUG::2013-11-20 03:27:43,831::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914267::DEBUG::2013-11-20 03:27:43,831::task::495::TaskManager.Task::(_debug) Task d313a45a-66a9-4b56-bfac-6b02fb5a579d: ref 0 aborting False >Thread-15914267::INFO::2013-11-20 03:27:43,832::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914268::DEBUG::2013-11-20 03:27:43,837::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914268::INFO::2013-11-20 03:27:43,838::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914268::DEBUG::2013-11-20 03:27:43,839::task::495::TaskManager.Task::(_debug) Task e6856ed0-dfd6-4c0c-9eac-1b5b60269736: moving from state init -> state preparing >Thread-15914268::DEBUG::2013-11-20 03:27:43,843::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3c0cc879-868b-4ebf-b053-5c4def961b78`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914268::DEBUG::2013-11-20 03:27:43,843::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914268::DEBUG::2013-11-20 03:27:43,845::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914268::DEBUG::2013-11-20 03:27:43,845::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3c0cc879-868b-4ebf-b053-5c4def961b78`::Granted request >Thread-15914268::DEBUG::2013-11-20 03:27:43,846::task::495::TaskManager.Task::(_debug) Task e6856ed0-dfd6-4c0c-9eac-1b5b60269736: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914268::DEBUG::2013-11-20 03:27:43,847::task::495::TaskManager.Task::(_debug) Task e6856ed0-dfd6-4c0c-9eac-1b5b60269736: ref 1 aborting False >Thread-15914268::WARNING::2013-11-20 03:27:43,849::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914268::DEBUG::2013-11-20 03:27:43,850::task::495::TaskManager.Task::(_debug) Task e6856ed0-dfd6-4c0c-9eac-1b5b60269736: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914268::DEBUG::2013-11-20 03:27:43,851::task::495::TaskManager.Task::(_debug) Task e6856ed0-dfd6-4c0c-9eac-1b5b60269736: moving from state preparing -> state finished >Thread-15914268::DEBUG::2013-11-20 03:27:43,852::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914268::DEBUG::2013-11-20 03:27:43,853::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914268::DEBUG::2013-11-20 03:27:43,854::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914268::DEBUG::2013-11-20 03:27:43,854::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914268::DEBUG::2013-11-20 03:27:43,855::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914268::DEBUG::2013-11-20 03:27:43,855::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914268::DEBUG::2013-11-20 03:27:43,856::task::495::TaskManager.Task::(_debug) Task e6856ed0-dfd6-4c0c-9eac-1b5b60269736: ref 0 aborting False >Thread-15914268::INFO::2013-11-20 03:27:43,857::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:27:44,255::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913695::INFO::2013-11-20 03:27:44,540::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:27:44,543::task::495::TaskManager.Task::(_debug) Task 67e7932c-8cd5-4362-91c8-9a13412b5d92: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:27:44,545::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b453272d-2bca-4e6e-8b98-9cd1587fe043`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:27:44,545::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:27:44,546::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:27:44,546::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b453272d-2bca-4e6e-8b98-9cd1587fe043`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:27:44,547::task::495::TaskManager.Task::(_debug) Task 67e7932c-8cd5-4362-91c8-9a13412b5d92: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:27:44,548::task::495::TaskManager.Task::(_debug) Task 67e7932c-8cd5-4362-91c8-9a13412b5d92: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:27:44,548::task::495::TaskManager.Task::(_debug) Task 67e7932c-8cd5-4362-91c8-9a13412b5d92: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:27:44,549::task::495::TaskManager.Task::(_debug) Task 67e7932c-8cd5-4362-91c8-9a13412b5d92: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:27:44,549::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:27:44,549::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:27:44,550::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:27:44,550::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:27:44,551::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:27:44,551::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:27:44,551::task::495::TaskManager.Task::(_debug) Task 67e7932c-8cd5-4362-91c8-9a13412b5d92: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:27:44,552::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:27:44,819::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0845021 s, 12.1 MB/s\n'; <rc> = 0 >Thread-15913658::INFO::2013-11-20 03:27:46,697::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:27:46,698::task::495::TaskManager.Task::(_debug) Task 31f53d8b-5e27-4086-9853-0847883d59a6: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:27:46,706::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f71a0cb3-133e-483e-816e-de44f34ac8da`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:27:46,707::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:27:46,708::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:27:46,709::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f71a0cb3-133e-483e-816e-de44f34ac8da`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:27:46,710::task::495::TaskManager.Task::(_debug) Task 31f53d8b-5e27-4086-9853-0847883d59a6: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:27:46,711::task::495::TaskManager.Task::(_debug) Task 31f53d8b-5e27-4086-9853-0847883d59a6: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:27:46,712::task::495::TaskManager.Task::(_debug) Task 31f53d8b-5e27-4086-9853-0847883d59a6: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:27:46,712::task::495::TaskManager.Task::(_debug) Task 31f53d8b-5e27-4086-9853-0847883d59a6: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:27:46,713::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:27:46,713::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:27:46,714::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:27:46,715::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:27:46,715::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:27:46,715::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:27:46,715::task::495::TaskManager.Task::(_debug) Task 31f53d8b-5e27-4086-9853-0847883d59a6: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:27:46,716::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:27:46,834::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:27:47,230::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0499246 s, 20.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:27:49,241::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:27:49,753::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0775638 s, 13.2 MB/s\n'; <rc> = 0 >Thread-15912392::INFO::2013-11-20 03:27:51,442::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:27:51,443::task::495::TaskManager.Task::(_debug) Task a49a3a13-d0cf-488e-a28b-0e8baaa6544d: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:27:51,445::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`08e3f4bd-3609-4279-9122-2404a084e552`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:27:51,445::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:27:51,446::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:27:51,446::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`08e3f4bd-3609-4279-9122-2404a084e552`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:27:51,447::task::495::TaskManager.Task::(_debug) Task a49a3a13-d0cf-488e-a28b-0e8baaa6544d: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:27:51,448::task::495::TaskManager.Task::(_debug) Task a49a3a13-d0cf-488e-a28b-0e8baaa6544d: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:27:51,448::task::495::TaskManager.Task::(_debug) Task a49a3a13-d0cf-488e-a28b-0e8baaa6544d: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:27:51,449::task::495::TaskManager.Task::(_debug) Task a49a3a13-d0cf-488e-a28b-0e8baaa6544d: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:27:51,449::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:27:51,450::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:27:51,450::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:27:51,451::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:27:51,451::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:27:51,452::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:27:51,452::task::495::TaskManager.Task::(_debug) Task a49a3a13-d0cf-488e-a28b-0e8baaa6544d: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:27:51,453::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:27:51,769::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:27:52,188::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0576808 s, 17.8 MB/s\n'; <rc> = 0 >Thread-15914272::DEBUG::2013-11-20 03:27:54,195::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914272::INFO::2013-11-20 03:27:54,195::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914272::DEBUG::2013-11-20 03:27:54,196::task::495::TaskManager.Task::(_debug) Task e1eeac32-e072-4b96-9b67-f780259b3d8f: moving from state init -> state preparing >Thread-15914272::DEBUG::2013-11-20 03:27:54,196::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914272::DEBUG::2013-11-20 03:27:54,197::task::495::TaskManager.Task::(_debug) Task e1eeac32-e072-4b96-9b67-f780259b3d8f: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914272::DEBUG::2013-11-20 03:27:54,197::task::495::TaskManager.Task::(_debug) Task e1eeac32-e072-4b96-9b67-f780259b3d8f: moving from state preparing -> state finished >Thread-15914272::DEBUG::2013-11-20 03:27:54,197::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914272::DEBUG::2013-11-20 03:27:54,198::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914272::DEBUG::2013-11-20 03:27:54,198::task::495::TaskManager.Task::(_debug) Task e1eeac32-e072-4b96-9b67-f780259b3d8f: ref 0 aborting False >Thread-15914272::INFO::2013-11-20 03:27:54,199::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Dummy-15912651::DEBUG::2013-11-20 03:27:54,203::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914274::INFO::2013-11-20 03:27:54,406::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914273::DEBUG::2013-11-20 03:27:54,408::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914274::DEBUG::2013-11-20 03:27:54,409::task::495::TaskManager.Task::(_debug) Task 9013f00e-e287-47d3-9689-2910f927bb1c: moving from state init -> state preparing >Thread-15914273::INFO::2013-11-20 03:27:54,410::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914274::DEBUG::2013-11-20 03:27:54,412::task::495::TaskManager.Task::(_debug) Task 9013f00e-e287-47d3-9689-2910f927bb1c: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000416994094849', 'lastCheck': 1384918073.523025, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00126504898071', 'lastCheck': 1384918069.5144041, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00130677223206', 'lastCheck': 1384918071.695323, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000401973724365', 'lastCheck': 1384918072.0846519, 'valid': True, 'code': 0}} >Thread-15914273::DEBUG::2013-11-20 03:27:54,413::task::495::TaskManager.Task::(_debug) Task d1a87c23-50dc-4f59-8024-274818e76465: moving from state init -> state preparing >Thread-15914274::DEBUG::2013-11-20 03:27:54,413::task::495::TaskManager.Task::(_debug) Task 9013f00e-e287-47d3-9689-2910f927bb1c: moving from state preparing -> state finished >Thread-15914273::DEBUG::2013-11-20 03:27:54,420::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`77c94afe-ded8-444b-8978-ce4985a7826a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914274::DEBUG::2013-11-20 03:27:54,421::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914273::DEBUG::2013-11-20 03:27:54,421::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914274::DEBUG::2013-11-20 03:27:54,422::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914274::DEBUG::2013-11-20 03:27:54,424::task::495::TaskManager.Task::(_debug) Task 9013f00e-e287-47d3-9689-2910f927bb1c: ref 0 aborting False >Thread-15914273::DEBUG::2013-11-20 03:27:54,425::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914274::INFO::2013-11-20 03:27:54,425::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000416994094849', 'lastCheck': 1384918073.523025, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00126504898071', 'lastCheck': 1384918069.5144041, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00130677223206', 'lastCheck': 1384918071.695323, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000401973724365', 'lastCheck': 1384918072.0846519, 'valid': True, 'code': 0}} >Thread-15914273::DEBUG::2013-11-20 03:27:54,426::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`77c94afe-ded8-444b-8978-ce4985a7826a`::Granted request >Thread-15914273::DEBUG::2013-11-20 03:27:54,428::task::495::TaskManager.Task::(_debug) Task d1a87c23-50dc-4f59-8024-274818e76465: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914273::DEBUG::2013-11-20 03:27:54,429::task::495::TaskManager.Task::(_debug) Task d1a87c23-50dc-4f59-8024-274818e76465: ref 1 aborting False >Thread-15914273::WARNING::2013-11-20 03:27:54,432::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914273::DEBUG::2013-11-20 03:27:54,433::task::495::TaskManager.Task::(_debug) Task d1a87c23-50dc-4f59-8024-274818e76465: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914273::DEBUG::2013-11-20 03:27:54,439::task::495::TaskManager.Task::(_debug) Task d1a87c23-50dc-4f59-8024-274818e76465: moving from state preparing -> state finished >Thread-15914273::DEBUG::2013-11-20 03:27:54,440::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914273::DEBUG::2013-11-20 03:27:54,440::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914273::DEBUG::2013-11-20 03:27:54,443::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914273::DEBUG::2013-11-20 03:27:54,444::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914273::DEBUG::2013-11-20 03:27:54,444::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914273::DEBUG::2013-11-20 03:27:54,445::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914273::DEBUG::2013-11-20 03:27:54,446::task::495::TaskManager.Task::(_debug) Task d1a87c23-50dc-4f59-8024-274818e76465: ref 0 aborting False >Thread-15914273::INFO::2013-11-20 03:27:54,447::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:27:54,689::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0443722 s, 23.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:27:56,700::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:27:57,216::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.074889 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:27:59,231::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:27:59,768::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0755837 s, 13.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:28:01,784::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:28:02,238::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0504062 s, 20.3 MB/s\n'; <rc> = 0 >Thread-15913518::INFO::2013-11-20 03:28:02,482::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:28:02,483::task::495::TaskManager.Task::(_debug) Task e5c716bb-32d9-43b2-ab14-89f460aed84d: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:28:02,484::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`2504678f-ec5a-4e73-9183-8f718dcdf0de`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:28:02,484::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:28:02,485::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:28:02,485::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`2504678f-ec5a-4e73-9183-8f718dcdf0de`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:28:02,486::task::495::TaskManager.Task::(_debug) Task e5c716bb-32d9-43b2-ab14-89f460aed84d: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:28:02,486::task::495::TaskManager.Task::(_debug) Task e5c716bb-32d9-43b2-ab14-89f460aed84d: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:28:02,487::task::495::TaskManager.Task::(_debug) Task e5c716bb-32d9-43b2-ab14-89f460aed84d: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:28:02,488::task::495::TaskManager.Task::(_debug) Task e5c716bb-32d9-43b2-ab14-89f460aed84d: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:28:02,488::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:28:02,489::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:28:02,490::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:28:02,490::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:28:02,491::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:28:02,491::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:28:02,492::task::495::TaskManager.Task::(_debug) Task e5c716bb-32d9-43b2-ab14-89f460aed84d: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:28:02,493::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:28:02,493::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:28:02,493::task::495::TaskManager.Task::(_debug) Task 19ed75ba-1d75-4e0e-a65f-985af1099396: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:28:02,495::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`dc780351-988a-4dd5-874f-0c3c2577beff`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:28:02,495::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:28:02,495::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:28:02,496::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`dc780351-988a-4dd5-874f-0c3c2577beff`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:28:02,496::task::495::TaskManager.Task::(_debug) Task 19ed75ba-1d75-4e0e-a65f-985af1099396: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:28:02,497::task::495::TaskManager.Task::(_debug) Task 19ed75ba-1d75-4e0e-a65f-985af1099396: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:28:02,497::task::495::TaskManager.Task::(_debug) Task 19ed75ba-1d75-4e0e-a65f-985af1099396: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:28:02,498::task::495::TaskManager.Task::(_debug) Task 19ed75ba-1d75-4e0e-a65f-985af1099396: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:28:02,498::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:28:02,499::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:28:02,500::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:28:02,501::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:28:02,501::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:28:02,501::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:28:02,502::task::495::TaskManager.Task::(_debug) Task 19ed75ba-1d75-4e0e-a65f-985af1099396: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:28:02,502::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:28:04,249::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:28:04,678::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0570404 s, 18.0 MB/s\n'; <rc> = 0 >Thread-15914279::DEBUG::2013-11-20 03:28:04,881::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914279::INFO::2013-11-20 03:28:04,882::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914279::DEBUG::2013-11-20 03:28:04,883::task::495::TaskManager.Task::(_debug) Task e270dff2-6f3a-4abf-9e45-2148dacdae99: moving from state init -> state preparing >Thread-15914279::DEBUG::2013-11-20 03:28:04,883::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914279::DEBUG::2013-11-20 03:28:04,884::task::495::TaskManager.Task::(_debug) Task e270dff2-6f3a-4abf-9e45-2148dacdae99: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914279::DEBUG::2013-11-20 03:28:04,884::task::495::TaskManager.Task::(_debug) Task e270dff2-6f3a-4abf-9e45-2148dacdae99: moving from state preparing -> state finished >Thread-15914279::DEBUG::2013-11-20 03:28:04,884::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914279::DEBUG::2013-11-20 03:28:04,885::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914279::DEBUG::2013-11-20 03:28:04,885::task::495::TaskManager.Task::(_debug) Task e270dff2-6f3a-4abf-9e45-2148dacdae99: ref 0 aborting False >Thread-15914279::INFO::2013-11-20 03:28:04,885::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914280::DEBUG::2013-11-20 03:28:04,897::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914280::INFO::2013-11-20 03:28:04,898::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914280::DEBUG::2013-11-20 03:28:04,899::task::495::TaskManager.Task::(_debug) Task c59c0c0b-13ea-4fd7-a1ae-f32fe6776afc: moving from state init -> state preparing >Thread-15914280::DEBUG::2013-11-20 03:28:04,900::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`7e305418-a961-437b-b24d-dfb454e3221e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914280::DEBUG::2013-11-20 03:28:04,900::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914280::DEBUG::2013-11-20 03:28:04,901::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914280::DEBUG::2013-11-20 03:28:04,902::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`7e305418-a961-437b-b24d-dfb454e3221e`::Granted request >Thread-15914280::DEBUG::2013-11-20 03:28:04,903::task::495::TaskManager.Task::(_debug) Task c59c0c0b-13ea-4fd7-a1ae-f32fe6776afc: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914280::DEBUG::2013-11-20 03:28:04,903::task::495::TaskManager.Task::(_debug) Task c59c0c0b-13ea-4fd7-a1ae-f32fe6776afc: ref 1 aborting False >Thread-15914280::WARNING::2013-11-20 03:28:04,905::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914280::DEBUG::2013-11-20 03:28:04,906::task::495::TaskManager.Task::(_debug) Task c59c0c0b-13ea-4fd7-a1ae-f32fe6776afc: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914280::DEBUG::2013-11-20 03:28:04,907::task::495::TaskManager.Task::(_debug) Task c59c0c0b-13ea-4fd7-a1ae-f32fe6776afc: moving from state preparing -> state finished >Thread-15914280::DEBUG::2013-11-20 03:28:04,907::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914280::DEBUG::2013-11-20 03:28:04,908::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914280::DEBUG::2013-11-20 03:28:04,908::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914280::DEBUG::2013-11-20 03:28:04,909::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914280::DEBUG::2013-11-20 03:28:04,909::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914280::DEBUG::2013-11-20 03:28:04,910::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914280::DEBUG::2013-11-20 03:28:04,910::task::495::TaskManager.Task::(_debug) Task c59c0c0b-13ea-4fd7-a1ae-f32fe6776afc: ref 0 aborting False >Thread-15914280::INFO::2013-11-20 03:28:04,911::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:28:06,694::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:28:07,215::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.073595 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:28:09,228::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:28:09,748::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0817558 s, 12.5 MB/s\n'; <rc> = 0 >Thread-15914282::INFO::2013-11-20 03:28:10,465::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914282::DEBUG::2013-11-20 03:28:10,466::task::495::TaskManager.Task::(_debug) Task bba06498-1820-48eb-be05-dbbb4f5f3813: moving from state init -> state preparing >Thread-15914282::DEBUG::2013-11-20 03:28:10,466::task::495::TaskManager.Task::(_debug) Task bba06498-1820-48eb-be05-dbbb4f5f3813: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000304937362671', 'lastCheck': 1384918083.535651, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00163698196411', 'lastCheck': 1384918089.5788319, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00153398513794', 'lastCheck': 1384918081.7109261, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00044322013855', 'lastCheck': 1384918082.0956869, 'valid': True, 'code': 0}} >Thread-15914282::DEBUG::2013-11-20 03:28:10,475::task::495::TaskManager.Task::(_debug) Task bba06498-1820-48eb-be05-dbbb4f5f3813: moving from state preparing -> state finished >Thread-15914282::DEBUG::2013-11-20 03:28:10,484::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914282::DEBUG::2013-11-20 03:28:10,486::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914282::DEBUG::2013-11-20 03:28:10,486::task::495::TaskManager.Task::(_debug) Task bba06498-1820-48eb-be05-dbbb4f5f3813: ref 0 aborting False >Thread-15914282::INFO::2013-11-20 03:28:10,488::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000304937362671', 'lastCheck': 1384918083.535651, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00163698196411', 'lastCheck': 1384918089.5788319, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00153398513794', 'lastCheck': 1384918081.7109261, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00044322013855', 'lastCheck': 1384918082.0956869, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:28:11,764::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:28:12,280::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0776105 s, 13.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:28:14,295::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:28:14,726::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0481314 s, 21.3 MB/s\n'; <rc> = 0 >Thread-15914285::DEBUG::2013-11-20 03:28:15,049::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914285::INFO::2013-11-20 03:28:15,050::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914285::DEBUG::2013-11-20 03:28:15,051::task::495::TaskManager.Task::(_debug) Task e53dcd56-18e1-42bc-97ba-83ff4f928f09: moving from state init -> state preparing >Thread-15914285::DEBUG::2013-11-20 03:28:15,052::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914285::DEBUG::2013-11-20 03:28:15,053::task::495::TaskManager.Task::(_debug) Task e53dcd56-18e1-42bc-97ba-83ff4f928f09: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914285::DEBUG::2013-11-20 03:28:15,053::task::495::TaskManager.Task::(_debug) Task e53dcd56-18e1-42bc-97ba-83ff4f928f09: moving from state preparing -> state finished >Thread-15914285::DEBUG::2013-11-20 03:28:15,054::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914285::DEBUG::2013-11-20 03:28:15,055::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914285::DEBUG::2013-11-20 03:28:15,055::task::495::TaskManager.Task::(_debug) Task e53dcd56-18e1-42bc-97ba-83ff4f928f09: ref 0 aborting False >Thread-15914285::INFO::2013-11-20 03:28:15,056::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914286::DEBUG::2013-11-20 03:28:15,064::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914286::INFO::2013-11-20 03:28:15,065::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914286::DEBUG::2013-11-20 03:28:15,066::task::495::TaskManager.Task::(_debug) Task 33cb7545-1572-48e7-b0c6-47e7bd55bcef: moving from state init -> state preparing >Thread-15914286::DEBUG::2013-11-20 03:28:15,071::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`692c3f80-ada3-4fea-abdc-34074462be5d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914286::DEBUG::2013-11-20 03:28:15,071::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914286::DEBUG::2013-11-20 03:28:15,080::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914286::DEBUG::2013-11-20 03:28:15,081::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`692c3f80-ada3-4fea-abdc-34074462be5d`::Granted request >Thread-15914286::DEBUG::2013-11-20 03:28:15,082::task::495::TaskManager.Task::(_debug) Task 33cb7545-1572-48e7-b0c6-47e7bd55bcef: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914286::DEBUG::2013-11-20 03:28:15,083::task::495::TaskManager.Task::(_debug) Task 33cb7545-1572-48e7-b0c6-47e7bd55bcef: ref 1 aborting False >Thread-15914286::WARNING::2013-11-20 03:28:15,085::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914286::DEBUG::2013-11-20 03:28:15,093::task::495::TaskManager.Task::(_debug) Task 33cb7545-1572-48e7-b0c6-47e7bd55bcef: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914286::DEBUG::2013-11-20 03:28:15,093::task::495::TaskManager.Task::(_debug) Task 33cb7545-1572-48e7-b0c6-47e7bd55bcef: moving from state preparing -> state finished >Thread-15914286::DEBUG::2013-11-20 03:28:15,094::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914286::DEBUG::2013-11-20 03:28:15,094::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914286::DEBUG::2013-11-20 03:28:15,094::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914286::DEBUG::2013-11-20 03:28:15,095::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914286::DEBUG::2013-11-20 03:28:15,095::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914286::DEBUG::2013-11-20 03:28:15,095::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914286::DEBUG::2013-11-20 03:28:15,096::task::495::TaskManager.Task::(_debug) Task 33cb7545-1572-48e7-b0c6-47e7bd55bcef: ref 0 aborting False >Thread-15914286::INFO::2013-11-20 03:28:15,096::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:28:16,737::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913546::INFO::2013-11-20 03:28:17,036::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:28:17,038::task::495::TaskManager.Task::(_debug) Task f47c7510-d862-45fb-8938-14ace3023757: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:28:17,040::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`e5e54a71-1dae-4c0c-8d9e-63ed7c400265`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:28:17,041::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:28:17,041::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:28:17,042::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`e5e54a71-1dae-4c0c-8d9e-63ed7c400265`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:28:17,042::task::495::TaskManager.Task::(_debug) Task f47c7510-d862-45fb-8938-14ace3023757: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:28:17,043::task::495::TaskManager.Task::(_debug) Task f47c7510-d862-45fb-8938-14ace3023757: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:28:17,043::task::495::TaskManager.Task::(_debug) Task f47c7510-d862-45fb-8938-14ace3023757: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:28:17,044::task::495::TaskManager.Task::(_debug) Task f47c7510-d862-45fb-8938-14ace3023757: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:28:17,044::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:28:17,045::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:28:17,045::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:28:17,046::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:28:17,046::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:28:17,046::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:28:17,047::task::495::TaskManager.Task::(_debug) Task f47c7510-d862-45fb-8938-14ace3023757: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:28:17,047::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:28:17,048::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:28:17,048::task::495::TaskManager.Task::(_debug) Task c40cdb3b-3f9e-4e31-9bbe-f7e6d0d26927: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:28:17,049::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`9ed91156-16af-477c-8b83-6bc3822d8fba`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:28:17,049::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:28:17,050::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:28:17,050::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`9ed91156-16af-477c-8b83-6bc3822d8fba`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:28:17,050::task::495::TaskManager.Task::(_debug) Task c40cdb3b-3f9e-4e31-9bbe-f7e6d0d26927: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:28:17,051::task::495::TaskManager.Task::(_debug) Task c40cdb3b-3f9e-4e31-9bbe-f7e6d0d26927: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:28:17,051::task::495::TaskManager.Task::(_debug) Task c40cdb3b-3f9e-4e31-9bbe-f7e6d0d26927: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:28:17,051::task::495::TaskManager.Task::(_debug) Task c40cdb3b-3f9e-4e31-9bbe-f7e6d0d26927: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:28:17,052::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:28:17,052::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:28:17,053::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:28:17,053::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:28:17,053::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:28:17,054::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:28:17,054::task::495::TaskManager.Task::(_debug) Task c40cdb3b-3f9e-4e31-9bbe-f7e6d0d26927: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:28:17,054::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:28:17,055::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:28:17,055::task::495::TaskManager.Task::(_debug) Task 7178c219-c10a-4faa-9a46-66dd3f988558: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:28:17,056::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`f28fc690-9a2d-459b-8ae0-1ce2c1f20ca8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:28:17,056::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:28:17,057::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:28:17,057::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`f28fc690-9a2d-459b-8ae0-1ce2c1f20ca8`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:28:17,058::task::495::TaskManager.Task::(_debug) Task 7178c219-c10a-4faa-9a46-66dd3f988558: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:28:17,058::task::495::TaskManager.Task::(_debug) Task 7178c219-c10a-4faa-9a46-66dd3f988558: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:28:17,058::task::495::TaskManager.Task::(_debug) Task 7178c219-c10a-4faa-9a46-66dd3f988558: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:28:17,059::task::495::TaskManager.Task::(_debug) Task 7178c219-c10a-4faa-9a46-66dd3f988558: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:28:17,059::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:28:17,060::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:28:17,060::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:28:17,060::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:28:17,061::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:28:17,061::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:28:17,061::task::495::TaskManager.Task::(_debug) Task 7178c219-c10a-4faa-9a46-66dd3f988558: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:28:17,062::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:28:17,062::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:28:17,062::task::495::TaskManager.Task::(_debug) Task 42024f08-6a09-4e14-8a9c-d2eb5308ebc5: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:28:17,063::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`5c0eba02-e66b-4108-ad7b-4428f06715a5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:28:17,063::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:28:17,064::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:28:17,064::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`5c0eba02-e66b-4108-ad7b-4428f06715a5`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:28:17,065::task::495::TaskManager.Task::(_debug) Task 42024f08-6a09-4e14-8a9c-d2eb5308ebc5: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:28:17,065::task::495::TaskManager.Task::(_debug) Task 42024f08-6a09-4e14-8a9c-d2eb5308ebc5: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:28:17,065::task::495::TaskManager.Task::(_debug) Task 42024f08-6a09-4e14-8a9c-d2eb5308ebc5: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:28:17,066::task::495::TaskManager.Task::(_debug) Task 42024f08-6a09-4e14-8a9c-d2eb5308ebc5: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:28:17,067::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:28:17,067::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:28:17,067::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:28:17,068::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:28:17,069::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:28:17,070::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:28:17,070::task::495::TaskManager.Task::(_debug) Task 42024f08-6a09-4e14-8a9c-d2eb5308ebc5: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:28:17,071::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:28:17,274::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0841064 s, 12.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:28:19,288::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:28:19,792::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0680418 s, 15.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:28:21,805::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:28:22,311::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0737721 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:28:24,327::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:28:24,847::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0847635 s, 12.1 MB/s\n'; <rc> = 0 >Thread-15914290::DEBUG::2013-11-20 03:28:25,212::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914290::INFO::2013-11-20 03:28:25,213::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914290::DEBUG::2013-11-20 03:28:25,214::task::495::TaskManager.Task::(_debug) Task 5ecedf84-a6d8-4fa2-8af2-c59736e253b1: moving from state init -> state preparing >Thread-15914290::DEBUG::2013-11-20 03:28:25,215::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914290::DEBUG::2013-11-20 03:28:25,216::task::495::TaskManager.Task::(_debug) Task 5ecedf84-a6d8-4fa2-8af2-c59736e253b1: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914290::DEBUG::2013-11-20 03:28:25,216::task::495::TaskManager.Task::(_debug) Task 5ecedf84-a6d8-4fa2-8af2-c59736e253b1: moving from state preparing -> state finished >Thread-15914290::DEBUG::2013-11-20 03:28:25,217::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914290::DEBUG::2013-11-20 03:28:25,217::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914290::DEBUG::2013-11-20 03:28:25,218::task::495::TaskManager.Task::(_debug) Task 5ecedf84-a6d8-4fa2-8af2-c59736e253b1: ref 0 aborting False >Thread-15914290::INFO::2013-11-20 03:28:25,219::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914291::DEBUG::2013-11-20 03:28:25,222::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914291::INFO::2013-11-20 03:28:25,223::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914291::DEBUG::2013-11-20 03:28:25,223::task::495::TaskManager.Task::(_debug) Task 4bf4754d-35a8-421a-9689-9688cb565c2a: moving from state init -> state preparing >Thread-15914291::DEBUG::2013-11-20 03:28:25,224::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`06a2a1f6-1150-49af-918f-f42a007ca52f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914291::DEBUG::2013-11-20 03:28:25,225::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914291::DEBUG::2013-11-20 03:28:25,225::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914291::DEBUG::2013-11-20 03:28:25,226::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`06a2a1f6-1150-49af-918f-f42a007ca52f`::Granted request >Thread-15914291::DEBUG::2013-11-20 03:28:25,226::task::495::TaskManager.Task::(_debug) Task 4bf4754d-35a8-421a-9689-9688cb565c2a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914291::DEBUG::2013-11-20 03:28:25,227::task::495::TaskManager.Task::(_debug) Task 4bf4754d-35a8-421a-9689-9688cb565c2a: ref 1 aborting False >Thread-15914291::WARNING::2013-11-20 03:28:25,228::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914291::DEBUG::2013-11-20 03:28:25,229::task::495::TaskManager.Task::(_debug) Task 4bf4754d-35a8-421a-9689-9688cb565c2a: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914291::DEBUG::2013-11-20 03:28:25,229::task::495::TaskManager.Task::(_debug) Task 4bf4754d-35a8-421a-9689-9688cb565c2a: moving from state preparing -> state finished >Thread-15914291::DEBUG::2013-11-20 03:28:25,230::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914291::DEBUG::2013-11-20 03:28:25,230::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914291::DEBUG::2013-11-20 03:28:25,231::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914291::DEBUG::2013-11-20 03:28:25,231::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914291::DEBUG::2013-11-20 03:28:25,231::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914291::DEBUG::2013-11-20 03:28:25,232::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914291::DEBUG::2013-11-20 03:28:25,232::task::495::TaskManager.Task::(_debug) Task 4bf4754d-35a8-421a-9689-9688cb565c2a: ref 0 aborting False >Thread-15914291::INFO::2013-11-20 03:28:25,232::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914292::INFO::2013-11-20 03:28:26,012::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914292::DEBUG::2013-11-20 03:28:26,013::task::495::TaskManager.Task::(_debug) Task 15e70cd1-7e94-4646-8563-0942416e9fa5: moving from state init -> state preparing >Thread-15914292::DEBUG::2013-11-20 03:28:26,013::task::495::TaskManager.Task::(_debug) Task 15e70cd1-7e94-4646-8563-0942416e9fa5: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000411033630371', 'lastCheck': 1384918103.567255, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00165200233459', 'lastCheck': 1384918099.595237, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00134110450745', 'lastCheck': 1384918102.0013189, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000554084777832', 'lastCheck': 1384918102.115217, 'valid': True, 'code': 0}} >Thread-15914292::DEBUG::2013-11-20 03:28:26,014::task::495::TaskManager.Task::(_debug) Task 15e70cd1-7e94-4646-8563-0942416e9fa5: moving from state preparing -> state finished >Thread-15914292::DEBUG::2013-11-20 03:28:26,015::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914292::DEBUG::2013-11-20 03:28:26,016::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914292::DEBUG::2013-11-20 03:28:26,017::task::495::TaskManager.Task::(_debug) Task 15e70cd1-7e94-4646-8563-0942416e9fa5: ref 0 aborting False >Thread-15914292::INFO::2013-11-20 03:28:26,017::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000411033630371', 'lastCheck': 1384918103.567255, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00165200233459', 'lastCheck': 1384918099.595237, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00134110450745', 'lastCheck': 1384918102.0013189, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000554084777832', 'lastCheck': 1384918102.115217, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:28:26,862::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913639::INFO::2013-11-20 03:28:27,068::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:28:27,076::task::495::TaskManager.Task::(_debug) Task aa4c8ab0-98d1-449e-be44-d198dcc8b010: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:28:27,079::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`aa373961-c850-4a91-96d4-9aa02a0abb89`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:28:27,079::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:28:27,080::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:28:27,081::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`aa373961-c850-4a91-96d4-9aa02a0abb89`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:28:27,082::task::495::TaskManager.Task::(_debug) Task aa4c8ab0-98d1-449e-be44-d198dcc8b010: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:28:27,083::task::495::TaskManager.Task::(_debug) Task aa4c8ab0-98d1-449e-be44-d198dcc8b010: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:28:27,084::task::495::TaskManager.Task::(_debug) Task aa4c8ab0-98d1-449e-be44-d198dcc8b010: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:28:27,085::task::495::TaskManager.Task::(_debug) Task aa4c8ab0-98d1-449e-be44-d198dcc8b010: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:28:27,086::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:28:27,086::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:28:27,087::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:28:27,088::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:28:27,088::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:28:27,089::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:28:27,090::task::495::TaskManager.Task::(_debug) Task aa4c8ab0-98d1-449e-be44-d198dcc8b010: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:28:27,091::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:28:27,377::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.067159 s, 15.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:28:29,391::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:28:29,887::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0762699 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:28:31,903::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:28:32,470::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0826379 s, 12.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:28:34,483::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:28:35,008::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.077081 s, 13.3 MB/s\n'; <rc> = 0 >Thread-15914297::DEBUG::2013-11-20 03:28:35,368::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914297::INFO::2013-11-20 03:28:35,370::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914297::DEBUG::2013-11-20 03:28:35,370::task::495::TaskManager.Task::(_debug) Task 6b56c5a1-b94f-48de-bcea-3067afcdde68: moving from state init -> state preparing >Thread-15914297::DEBUG::2013-11-20 03:28:35,371::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914297::DEBUG::2013-11-20 03:28:35,372::task::495::TaskManager.Task::(_debug) Task 6b56c5a1-b94f-48de-bcea-3067afcdde68: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914297::DEBUG::2013-11-20 03:28:35,372::task::495::TaskManager.Task::(_debug) Task 6b56c5a1-b94f-48de-bcea-3067afcdde68: moving from state preparing -> state finished >Thread-15914297::DEBUG::2013-11-20 03:28:35,373::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914297::DEBUG::2013-11-20 03:28:35,373::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914297::DEBUG::2013-11-20 03:28:35,374::task::495::TaskManager.Task::(_debug) Task 6b56c5a1-b94f-48de-bcea-3067afcdde68: ref 0 aborting False >Thread-15914297::INFO::2013-11-20 03:28:35,374::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914298::DEBUG::2013-11-20 03:28:35,379::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914298::INFO::2013-11-20 03:28:35,380::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914298::DEBUG::2013-11-20 03:28:35,381::task::495::TaskManager.Task::(_debug) Task 9d10d974-231b-4252-95b5-feb80f655afe: moving from state init -> state preparing >Thread-15914298::DEBUG::2013-11-20 03:28:35,382::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`49f3a758-8bc0-4be2-aaa4-2a80a2a00dcf`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914298::DEBUG::2013-11-20 03:28:35,383::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914298::DEBUG::2013-11-20 03:28:35,384::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914298::DEBUG::2013-11-20 03:28:35,384::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`49f3a758-8bc0-4be2-aaa4-2a80a2a00dcf`::Granted request >Thread-15914298::DEBUG::2013-11-20 03:28:35,385::task::495::TaskManager.Task::(_debug) Task 9d10d974-231b-4252-95b5-feb80f655afe: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914298::DEBUG::2013-11-20 03:28:35,386::task::495::TaskManager.Task::(_debug) Task 9d10d974-231b-4252-95b5-feb80f655afe: ref 1 aborting False >Thread-15914298::WARNING::2013-11-20 03:28:35,388::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914298::DEBUG::2013-11-20 03:28:35,389::task::495::TaskManager.Task::(_debug) Task 9d10d974-231b-4252-95b5-feb80f655afe: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914298::DEBUG::2013-11-20 03:28:35,390::task::495::TaskManager.Task::(_debug) Task 9d10d974-231b-4252-95b5-feb80f655afe: moving from state preparing -> state finished >Thread-15914298::DEBUG::2013-11-20 03:28:35,391::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914298::DEBUG::2013-11-20 03:28:35,391::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914298::DEBUG::2013-11-20 03:28:35,392::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914298::DEBUG::2013-11-20 03:28:35,393::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914298::DEBUG::2013-11-20 03:28:35,393::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914298::DEBUG::2013-11-20 03:28:35,394::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914298::DEBUG::2013-11-20 03:28:35,394::task::495::TaskManager.Task::(_debug) Task 9d10d974-231b-4252-95b5-feb80f655afe: ref 0 aborting False >Thread-15914298::INFO::2013-11-20 03:28:35,395::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115404177408', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:28:37,023::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:28:37,538::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0780063 s, 13.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:28:39,553::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:28:40,001::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0584619 s, 17.5 MB/s\n'; <rc> = 0 >Thread-15914300::INFO::2013-11-20 03:28:41,412::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914300::DEBUG::2013-11-20 03:28:41,413::task::495::TaskManager.Task::(_debug) Task 86272dbf-cf7b-4282-898f-d48a42af6929: moving from state init -> state preparing >Thread-15914300::DEBUG::2013-11-20 03:28:41,413::task::495::TaskManager.Task::(_debug) Task 86272dbf-cf7b-4282-898f-d48a42af6929: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000328063964844', 'lastCheck': 1384918113.583462, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.139843940735', 'lastCheck': 1384918119.8902819, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00206017494202', 'lastCheck': 1384918112.1712279, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0167138576508', 'lastCheck': 1384918112.156785, 'valid': True, 'code': 0}} >Thread-15914300::DEBUG::2013-11-20 03:28:41,426::task::495::TaskManager.Task::(_debug) Task 86272dbf-cf7b-4282-898f-d48a42af6929: moving from state preparing -> state finished >Thread-15914300::DEBUG::2013-11-20 03:28:41,427::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914300::DEBUG::2013-11-20 03:28:41,427::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914300::DEBUG::2013-11-20 03:28:41,428::task::495::TaskManager.Task::(_debug) Task 86272dbf-cf7b-4282-898f-d48a42af6929: ref 0 aborting False >Thread-15914300::INFO::2013-11-20 03:28:41,429::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000328063964844', 'lastCheck': 1384918113.583462, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.139843940735', 'lastCheck': 1384918119.8902819, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00206017494202', 'lastCheck': 1384918112.1712279, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.0167138576508', 'lastCheck': 1384918112.156785, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:28:42,011::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:28:42,463::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.072042 s, 14.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:28:44,478::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913695::INFO::2013-11-20 03:28:44,825::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:28:44,826::task::495::TaskManager.Task::(_debug) Task 377782b3-8556-4f5d-9438-6171e2c0dd58: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:28:44,827::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`947396ca-7ec7-4cb0-991a-bc80b5bf4411`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:28:44,827::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:28:44,828::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:28:44,828::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`947396ca-7ec7-4cb0-991a-bc80b5bf4411`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:28:44,829::task::495::TaskManager.Task::(_debug) Task 377782b3-8556-4f5d-9438-6171e2c0dd58: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:28:44,829::task::495::TaskManager.Task::(_debug) Task 377782b3-8556-4f5d-9438-6171e2c0dd58: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:28:44,830::task::495::TaskManager.Task::(_debug) Task 377782b3-8556-4f5d-9438-6171e2c0dd58: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:28:44,830::task::495::TaskManager.Task::(_debug) Task 377782b3-8556-4f5d-9438-6171e2c0dd58: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:28:44,831::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:28:44,831::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:28:44,831::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:28:44,832::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:28:44,832::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:28:44,832::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:28:44,833::task::495::TaskManager.Task::(_debug) Task 377782b3-8556-4f5d-9438-6171e2c0dd58: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:28:44,833::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:28:44,996::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0811123 s, 12.6 MB/s\n'; <rc> = 0 >Thread-15914303::DEBUG::2013-11-20 03:28:46,667::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914303::INFO::2013-11-20 03:28:46,668::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914303::DEBUG::2013-11-20 03:28:46,668::task::495::TaskManager.Task::(_debug) Task 5104de9f-427e-4cc1-a9d9-b3d4bded23b4: moving from state init -> state preparing >Thread-15914303::DEBUG::2013-11-20 03:28:46,669::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914303::DEBUG::2013-11-20 03:28:46,669::task::495::TaskManager.Task::(_debug) Task 5104de9f-427e-4cc1-a9d9-b3d4bded23b4: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914303::DEBUG::2013-11-20 03:28:46,670::task::495::TaskManager.Task::(_debug) Task 5104de9f-427e-4cc1-a9d9-b3d4bded23b4: moving from state preparing -> state finished >Thread-15914303::DEBUG::2013-11-20 03:28:46,671::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914303::DEBUG::2013-11-20 03:28:46,671::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914303::DEBUG::2013-11-20 03:28:46,672::task::495::TaskManager.Task::(_debug) Task 5104de9f-427e-4cc1-a9d9-b3d4bded23b4: ref 0 aborting False >Thread-15914303::INFO::2013-11-20 03:28:46,673::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914304::DEBUG::2013-11-20 03:28:46,676::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914304::INFO::2013-11-20 03:28:46,677::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914304::DEBUG::2013-11-20 03:28:46,677::task::495::TaskManager.Task::(_debug) Task 92dbe56d-09af-41cf-90e2-dfa8728e0c58: moving from state init -> state preparing >Thread-15914304::DEBUG::2013-11-20 03:28:46,678::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f55704db-070f-4c4f-bfb8-4ecdd4508e79`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914304::DEBUG::2013-11-20 03:28:46,679::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914304::DEBUG::2013-11-20 03:28:46,679::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914304::DEBUG::2013-11-20 03:28:46,680::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f55704db-070f-4c4f-bfb8-4ecdd4508e79`::Granted request >Thread-15914304::DEBUG::2013-11-20 03:28:46,681::task::495::TaskManager.Task::(_debug) Task 92dbe56d-09af-41cf-90e2-dfa8728e0c58: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914304::DEBUG::2013-11-20 03:28:46,682::task::495::TaskManager.Task::(_debug) Task 92dbe56d-09af-41cf-90e2-dfa8728e0c58: ref 1 aborting False >Thread-15914304::WARNING::2013-11-20 03:28:46,683::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914304::DEBUG::2013-11-20 03:28:46,691::task::495::TaskManager.Task::(_debug) Task 92dbe56d-09af-41cf-90e2-dfa8728e0c58: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115214385152', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115214385152', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914304::DEBUG::2013-11-20 03:28:46,696::task::495::TaskManager.Task::(_debug) Task 92dbe56d-09af-41cf-90e2-dfa8728e0c58: moving from state preparing -> state finished >Thread-15914304::DEBUG::2013-11-20 03:28:46,697::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914304::DEBUG::2013-11-20 03:28:46,698::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914304::DEBUG::2013-11-20 03:28:46,699::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914304::DEBUG::2013-11-20 03:28:46,700::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914304::DEBUG::2013-11-20 03:28:46,701::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914304::DEBUG::2013-11-20 03:28:46,701::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914304::DEBUG::2013-11-20 03:28:46,702::task::495::TaskManager.Task::(_debug) Task 92dbe56d-09af-41cf-90e2-dfa8728e0c58: ref 0 aborting False >Thread-15914304::INFO::2013-11-20 03:28:46,703::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115214385152', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115214385152', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:28:47,011::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913658::INFO::2013-11-20 03:28:47,245::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:28:47,254::task::495::TaskManager.Task::(_debug) Task 436f729b-2c91-43e7-9b0e-555656dd9d7d: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:28:47,256::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`9f41fc22-685a-4c04-b6e8-f82a60be91f6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:28:47,257::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:28:47,257::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:28:47,258::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`9f41fc22-685a-4c04-b6e8-f82a60be91f6`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:28:47,258::task::495::TaskManager.Task::(_debug) Task 436f729b-2c91-43e7-9b0e-555656dd9d7d: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:28:47,259::task::495::TaskManager.Task::(_debug) Task 436f729b-2c91-43e7-9b0e-555656dd9d7d: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:28:47,259::task::495::TaskManager.Task::(_debug) Task 436f729b-2c91-43e7-9b0e-555656dd9d7d: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:28:47,260::task::495::TaskManager.Task::(_debug) Task 436f729b-2c91-43e7-9b0e-555656dd9d7d: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:28:47,260::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:28:47,260::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:28:47,261::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:28:47,261::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:28:47,261::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:28:47,262::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:28:47,262::task::495::TaskManager.Task::(_debug) Task 436f729b-2c91-43e7-9b0e-555656dd9d7d: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:28:47,263::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:28:47,454::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0551589 s, 18.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:28:49,464::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:28:49,997::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0838671 s, 12.2 MB/s\n'; <rc> = 0 >Thread-15912392::INFO::2013-11-20 03:28:51,752::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:28:51,753::task::495::TaskManager.Task::(_debug) Task ac0cede7-ef04-4cc4-a6e1-ac76f2644265: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:28:51,755::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`53f835cc-9c4a-4dbf-b455-72a5a5981ca7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:28:51,756::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:28:51,757::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:28:51,757::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`53f835cc-9c4a-4dbf-b455-72a5a5981ca7`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:28:51,758::task::495::TaskManager.Task::(_debug) Task ac0cede7-ef04-4cc4-a6e1-ac76f2644265: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:28:51,759::task::495::TaskManager.Task::(_debug) Task ac0cede7-ef04-4cc4-a6e1-ac76f2644265: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:28:51,760::task::495::TaskManager.Task::(_debug) Task ac0cede7-ef04-4cc4-a6e1-ac76f2644265: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:28:51,760::task::495::TaskManager.Task::(_debug) Task ac0cede7-ef04-4cc4-a6e1-ac76f2644265: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:28:51,761::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:28:51,762::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:28:51,762::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:28:51,763::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:28:51,763::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:28:51,764::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:28:51,765::task::495::TaskManager.Task::(_debug) Task ac0cede7-ef04-4cc4-a6e1-ac76f2644265: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:28:51,765::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:28:52,012::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:28:52,568::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0752094 s, 13.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:28:54,583::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:28:55,144::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0757156 s, 13.5 MB/s\n'; <rc> = 0 >Thread-15914308::DEBUG::2013-11-20 03:28:57,077::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914308::INFO::2013-11-20 03:28:57,078::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914308::DEBUG::2013-11-20 03:28:57,078::task::495::TaskManager.Task::(_debug) Task 4f8b3d2f-0a97-4897-9d7a-1d2b300fa468: moving from state init -> state preparing >Thread-15914308::DEBUG::2013-11-20 03:28:57,079::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914308::DEBUG::2013-11-20 03:28:57,079::task::495::TaskManager.Task::(_debug) Task 4f8b3d2f-0a97-4897-9d7a-1d2b300fa468: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914308::DEBUG::2013-11-20 03:28:57,080::task::495::TaskManager.Task::(_debug) Task 4f8b3d2f-0a97-4897-9d7a-1d2b300fa468: moving from state preparing -> state finished >Thread-15914308::DEBUG::2013-11-20 03:28:57,080::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914308::DEBUG::2013-11-20 03:28:57,081::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914308::DEBUG::2013-11-20 03:28:57,082::task::495::TaskManager.Task::(_debug) Task 4f8b3d2f-0a97-4897-9d7a-1d2b300fa468: ref 0 aborting False >Thread-15914308::INFO::2013-11-20 03:28:57,082::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914309::DEBUG::2013-11-20 03:28:57,095::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914309::INFO::2013-11-20 03:28:57,097::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914309::DEBUG::2013-11-20 03:28:57,097::task::495::TaskManager.Task::(_debug) Task 2c244ace-bd66-46b2-87ff-9a409a53d7b8: moving from state init -> state preparing >Thread-15914309::DEBUG::2013-11-20 03:28:57,099::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d8974292-3246-459a-b5b5-8c73704ecb6f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914309::DEBUG::2013-11-20 03:28:57,099::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914309::DEBUG::2013-11-20 03:28:57,100::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914309::DEBUG::2013-11-20 03:28:57,101::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d8974292-3246-459a-b5b5-8c73704ecb6f`::Granted request >Thread-15914309::DEBUG::2013-11-20 03:28:57,107::task::495::TaskManager.Task::(_debug) Task 2c244ace-bd66-46b2-87ff-9a409a53d7b8: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914309::DEBUG::2013-11-20 03:28:57,107::task::495::TaskManager.Task::(_debug) Task 2c244ace-bd66-46b2-87ff-9a409a53d7b8: ref 1 aborting False >Thread-15914309::WARNING::2013-11-20 03:28:57,109::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914309::DEBUG::2013-11-20 03:28:57,110::task::495::TaskManager.Task::(_debug) Task 2c244ace-bd66-46b2-87ff-9a409a53d7b8: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115214385152', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115214385152', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914309::DEBUG::2013-11-20 03:28:57,111::task::495::TaskManager.Task::(_debug) Task 2c244ace-bd66-46b2-87ff-9a409a53d7b8: moving from state preparing -> state finished >Thread-15914309::DEBUG::2013-11-20 03:28:57,112::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914309::DEBUG::2013-11-20 03:28:57,113::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914309::DEBUG::2013-11-20 03:28:57,119::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914309::DEBUG::2013-11-20 03:28:57,120::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914309::DEBUG::2013-11-20 03:28:57,120::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914309::DEBUG::2013-11-20 03:28:57,121::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914309::DEBUG::2013-11-20 03:28:57,121::task::495::TaskManager.Task::(_debug) Task 2c244ace-bd66-46b2-87ff-9a409a53d7b8: ref 0 aborting False >Thread-15914309::INFO::2013-11-20 03:28:57,122::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115214385152', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115214385152', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914310::INFO::2013-11-20 03:28:57,152::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914310::DEBUG::2013-11-20 03:28:57,153::task::495::TaskManager.Task::(_debug) Task 4f928763-d719-46a4-b86a-9d3e433205e1: moving from state init -> state preparing >Thread-15914310::DEBUG::2013-11-20 03:28:57,153::task::495::TaskManager.Task::(_debug) Task 4f928763-d719-46a4-b86a-9d3e433205e1: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000302791595459', 'lastCheck': 1384918133.767503, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150799751282', 'lastCheck': 1384918129.9062841, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00165009498596', 'lastCheck': 1384918132.2686639, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000535011291504', 'lastCheck': 1384918132.2409589, 'valid': True, 'code': 0}} >Thread-15914310::DEBUG::2013-11-20 03:28:57,157::task::495::TaskManager.Task::(_debug) Task 4f928763-d719-46a4-b86a-9d3e433205e1: moving from state preparing -> state finished >Thread-15914310::DEBUG::2013-11-20 03:28:57,161::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Dummy-15912651::DEBUG::2013-11-20 03:28:57,161::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914310::DEBUG::2013-11-20 03:28:57,161::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914310::DEBUG::2013-11-20 03:28:57,162::task::495::TaskManager.Task::(_debug) Task 4f928763-d719-46a4-b86a-9d3e433205e1: ref 0 aborting False >Thread-15914310::INFO::2013-11-20 03:28:57,345::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000302791595459', 'lastCheck': 1384918133.767503, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00150799751282', 'lastCheck': 1384918129.9062841, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00165009498596', 'lastCheck': 1384918132.2686639, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000535011291504', 'lastCheck': 1384918132.2409589, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:28:57,632::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0767575 s, 13.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:28:59,647::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:29:00,225::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.076334 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:29:02,240::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:29:02,825::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0842621 s, 12.2 MB/s\n'; <rc> = 0 >Thread-15913518::INFO::2013-11-20 03:29:03,078::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:29:03,079::task::495::TaskManager.Task::(_debug) Task d05ebdf6-c6a6-44fd-8431-9b87e87f77c8: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:29:03,081::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`6ebdc1ef-c234-4fcf-8086-9f07eae56d93`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:29:03,082::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:29:03,083::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:29:03,083::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`6ebdc1ef-c234-4fcf-8086-9f07eae56d93`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:29:03,085::task::495::TaskManager.Task::(_debug) Task d05ebdf6-c6a6-44fd-8431-9b87e87f77c8: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:29:03,085::task::495::TaskManager.Task::(_debug) Task d05ebdf6-c6a6-44fd-8431-9b87e87f77c8: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:29:03,086::task::495::TaskManager.Task::(_debug) Task d05ebdf6-c6a6-44fd-8431-9b87e87f77c8: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:29:03,087::task::495::TaskManager.Task::(_debug) Task d05ebdf6-c6a6-44fd-8431-9b87e87f77c8: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:29:03,088::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:29:03,088::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:29:03,089::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:29:03,090::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:29:03,091::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:29:03,091::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:29:03,092::task::495::TaskManager.Task::(_debug) Task d05ebdf6-c6a6-44fd-8431-9b87e87f77c8: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:29:03,093::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:29:03,094::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:29:03,095::task::495::TaskManager.Task::(_debug) Task 00f023f3-0236-4df3-978f-c1f714afc2d1: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:29:03,096::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`143c4c0f-9750-4a00-8678-6b77ce0a0a82`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:29:03,097::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:29:03,098::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:29:03,098::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`143c4c0f-9750-4a00-8678-6b77ce0a0a82`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:29:03,099::task::495::TaskManager.Task::(_debug) Task 00f023f3-0236-4df3-978f-c1f714afc2d1: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:29:03,100::task::495::TaskManager.Task::(_debug) Task 00f023f3-0236-4df3-978f-c1f714afc2d1: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:29:03,101::task::495::TaskManager.Task::(_debug) Task 00f023f3-0236-4df3-978f-c1f714afc2d1: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:29:03,101::task::495::TaskManager.Task::(_debug) Task 00f023f3-0236-4df3-978f-c1f714afc2d1: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:29:03,102::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:29:03,103::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:29:03,104::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:29:03,104::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:29:03,105::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:29:03,105::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:29:03,106::task::495::TaskManager.Task::(_debug) Task 00f023f3-0236-4df3-978f-c1f714afc2d1: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:29:03,107::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:29:04,841::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:29:05,386::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0804396 s, 12.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:29:07,400::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:29:07,939::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0754296 s, 13.6 MB/s\n'; <rc> = 0 >Thread-15914315::DEBUG::2013-11-20 03:29:08,075::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914315::INFO::2013-11-20 03:29:08,077::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914315::DEBUG::2013-11-20 03:29:08,077::task::495::TaskManager.Task::(_debug) Task 6989ad9f-fd18-4177-b5ea-b25b8fe6fd8f: moving from state init -> state preparing >Thread-15914315::DEBUG::2013-11-20 03:29:08,078::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914315::DEBUG::2013-11-20 03:29:08,079::task::495::TaskManager.Task::(_debug) Task 6989ad9f-fd18-4177-b5ea-b25b8fe6fd8f: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914315::DEBUG::2013-11-20 03:29:08,080::task::495::TaskManager.Task::(_debug) Task 6989ad9f-fd18-4177-b5ea-b25b8fe6fd8f: moving from state preparing -> state finished >Thread-15914315::DEBUG::2013-11-20 03:29:08,080::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914315::DEBUG::2013-11-20 03:29:08,081::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914315::DEBUG::2013-11-20 03:29:08,081::task::495::TaskManager.Task::(_debug) Task 6989ad9f-fd18-4177-b5ea-b25b8fe6fd8f: ref 0 aborting False >Thread-15914315::INFO::2013-11-20 03:29:08,082::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914316::DEBUG::2013-11-20 03:29:08,092::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914316::INFO::2013-11-20 03:29:08,093::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914316::DEBUG::2013-11-20 03:29:08,093::task::495::TaskManager.Task::(_debug) Task 30e7c5a0-701e-4546-8b2a-cf6dc5e07b79: moving from state init -> state preparing >Thread-15914316::DEBUG::2013-11-20 03:29:08,095::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ae6af5fc-6171-47bd-bfbf-86d5d10abe07`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914316::DEBUG::2013-11-20 03:29:08,095::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914316::DEBUG::2013-11-20 03:29:08,096::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914316::DEBUG::2013-11-20 03:29:08,097::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ae6af5fc-6171-47bd-bfbf-86d5d10abe07`::Granted request >Thread-15914316::DEBUG::2013-11-20 03:29:08,098::task::495::TaskManager.Task::(_debug) Task 30e7c5a0-701e-4546-8b2a-cf6dc5e07b79: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914316::DEBUG::2013-11-20 03:29:08,099::task::495::TaskManager.Task::(_debug) Task 30e7c5a0-701e-4546-8b2a-cf6dc5e07b79: ref 1 aborting False >Thread-15914316::WARNING::2013-11-20 03:29:08,101::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914316::DEBUG::2013-11-20 03:29:08,102::task::495::TaskManager.Task::(_debug) Task 30e7c5a0-701e-4546-8b2a-cf6dc5e07b79: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115214385152', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115214385152', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914316::DEBUG::2013-11-20 03:29:08,103::task::495::TaskManager.Task::(_debug) Task 30e7c5a0-701e-4546-8b2a-cf6dc5e07b79: moving from state preparing -> state finished >Thread-15914316::DEBUG::2013-11-20 03:29:08,104::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914316::DEBUG::2013-11-20 03:29:08,104::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914316::DEBUG::2013-11-20 03:29:08,105::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914316::DEBUG::2013-11-20 03:29:08,106::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914316::DEBUG::2013-11-20 03:29:08,106::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914316::DEBUG::2013-11-20 03:29:08,107::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914316::DEBUG::2013-11-20 03:29:08,108::task::495::TaskManager.Task::(_debug) Task 30e7c5a0-701e-4546-8b2a-cf6dc5e07b79: ref 0 aborting False >Thread-15914316::INFO::2013-11-20 03:29:08,108::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115214385152', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115214385152', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:29:09,954::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:29:10,490::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0887183 s, 11.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:29:12,506::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:29:13,085::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0863412 s, 11.9 MB/s\n'; <rc> = 0 >Thread-15914318::INFO::2013-11-20 03:29:13,200::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914318::DEBUG::2013-11-20 03:29:13,201::task::495::TaskManager.Task::(_debug) Task 807f4938-c7f8-44c3-9ce4-b58f2d861d93: moving from state init -> state preparing >Thread-15914318::DEBUG::2013-11-20 03:29:13,202::task::495::TaskManager.Task::(_debug) Task 807f4938-c7f8-44c3-9ce4-b58f2d861d93: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000249862670898', 'lastCheck': 1384918143.8883941, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0373349189758', 'lastCheck': 1384918150.1951139, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00139307975769', 'lastCheck': 1384918152.735827, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00135612487793', 'lastCheck': 1384918152.7317009, 'valid': True, 'code': 0}} >Thread-15914318::DEBUG::2013-11-20 03:29:13,203::task::495::TaskManager.Task::(_debug) Task 807f4938-c7f8-44c3-9ce4-b58f2d861d93: moving from state preparing -> state finished >Thread-15914318::DEBUG::2013-11-20 03:29:13,203::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914318::DEBUG::2013-11-20 03:29:13,204::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914318::DEBUG::2013-11-20 03:29:13,204::task::495::TaskManager.Task::(_debug) Task 807f4938-c7f8-44c3-9ce4-b58f2d861d93: ref 0 aborting False >Thread-15914318::INFO::2013-11-20 03:29:13,205::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000249862670898', 'lastCheck': 1384918143.8883941, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0373349189758', 'lastCheck': 1384918150.1951139, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00139307975769', 'lastCheck': 1384918152.735827, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00135612487793', 'lastCheck': 1384918152.7317009, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:29:15,098::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:29:15,673::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0734134 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:29:17,688::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913546::INFO::2013-11-20 03:29:17,909::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:29:17,921::task::495::TaskManager.Task::(_debug) Task 0d1e3cb2-4ca1-48d3-87ab-b252de93e553: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:29:17,944::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`179cfacf-6192-448c-92b4-a45ddc510d2c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:29:17,949::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:29:17,954::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:29:17,955::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`179cfacf-6192-448c-92b4-a45ddc510d2c`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:29:17,957::task::495::TaskManager.Task::(_debug) Task 0d1e3cb2-4ca1-48d3-87ab-b252de93e553: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:29:17,958::task::495::TaskManager.Task::(_debug) Task 0d1e3cb2-4ca1-48d3-87ab-b252de93e553: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:29:17,959::task::495::TaskManager.Task::(_debug) Task 0d1e3cb2-4ca1-48d3-87ab-b252de93e553: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:29:17,959::task::495::TaskManager.Task::(_debug) Task 0d1e3cb2-4ca1-48d3-87ab-b252de93e553: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:29:17,960::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:29:17,960::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:29:17,961::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:29:17,962::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:29:17,962::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:29:17,963::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:29:17,964::task::495::TaskManager.Task::(_debug) Task 0d1e3cb2-4ca1-48d3-87ab-b252de93e553: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:29:17,965::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:29:17,965::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:29:17,966::task::495::TaskManager.Task::(_debug) Task c4669175-95e7-42cf-9499-3dcb0b2bcdf1: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:29:17,967::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`a6f9970f-694e-4848-b0a4-bc0e45820642`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:29:17,968::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:29:17,968::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:29:17,969::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`a6f9970f-694e-4848-b0a4-bc0e45820642`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:29:17,970::task::495::TaskManager.Task::(_debug) Task c4669175-95e7-42cf-9499-3dcb0b2bcdf1: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:29:17,970::task::495::TaskManager.Task::(_debug) Task c4669175-95e7-42cf-9499-3dcb0b2bcdf1: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:29:17,971::task::495::TaskManager.Task::(_debug) Task c4669175-95e7-42cf-9499-3dcb0b2bcdf1: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:29:17,971::task::495::TaskManager.Task::(_debug) Task c4669175-95e7-42cf-9499-3dcb0b2bcdf1: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:29:17,972::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:29:17,972::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:29:17,973::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:29:17,974::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:29:17,974::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:29:17,975::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:29:17,975::task::495::TaskManager.Task::(_debug) Task c4669175-95e7-42cf-9499-3dcb0b2bcdf1: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:29:17,976::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:29:17,977::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:29:17,977::task::495::TaskManager.Task::(_debug) Task 8d61623d-f2c3-478a-a92d-bb394a0206f0: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:29:17,978::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fe048a2f-7777-48e1-8b01-0f7566cf7146`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:29:17,979::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:29:17,979::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:29:17,980::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fe048a2f-7777-48e1-8b01-0f7566cf7146`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:29:17,981::task::495::TaskManager.Task::(_debug) Task 8d61623d-f2c3-478a-a92d-bb394a0206f0: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:29:17,981::task::495::TaskManager.Task::(_debug) Task 8d61623d-f2c3-478a-a92d-bb394a0206f0: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:29:17,982::task::495::TaskManager.Task::(_debug) Task 8d61623d-f2c3-478a-a92d-bb394a0206f0: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:29:17,982::task::495::TaskManager.Task::(_debug) Task 8d61623d-f2c3-478a-a92d-bb394a0206f0: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:29:17,983::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:29:17,984::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:29:17,984::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:29:17,985::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:29:17,985::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:29:17,986::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:29:17,986::task::495::TaskManager.Task::(_debug) Task 8d61623d-f2c3-478a-a92d-bb394a0206f0: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:29:17,987::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:29:17,988::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:29:17,988::task::495::TaskManager.Task::(_debug) Task 58674057-362b-4ef6-961a-bc396892fb6e: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:29:17,989::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`dcfedc2a-ef31-47f3-a3a6-a6e5254113e9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:29:17,990::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:29:17,990::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:29:17,991::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`dcfedc2a-ef31-47f3-a3a6-a6e5254113e9`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:29:17,992::task::495::TaskManager.Task::(_debug) Task 58674057-362b-4ef6-961a-bc396892fb6e: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:29:17,992::task::495::TaskManager.Task::(_debug) Task 58674057-362b-4ef6-961a-bc396892fb6e: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:29:17,993::task::495::TaskManager.Task::(_debug) Task 58674057-362b-4ef6-961a-bc396892fb6e: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:29:17,993::task::495::TaskManager.Task::(_debug) Task 58674057-362b-4ef6-961a-bc396892fb6e: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:29:17,994::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:29:17,994::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:29:17,995::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:29:17,996::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:29:17,996::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:29:17,997::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:29:17,997::task::495::TaskManager.Task::(_debug) Task 58674057-362b-4ef6-961a-bc396892fb6e: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:29:17,998::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:29:18,191::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0862049 s, 11.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:29:20,206::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:29:20,690::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0716282 s, 14.3 MB/s\n'; <rc> = 0 >Thread-15914322::DEBUG::2013-11-20 03:29:21,201::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914322::INFO::2013-11-20 03:29:21,202::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914322::DEBUG::2013-11-20 03:29:21,203::task::495::TaskManager.Task::(_debug) Task 2aee0790-8f50-4b48-838f-2b5026c05f51: moving from state init -> state preparing >Thread-15914322::DEBUG::2013-11-20 03:29:21,204::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914322::DEBUG::2013-11-20 03:29:21,204::task::495::TaskManager.Task::(_debug) Task 2aee0790-8f50-4b48-838f-2b5026c05f51: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914322::DEBUG::2013-11-20 03:29:21,205::task::495::TaskManager.Task::(_debug) Task 2aee0790-8f50-4b48-838f-2b5026c05f51: moving from state preparing -> state finished >Thread-15914322::DEBUG::2013-11-20 03:29:21,205::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914322::DEBUG::2013-11-20 03:29:21,206::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914322::DEBUG::2013-11-20 03:29:21,206::task::495::TaskManager.Task::(_debug) Task 2aee0790-8f50-4b48-838f-2b5026c05f51: ref 0 aborting False >Thread-15914322::INFO::2013-11-20 03:29:21,207::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914323::DEBUG::2013-11-20 03:29:21,211::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914323::INFO::2013-11-20 03:29:21,212::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914323::DEBUG::2013-11-20 03:29:21,212::task::495::TaskManager.Task::(_debug) Task 21136643-7a33-4eea-b99d-d2c0f60e799e: moving from state init -> state preparing >Thread-15914323::DEBUG::2013-11-20 03:29:21,213::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f4f3ca66-43de-49d9-8f4a-de9e0cbfcaf5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914323::DEBUG::2013-11-20 03:29:21,213::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914323::DEBUG::2013-11-20 03:29:21,214::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914323::DEBUG::2013-11-20 03:29:21,215::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f4f3ca66-43de-49d9-8f4a-de9e0cbfcaf5`::Granted request >Thread-15914323::DEBUG::2013-11-20 03:29:21,219::task::495::TaskManager.Task::(_debug) Task 21136643-7a33-4eea-b99d-d2c0f60e799e: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914323::DEBUG::2013-11-20 03:29:21,220::task::495::TaskManager.Task::(_debug) Task 21136643-7a33-4eea-b99d-d2c0f60e799e: ref 1 aborting False >Thread-15914323::WARNING::2013-11-20 03:29:21,221::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914323::DEBUG::2013-11-20 03:29:21,222::task::495::TaskManager.Task::(_debug) Task 21136643-7a33-4eea-b99d-d2c0f60e799e: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914323::DEBUG::2013-11-20 03:29:21,223::task::495::TaskManager.Task::(_debug) Task 21136643-7a33-4eea-b99d-d2c0f60e799e: moving from state preparing -> state finished >Thread-15914323::DEBUG::2013-11-20 03:29:21,223::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914323::DEBUG::2013-11-20 03:29:21,224::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914323::DEBUG::2013-11-20 03:29:21,224::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914323::DEBUG::2013-11-20 03:29:21,225::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914323::DEBUG::2013-11-20 03:29:21,225::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914323::DEBUG::2013-11-20 03:29:21,225::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914323::DEBUG::2013-11-20 03:29:21,226::task::495::TaskManager.Task::(_debug) Task 21136643-7a33-4eea-b99d-d2c0f60e799e: ref 0 aborting False >Thread-15914323::INFO::2013-11-20 03:29:21,226::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:29:22,706::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-42840::DEBUG::2013-11-20 03:29:22,905::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:29:23,314::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0682476 s, 15.0 MB/s\n'; <rc> = 0 >Thread-42840::DEBUG::2013-11-20 03:29:23,775::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:29:25,324::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:29:25,845::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0759301 s, 13.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:29:27,860::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913639::INFO::2013-11-20 03:29:28,226::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:29:28,244::task::495::TaskManager.Task::(_debug) Task 40925be5-ab53-4a27-aa66-f9e9e110ad10: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:29:28,248::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b37c09b2-563a-4e63-9958-c8de8fa34f87`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:29:28,249::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:29:28,250::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:29:28,251::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b37c09b2-563a-4e63-9958-c8de8fa34f87`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:29:28,252::task::495::TaskManager.Task::(_debug) Task 40925be5-ab53-4a27-aa66-f9e9e110ad10: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:29:28,253::task::495::TaskManager.Task::(_debug) Task 40925be5-ab53-4a27-aa66-f9e9e110ad10: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:29:28,254::task::495::TaskManager.Task::(_debug) Task 40925be5-ab53-4a27-aa66-f9e9e110ad10: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:29:28,254::task::495::TaskManager.Task::(_debug) Task 40925be5-ab53-4a27-aa66-f9e9e110ad10: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:29:28,255::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:29:28,256::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:29:28,256::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:29:28,257::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:29:28,257::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:29:28,258::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:29:28,259::task::495::TaskManager.Task::(_debug) Task 40925be5-ab53-4a27-aa66-f9e9e110ad10: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:29:28,259::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:29:28,428::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0765887 s, 13.4 MB/s\n'; <rc> = 0 >Thread-15914326::INFO::2013-11-20 03:29:28,615::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914326::DEBUG::2013-11-20 03:29:28,616::task::495::TaskManager.Task::(_debug) Task 19cc637e-f023-43da-a799-b2499934456d: moving from state init -> state preparing >Thread-15914326::DEBUG::2013-11-20 03:29:28,616::task::495::TaskManager.Task::(_debug) Task 19cc637e-f023-43da-a799-b2499934456d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000354051589966', 'lastCheck': 1384918164.4194181, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00136303901672', 'lastCheck': 1384918160.420423, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00369501113892', 'lastCheck': 1384918163.219424, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000316143035889', 'lastCheck': 1384918163.7770109, 'valid': True, 'code': 0}} >Thread-15914326::DEBUG::2013-11-20 03:29:28,627::task::495::TaskManager.Task::(_debug) Task 19cc637e-f023-43da-a799-b2499934456d: moving from state preparing -> state finished >Thread-15914326::DEBUG::2013-11-20 03:29:28,628::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914326::DEBUG::2013-11-20 03:29:28,628::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914326::DEBUG::2013-11-20 03:29:28,629::task::495::TaskManager.Task::(_debug) Task 19cc637e-f023-43da-a799-b2499934456d: ref 0 aborting False >Thread-15914326::INFO::2013-11-20 03:29:28,630::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000354051589966', 'lastCheck': 1384918164.4194181, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00136303901672', 'lastCheck': 1384918160.420423, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00369501113892', 'lastCheck': 1384918163.219424, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000316143035889', 'lastCheck': 1384918163.7770109, 'valid': True, 'code': 0}} >Thread-15914328::DEBUG::2013-11-20 03:29:30,405::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914328::INFO::2013-11-20 03:29:30,406::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: updateVM, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 vmList=[{'imglist': '5af747d8-95b1-4b7e-af53-cb77e4fcc896', 'ovf': u'<?xml version=\'1.0\' encoding=\'UTF-8\'?><ovf:Envelope xmlns:ovf="http://schemas.dmtf.org/ovf/envelope/1/" xmlns:rasd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationSettingData" xmlns:vssd="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettingData" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ovf:version="3.2.0.0"><References><File ovf:href="5af747d8-95b1-4b7e-af53-cb77e4fcc896/8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f" ovf:id="8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f" ovf:size="85899345920" ovf:description="Active VM snapshot"/><Nic ovf:id="fd38b98a-8c16-4dae-817c-794715ec6b17"/></References><Section xsi:type="ovf:NetworkSection_Type"><Info>List of networks</Info><Network ovf:name="Network 1"/></Section><Section xsi:type="ovf:DiskSection_Type"><Info>List of Virtual Disks</Info><Disk ovf:diskId="8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f" ovf:size="80" ovf:actual_size="80" ovf:vm_snapshot_id="a8605287-4e5e-4bcc-965b-707787b502ee" ovf:parentRef="" ovf:fileRef="5af747d8-95b1-4b7e-af53-cb77e4fcc896/8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f" ovf:format="http://www.vmware.com/specifications/vmdk.html#sparse" ovf:volume-format="RAW" ovf:volume-type="Preallocated" ovf:disk-interface="IDE" ovf:boot="true" ovf:disk-alias="sukhumi_Disk0" ovf:wipe-after-delete="false"/></Section><Content ovf:id="out" xsi:type="ovf:VirtualSystem_Type"><Description>Windows 2008 - App server</Description><Domain>brd-klee.dk</Domain><CreationDate>2013/01/15 01:20:27</CreationDate><ExportDate>2013/11/20 03:29:29</ExportDate><IsAutoSuspend>false</IsAutoSuspend><DeleteProtected>false</DeleteProtected><IsSmartcardEnabled>false</IsSmartcardEnabled><TimeZone></TimeZone><default_boot_sequence>0</default_boot_sequence><Generation>2</Generation><VmType>1</VmType><MinAllocatedMem>4500</MinAllocatedMem><Name>sukhumi</Name><TemplateId>00000000-0000-0000-0000-000000000000</TemplateId><TemplateName>Blank</TemplateName><IsInitilized>true</IsInitilized><IsStateless>false</IsStateless><Origin>0</Origin><app_list>Foxit Reader,Experian DK X-Point 6.6.26,Microsoft SQL Server 2005,TCSconnexion_3607,Zabbix Agent 2.0.8,PM,RHEV-Tools 3.0.39,RHEV-Network64 3.0.7,Java(TM) 6 Update 30,Microsoft SQL Server 2005 Tools Express Edition,Microsoft SQL Server 2005 Express Edition (SQLEXPRESS),RHEV-Block64 3.0.9,RHEV-Spice-Agent64 3.0.5,Java Auto Updater,Microsoft SQL Server Setup Support Files (English),Acronis Backup & Recovery 11.5 Agent Core,Acronis Backup & Recovery 11.5\xa0Agent for Windows,RHEV-Serial64 3.0.6,RHEV-Agent64 3.0.10,Consignor Server 9.1.0.6,Update for Microsoft .NET Framework 3.5 SP1 (KB963707)</app_list><quota_id>00000000-0000-0000-0000-000000000000</quota_id><DefaultDisplayType>1</DefaultDisplayType><Section ovf:id="e97d1c52-b688-4192-bdeb-2dfadcc79786" ovf:required="false" xsi:type="ovf:OperatingSystemSection_Type"><Info>Guest Operating System</Info><Description>Windows2008x64</Description></Section><Section xsi:type="ovf:VirtualHardwareSection_Type"><Info>4 CPU, 4500 Memeory</Info><System><vssd:VirtualSystemType>RHEVM 3.2.0.0</vssd:VirtualSystemType></System><Item><rasd:Caption>4 virtual cpu</rasd:Caption><rasd:Description>Number of virtual CPU</rasd:Description><rasd:InstanceId>1</rasd:InstanceId><rasd:ResourceType>3</rasd:ResourceType><rasd:num_of_sockets>4</rasd:num_of_sockets><rasd:cpu_per_socket>1</rasd:cpu_per_socket></Item><Item><rasd:Caption>4500 MB of memory</rasd:Caption><rasd:Description>Memory Size</rasd:Description><rasd:InstanceId>2</rasd:InstanceId><rasd:ResourceType>4</rasd:ResourceType><rasd:AllocationUnits>MegaBytes</rasd:AllocationUnits><rasd:VirtualQuantity>4500</rasd:VirtualQuantity></Item><Item><rasd:Caption>Drive 1</rasd:Caption><rasd:InstanceId>8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f</rasd:InstanceId><rasd:ResourceType>17</rasd:ResourceType><rasd:HostResource>5af747d8-95b1-4b7e-af53-cb77e4fcc896/8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f</rasd:HostResource><rasd:Parent>00000000-0000-0000-0000-000000000000</rasd:Parent><rasd:Template>00000000-0000-0000-0000-000000000000</rasd:Template><rasd:ApplicationList></rasd:ApplicationList><rasd:StorageId>6b892067-ac7e-4039-a7be-5b7387525b16</rasd:StorageId><rasd:StoragePoolId>c4e5889a-348d-11e2-83fe-9f72369eab30</rasd:StoragePoolId><rasd:CreationDate>2013/01/15 02:06:28</rasd:CreationDate><rasd:LastModified>2013/01/15 02:06:28</rasd:LastModified><rasd:last_modified_date>2013/11/20 03:29:28</rasd:last_modified_date></Item><Item><rasd:Caption>Ethernet adapter on rhevm</rasd:Caption><rasd:InstanceId>fd38b98a-8c16-4dae-817c-794715ec6b17</rasd:InstanceId><rasd:ResourceType>10</rasd:ResourceType><rasd:ResourceSubType>3</rasd:ResourceSubType><rasd:Connection>rhevm</rasd:Connection><rasd:Linked>true</rasd:Linked><rasd:Name>nic1</rasd:Name><rasd:MACAddress>00:1a:4a:a8:64:13</rasd:MACAddress><rasd:speed>1000</rasd:speed></Item><Item><rasd:Caption>USB Controller</rasd:Caption><rasd:InstanceId>3</rasd:InstanceId><rasd:ResourceType>23</rasd:ResourceType><rasd:UsbPolicy>Disabled</rasd:UsbPolicy></Item></Section></Content></ovf:Envelope>\n', 'vm': 'e97d1c52-b688-4192-bdeb-2dfadcc79786'}]) >Thread-15914328::DEBUG::2013-11-20 03:29:30,407::task::495::TaskManager.Task::(_debug) Task 2ca55320-b992-49d0-a570-ded1ea4ae0d5: moving from state init -> state preparing >Thread-15914328::DEBUG::2013-11-20 03:29:30,408::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ce1bf18b-0578-427b-a79a-2623d910577a`::Request was made in '/usr/share/vdsm/storage/spm.py' line '1240' at 'public_updateVM' >Thread-15914328::DEBUG::2013-11-20 03:29:30,409::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914328::DEBUG::2013-11-20 03:29:30,410::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914328::DEBUG::2013-11-20 03:29:30,410::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ce1bf18b-0578-427b-a79a-2623d910577a`::Granted request >Thread-15914328::DEBUG::2013-11-20 03:29:30,411::task::495::TaskManager.Task::(_debug) Task 2ca55320-b992-49d0-a570-ded1ea4ae0d5: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914328::DEBUG::2013-11-20 03:29:30,412::task::495::TaskManager.Task::(_debug) Task 2ca55320-b992-49d0-a570-ded1ea4ae0d5: ref 1 aborting False >Thread-15914328::INFO::2013-11-20 03:29:30,412::sp::927::Storage.StoragePool::(updateVM) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 >Thread-15914328::INFO::2013-11-20 03:29:30,413::sp::938::Storage.StoragePool::(updateVM) vmUUID=e97d1c52-b688-4192-bdeb-2dfadcc79786 imgList=['5af747d8-95b1-4b7e-af53-cb77e4fcc896'] >Dummy-15912651::DEBUG::2013-11-20 03:29:30,443::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914328::DEBUG::2013-11-20 03:29:30,665::task::495::TaskManager.Task::(_debug) Task 2ca55320-b992-49d0-a570-ded1ea4ae0d5: finished: None >Thread-15914328::DEBUG::2013-11-20 03:29:30,667::task::495::TaskManager.Task::(_debug) Task 2ca55320-b992-49d0-a570-ded1ea4ae0d5: moving from state preparing -> state finished >Thread-15914328::DEBUG::2013-11-20 03:29:30,668::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914328::DEBUG::2013-11-20 03:29:30,669::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914328::DEBUG::2013-11-20 03:29:30,670::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914328::DEBUG::2013-11-20 03:29:30,671::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914328::DEBUG::2013-11-20 03:29:30,672::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914328::DEBUG::2013-11-20 03:29:30,673::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914328::DEBUG::2013-11-20 03:29:30,674::task::495::TaskManager.Task::(_debug) Task 2ca55320-b992-49d0-a570-ded1ea4ae0d5: ref 0 aborting False >Thread-15914328::INFO::2013-11-20 03:29:30,674::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: updateVM, Return response: {'status': {'message': 'OK', 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:29:31,007::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0827341 s, 12.4 MB/s\n'; <rc> = 0 >Thread-15914329::DEBUG::2013-11-20 03:29:31,383::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914329::INFO::2013-11-20 03:29:31,384::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914329::DEBUG::2013-11-20 03:29:31,385::task::495::TaskManager.Task::(_debug) Task ecb7f167-2acd-4cd2-bfd3-27f59afe7093: moving from state init -> state preparing >Thread-15914329::DEBUG::2013-11-20 03:29:31,386::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914329::DEBUG::2013-11-20 03:29:31,386::task::495::TaskManager.Task::(_debug) Task ecb7f167-2acd-4cd2-bfd3-27f59afe7093: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914329::DEBUG::2013-11-20 03:29:31,387::task::495::TaskManager.Task::(_debug) Task ecb7f167-2acd-4cd2-bfd3-27f59afe7093: moving from state preparing -> state finished >Thread-15914329::DEBUG::2013-11-20 03:29:31,388::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914329::DEBUG::2013-11-20 03:29:31,388::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914329::DEBUG::2013-11-20 03:29:31,389::task::495::TaskManager.Task::(_debug) Task ecb7f167-2acd-4cd2-bfd3-27f59afe7093: ref 0 aborting False >Thread-15914329::INFO::2013-11-20 03:29:31,390::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914330::DEBUG::2013-11-20 03:29:31,396::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914330::INFO::2013-11-20 03:29:31,396::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914330::DEBUG::2013-11-20 03:29:31,397::task::495::TaskManager.Task::(_debug) Task ad0b0f8b-f154-4962-89cf-ef6ff3e7e67d: moving from state init -> state preparing >Thread-15914330::DEBUG::2013-11-20 03:29:31,398::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5a191715-3d0c-4425-98e4-d0ae675771e9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914330::DEBUG::2013-11-20 03:29:31,399::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914330::DEBUG::2013-11-20 03:29:31,400::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914330::DEBUG::2013-11-20 03:29:31,400::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5a191715-3d0c-4425-98e4-d0ae675771e9`::Granted request >Thread-15914330::DEBUG::2013-11-20 03:29:31,401::task::495::TaskManager.Task::(_debug) Task ad0b0f8b-f154-4962-89cf-ef6ff3e7e67d: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914330::DEBUG::2013-11-20 03:29:31,402::task::495::TaskManager.Task::(_debug) Task ad0b0f8b-f154-4962-89cf-ef6ff3e7e67d: ref 1 aborting False >Thread-15914330::WARNING::2013-11-20 03:29:31,404::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914330::DEBUG::2013-11-20 03:29:31,405::task::495::TaskManager.Task::(_debug) Task ad0b0f8b-f154-4962-89cf-ef6ff3e7e67d: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914330::DEBUG::2013-11-20 03:29:31,406::task::495::TaskManager.Task::(_debug) Task ad0b0f8b-f154-4962-89cf-ef6ff3e7e67d: moving from state preparing -> state finished >Thread-15914330::DEBUG::2013-11-20 03:29:31,407::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914330::DEBUG::2013-11-20 03:29:31,408::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914330::DEBUG::2013-11-20 03:29:31,408::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914330::DEBUG::2013-11-20 03:29:31,409::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914330::DEBUG::2013-11-20 03:29:31,410::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914330::DEBUG::2013-11-20 03:29:31,410::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914330::DEBUG::2013-11-20 03:29:31,411::task::495::TaskManager.Task::(_debug) Task ad0b0f8b-f154-4962-89cf-ef6ff3e7e67d: ref 0 aborting False >Thread-15914330::INFO::2013-11-20 03:29:31,412::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:29:33,021::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:29:33,557::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0774062 s, 13.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:29:35,574::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:29:36,130::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0796798 s, 12.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:29:38,146::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:29:38,721::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0771647 s, 13.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:29:40,731::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:29:41,315::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.085683 s, 12.0 MB/s\n'; <rc> = 0 >Thread-15914335::DEBUG::2013-11-20 03:29:42,095::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914335::INFO::2013-11-20 03:29:42,097::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914335::DEBUG::2013-11-20 03:29:42,097::task::495::TaskManager.Task::(_debug) Task 69065b85-1014-4e47-826e-b7a7cac4d12e: moving from state init -> state preparing >Thread-15914335::DEBUG::2013-11-20 03:29:42,098::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914335::DEBUG::2013-11-20 03:29:42,099::task::495::TaskManager.Task::(_debug) Task 69065b85-1014-4e47-826e-b7a7cac4d12e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914335::DEBUG::2013-11-20 03:29:42,100::task::495::TaskManager.Task::(_debug) Task 69065b85-1014-4e47-826e-b7a7cac4d12e: moving from state preparing -> state finished >Thread-15914335::DEBUG::2013-11-20 03:29:42,100::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914335::DEBUG::2013-11-20 03:29:42,101::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914335::DEBUG::2013-11-20 03:29:42,102::task::495::TaskManager.Task::(_debug) Task 69065b85-1014-4e47-826e-b7a7cac4d12e: ref 0 aborting False >Thread-15914335::INFO::2013-11-20 03:29:42,103::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914336::DEBUG::2013-11-20 03:29:42,108::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914336::INFO::2013-11-20 03:29:42,109::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914336::DEBUG::2013-11-20 03:29:42,110::task::495::TaskManager.Task::(_debug) Task ab980c16-6e03-45b4-b71b-79764ce0476a: moving from state init -> state preparing >Thread-15914336::DEBUG::2013-11-20 03:29:42,111::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5f06e5f9-d067-4bd0-b447-840a6c28f147`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914336::DEBUG::2013-11-20 03:29:42,112::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914336::DEBUG::2013-11-20 03:29:42,113::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914336::DEBUG::2013-11-20 03:29:42,113::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5f06e5f9-d067-4bd0-b447-840a6c28f147`::Granted request >Thread-15914336::DEBUG::2013-11-20 03:29:42,114::task::495::TaskManager.Task::(_debug) Task ab980c16-6e03-45b4-b71b-79764ce0476a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914336::DEBUG::2013-11-20 03:29:42,115::task::495::TaskManager.Task::(_debug) Task ab980c16-6e03-45b4-b71b-79764ce0476a: ref 1 aborting False >Thread-15914336::WARNING::2013-11-20 03:29:42,117::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914336::DEBUG::2013-11-20 03:29:42,125::task::495::TaskManager.Task::(_debug) Task ab980c16-6e03-45b4-b71b-79764ce0476a: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914336::DEBUG::2013-11-20 03:29:42,126::task::495::TaskManager.Task::(_debug) Task ab980c16-6e03-45b4-b71b-79764ce0476a: moving from state preparing -> state finished >Thread-15914336::DEBUG::2013-11-20 03:29:42,127::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914336::DEBUG::2013-11-20 03:29:42,128::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914336::DEBUG::2013-11-20 03:29:42,129::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914336::DEBUG::2013-11-20 03:29:42,129::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914336::DEBUG::2013-11-20 03:29:42,130::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914336::DEBUG::2013-11-20 03:29:42,131::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914336::DEBUG::2013-11-20 03:29:42,132::task::495::TaskManager.Task::(_debug) Task ab980c16-6e03-45b4-b71b-79764ce0476a: ref 0 aborting False >Thread-15914336::INFO::2013-11-20 03:29:42,132::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:29:43,331::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:29:43,871::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0791102 s, 12.9 MB/s\n'; <rc> = 0 >Thread-15914337::INFO::2013-11-20 03:29:44,582::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914337::DEBUG::2013-11-20 03:29:44,583::task::495::TaskManager.Task::(_debug) Task 5090d5cd-5c52-4ea4-8376-acb43f1f5aba: moving from state init -> state preparing >Thread-15914337::DEBUG::2013-11-20 03:29:44,584::task::495::TaskManager.Task::(_debug) Task 5090d5cd-5c52-4ea4-8376-acb43f1f5aba: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000332117080688', 'lastCheck': 1384918174.6563959, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00163912773132', 'lastCheck': 1384918180.9701509, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00172901153564', 'lastCheck': 1384918183.5476329, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000399112701416', 'lastCheck': 1384918183.9586871, 'valid': True, 'code': 0}} >Thread-15914337::DEBUG::2013-11-20 03:29:44,585::task::495::TaskManager.Task::(_debug) Task 5090d5cd-5c52-4ea4-8376-acb43f1f5aba: moving from state preparing -> state finished >Thread-15914337::DEBUG::2013-11-20 03:29:44,585::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914337::DEBUG::2013-11-20 03:29:44,586::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914337::DEBUG::2013-11-20 03:29:44,587::task::495::TaskManager.Task::(_debug) Task 5090d5cd-5c52-4ea4-8376-acb43f1f5aba: ref 0 aborting False >Thread-15914337::INFO::2013-11-20 03:29:44,587::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000332117080688', 'lastCheck': 1384918174.6563959, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00163912773132', 'lastCheck': 1384918180.9701509, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00172901153564', 'lastCheck': 1384918183.5476329, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000399112701416', 'lastCheck': 1384918183.9586871, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:29:45,889::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913695::INFO::2013-11-20 03:29:46,126::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:29:46,128::task::495::TaskManager.Task::(_debug) Task 714bec9a-5a98-436b-9271-95f52da843bd: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:29:46,131::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d1fb841c-7243-400c-9abb-84da38a0812b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:29:46,131::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:29:46,133::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:29:46,134::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d1fb841c-7243-400c-9abb-84da38a0812b`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:29:46,135::task::495::TaskManager.Task::(_debug) Task 714bec9a-5a98-436b-9271-95f52da843bd: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:29:46,136::task::495::TaskManager.Task::(_debug) Task 714bec9a-5a98-436b-9271-95f52da843bd: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:29:46,138::task::495::TaskManager.Task::(_debug) Task 714bec9a-5a98-436b-9271-95f52da843bd: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:29:46,139::task::495::TaskManager.Task::(_debug) Task 714bec9a-5a98-436b-9271-95f52da843bd: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:29:46,139::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:29:46,140::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:29:46,141::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:29:46,141::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:29:46,142::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:29:46,142::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:29:46,143::task::495::TaskManager.Task::(_debug) Task 714bec9a-5a98-436b-9271-95f52da843bd: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:29:46,144::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:29:46,438::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0797128 s, 12.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:29:48,454::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913658::INFO::2013-11-20 03:29:48,651::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:29:48,661::task::495::TaskManager.Task::(_debug) Task 3ade764b-5c45-4ab6-ab09-b9045fea3dd4: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:29:48,666::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c1ddcb24-7d75-4399-bb58-dd73079f5414`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:29:48,668::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:29:48,670::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:29:48,671::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c1ddcb24-7d75-4399-bb58-dd73079f5414`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:29:48,673::task::495::TaskManager.Task::(_debug) Task 3ade764b-5c45-4ab6-ab09-b9045fea3dd4: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:29:48,673::task::495::TaskManager.Task::(_debug) Task 3ade764b-5c45-4ab6-ab09-b9045fea3dd4: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:29:48,674::task::495::TaskManager.Task::(_debug) Task 3ade764b-5c45-4ab6-ab09-b9045fea3dd4: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:29:48,675::task::495::TaskManager.Task::(_debug) Task 3ade764b-5c45-4ab6-ab09-b9045fea3dd4: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:29:48,676::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:29:48,676::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:29:48,677::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:29:48,677::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:29:48,678::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:29:48,678::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:29:48,679::task::495::TaskManager.Task::(_debug) Task 3ade764b-5c45-4ab6-ab09-b9045fea3dd4: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:29:48,680::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:29:48,986::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0833071 s, 12.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:29:51,000::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:29:51,555::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0786856 s, 13.0 MB/s\n'; <rc> = 0 >Thread-15912392::INFO::2013-11-20 03:29:52,118::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:29:52,119::task::495::TaskManager.Task::(_debug) Task 15d70b9c-ea64-4ca7-ae3d-d5c89c1ce0d1: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:29:52,121::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c3925e77-3117-4202-b362-1989915ab5d6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:29:52,121::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:29:52,122::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:29:52,123::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c3925e77-3117-4202-b362-1989915ab5d6`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:29:52,124::task::495::TaskManager.Task::(_debug) Task 15d70b9c-ea64-4ca7-ae3d-d5c89c1ce0d1: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:29:52,124::task::495::TaskManager.Task::(_debug) Task 15d70b9c-ea64-4ca7-ae3d-d5c89c1ce0d1: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:29:52,125::task::495::TaskManager.Task::(_debug) Task 15d70b9c-ea64-4ca7-ae3d-d5c89c1ce0d1: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:29:52,126::task::495::TaskManager.Task::(_debug) Task 15d70b9c-ea64-4ca7-ae3d-d5c89c1ce0d1: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:29:52,126::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:29:52,127::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:29:52,128::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:29:52,128::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:29:52,129::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:29:52,129::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:29:52,130::task::495::TaskManager.Task::(_debug) Task 15d70b9c-ea64-4ca7-ae3d-d5c89c1ce0d1: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:29:52,131::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15914341::DEBUG::2013-11-20 03:29:52,285::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914341::INFO::2013-11-20 03:29:52,286::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914341::DEBUG::2013-11-20 03:29:52,287::task::495::TaskManager.Task::(_debug) Task ac4da876-4cb4-4e89-8eaa-ecb46767e156: moving from state init -> state preparing >Thread-15914341::DEBUG::2013-11-20 03:29:52,288::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914341::DEBUG::2013-11-20 03:29:52,288::task::495::TaskManager.Task::(_debug) Task ac4da876-4cb4-4e89-8eaa-ecb46767e156: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914341::DEBUG::2013-11-20 03:29:52,289::task::495::TaskManager.Task::(_debug) Task ac4da876-4cb4-4e89-8eaa-ecb46767e156: moving from state preparing -> state finished >Thread-15914341::DEBUG::2013-11-20 03:29:52,290::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914341::DEBUG::2013-11-20 03:29:52,290::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914341::DEBUG::2013-11-20 03:29:52,291::task::495::TaskManager.Task::(_debug) Task ac4da876-4cb4-4e89-8eaa-ecb46767e156: ref 0 aborting False >Thread-15914341::INFO::2013-11-20 03:29:52,292::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914342::DEBUG::2013-11-20 03:29:52,296::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914342::INFO::2013-11-20 03:29:52,297::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914342::DEBUG::2013-11-20 03:29:52,298::task::495::TaskManager.Task::(_debug) Task 7a4d4826-8cdb-44a5-9823-cb455582aa6f: moving from state init -> state preparing >Thread-15914342::DEBUG::2013-11-20 03:29:52,299::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fa028e58-d974-43de-a968-5a676ea23a8a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914342::DEBUG::2013-11-20 03:29:52,299::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914342::DEBUG::2013-11-20 03:29:52,300::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914342::DEBUG::2013-11-20 03:29:52,301::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fa028e58-d974-43de-a968-5a676ea23a8a`::Granted request >Thread-15914342::DEBUG::2013-11-20 03:29:52,302::task::495::TaskManager.Task::(_debug) Task 7a4d4826-8cdb-44a5-9823-cb455582aa6f: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914342::DEBUG::2013-11-20 03:29:52,302::task::495::TaskManager.Task::(_debug) Task 7a4d4826-8cdb-44a5-9823-cb455582aa6f: ref 1 aborting False >Thread-15914342::WARNING::2013-11-20 03:29:52,304::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914342::DEBUG::2013-11-20 03:29:52,305::task::495::TaskManager.Task::(_debug) Task 7a4d4826-8cdb-44a5-9823-cb455582aa6f: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914342::DEBUG::2013-11-20 03:29:52,306::task::495::TaskManager.Task::(_debug) Task 7a4d4826-8cdb-44a5-9823-cb455582aa6f: moving from state preparing -> state finished >Thread-15914342::DEBUG::2013-11-20 03:29:52,307::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914342::DEBUG::2013-11-20 03:29:52,307::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914342::DEBUG::2013-11-20 03:29:52,308::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914342::DEBUG::2013-11-20 03:29:52,309::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914342::DEBUG::2013-11-20 03:29:52,309::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914342::DEBUG::2013-11-20 03:29:52,310::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914342::DEBUG::2013-11-20 03:29:52,310::task::495::TaskManager.Task::(_debug) Task 7a4d4826-8cdb-44a5-9823-cb455582aa6f: ref 0 aborting False >Thread-15914342::INFO::2013-11-20 03:29:52,311::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:29:53,572::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:29:54,139::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0767546 s, 13.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:29:56,156::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:29:56,701::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0756895 s, 13.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:29:58,717::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:29:59,240::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0816407 s, 12.5 MB/s\n'; <rc> = 0 >Thread-15914345::INFO::2013-11-20 03:30:00,274::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914345::DEBUG::2013-11-20 03:30:00,274::task::495::TaskManager.Task::(_debug) Task 1f2523d6-e397-4b02-9b5f-864ce6c8a2e7: moving from state init -> state preparing >Thread-15914345::DEBUG::2013-11-20 03:30:00,275::task::495::TaskManager.Task::(_debug) Task 1f2523d6-e397-4b02-9b5f-864ce6c8a2e7: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000378847122192', 'lastCheck': 1384918195.2213161, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144815444946', 'lastCheck': 1384918191.2276659, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00191307067871', 'lastCheck': 1384918193.7932119, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000526189804077', 'lastCheck': 1384918194.2159481, 'valid': True, 'code': 0}} >Thread-15914345::DEBUG::2013-11-20 03:30:00,280::task::495::TaskManager.Task::(_debug) Task 1f2523d6-e397-4b02-9b5f-864ce6c8a2e7: moving from state preparing -> state finished >Thread-15914345::DEBUG::2013-11-20 03:30:00,280::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914345::DEBUG::2013-11-20 03:30:00,280::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914345::DEBUG::2013-11-20 03:30:00,281::task::495::TaskManager.Task::(_debug) Task 1f2523d6-e397-4b02-9b5f-864ce6c8a2e7: ref 0 aborting False >Thread-15914345::INFO::2013-11-20 03:30:00,281::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000378847122192', 'lastCheck': 1384918195.2213161, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144815444946', 'lastCheck': 1384918191.2276659, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00191307067871', 'lastCheck': 1384918193.7932119, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000526189804077', 'lastCheck': 1384918194.2159481, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:30:01,257::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:30:01,828::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0938741 s, 10.9 MB/s\n'; <rc> = 0 >Thread-15914347::DEBUG::2013-11-20 03:30:02,495::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914347::INFO::2013-11-20 03:30:02,499::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914347::DEBUG::2013-11-20 03:30:02,502::task::495::TaskManager.Task::(_debug) Task 0772e2ab-17fe-4c99-b9fe-0e7103a4e18e: moving from state init -> state preparing >Thread-15914347::DEBUG::2013-11-20 03:30:02,504::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914347::DEBUG::2013-11-20 03:30:02,505::task::495::TaskManager.Task::(_debug) Task 0772e2ab-17fe-4c99-b9fe-0e7103a4e18e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914347::DEBUG::2013-11-20 03:30:02,506::task::495::TaskManager.Task::(_debug) Task 0772e2ab-17fe-4c99-b9fe-0e7103a4e18e: moving from state preparing -> state finished >Thread-15914347::DEBUG::2013-11-20 03:30:02,506::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914347::DEBUG::2013-11-20 03:30:02,507::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914347::DEBUG::2013-11-20 03:30:02,507::task::495::TaskManager.Task::(_debug) Task 0772e2ab-17fe-4c99-b9fe-0e7103a4e18e: ref 0 aborting False >Thread-15914347::INFO::2013-11-20 03:30:02,508::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914348::DEBUG::2013-11-20 03:30:02,512::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914348::INFO::2013-11-20 03:30:02,513::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914348::DEBUG::2013-11-20 03:30:02,514::task::495::TaskManager.Task::(_debug) Task d96ee0b8-3cdb-474b-9eb9-27dd24c52305: moving from state init -> state preparing >Thread-15914348::DEBUG::2013-11-20 03:30:02,515::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b6e8de14-1a72-4b85-81b6-91dd672481cf`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914348::DEBUG::2013-11-20 03:30:02,516::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914348::DEBUG::2013-11-20 03:30:02,517::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914348::DEBUG::2013-11-20 03:30:02,517::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b6e8de14-1a72-4b85-81b6-91dd672481cf`::Granted request >Thread-15914348::DEBUG::2013-11-20 03:30:02,518::task::495::TaskManager.Task::(_debug) Task d96ee0b8-3cdb-474b-9eb9-27dd24c52305: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914348::DEBUG::2013-11-20 03:30:02,519::task::495::TaskManager.Task::(_debug) Task d96ee0b8-3cdb-474b-9eb9-27dd24c52305: ref 1 aborting False >Thread-15914348::WARNING::2013-11-20 03:30:02,520::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914348::DEBUG::2013-11-20 03:30:02,526::task::495::TaskManager.Task::(_debug) Task d96ee0b8-3cdb-474b-9eb9-27dd24c52305: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914348::DEBUG::2013-11-20 03:30:02,527::task::495::TaskManager.Task::(_debug) Task d96ee0b8-3cdb-474b-9eb9-27dd24c52305: moving from state preparing -> state finished >Thread-15914348::DEBUG::2013-11-20 03:30:02,528::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914348::DEBUG::2013-11-20 03:30:02,528::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914348::DEBUG::2013-11-20 03:30:02,529::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914348::DEBUG::2013-11-20 03:30:02,530::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914348::DEBUG::2013-11-20 03:30:02,530::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914348::DEBUG::2013-11-20 03:30:02,530::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914348::DEBUG::2013-11-20 03:30:02,531::task::495::TaskManager.Task::(_debug) Task d96ee0b8-3cdb-474b-9eb9-27dd24c52305: ref 0 aborting False >Thread-15914348::INFO::2013-11-20 03:30:02,532::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:30:03,843::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:30:04,342::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0727515 s, 14.1 MB/s\n'; <rc> = 0 >Thread-15913518::INFO::2013-11-20 03:30:04,467::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:30:04,468::task::495::TaskManager.Task::(_debug) Task 5a79e085-3216-4ec3-8db8-9777c9ea33b2: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:30:04,470::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ad518400-3a8c-4193-bacc-cfb78a09589d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:30:04,470::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:30:04,471::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:30:04,472::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ad518400-3a8c-4193-bacc-cfb78a09589d`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:30:04,473::task::495::TaskManager.Task::(_debug) Task 5a79e085-3216-4ec3-8db8-9777c9ea33b2: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:30:04,473::task::495::TaskManager.Task::(_debug) Task 5a79e085-3216-4ec3-8db8-9777c9ea33b2: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:30:04,474::task::495::TaskManager.Task::(_debug) Task 5a79e085-3216-4ec3-8db8-9777c9ea33b2: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:30:04,475::task::495::TaskManager.Task::(_debug) Task 5a79e085-3216-4ec3-8db8-9777c9ea33b2: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:30:04,475::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:30:04,476::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:30:04,478::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:30:04,479::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:30:04,479::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:30:04,480::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:30:04,480::task::495::TaskManager.Task::(_debug) Task 5a79e085-3216-4ec3-8db8-9777c9ea33b2: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:30:04,481::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:30:04,482::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:30:04,482::task::495::TaskManager.Task::(_debug) Task 54542d2b-9b21-4e96-a5f6-803058f1798c: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:30:04,483::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`3323da51-2fd9-49ba-8321-7105a3dba8bc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:30:04,484::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:30:04,485::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:30:04,485::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`3323da51-2fd9-49ba-8321-7105a3dba8bc`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:30:04,486::task::495::TaskManager.Task::(_debug) Task 54542d2b-9b21-4e96-a5f6-803058f1798c: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:30:04,487::task::495::TaskManager.Task::(_debug) Task 54542d2b-9b21-4e96-a5f6-803058f1798c: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:30:04,487::task::495::TaskManager.Task::(_debug) Task 54542d2b-9b21-4e96-a5f6-803058f1798c: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:30:04,488::task::495::TaskManager.Task::(_debug) Task 54542d2b-9b21-4e96-a5f6-803058f1798c: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:30:04,488::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:30:04,489::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:30:04,490::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:30:04,491::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:30:04,492::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:30:04,492::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:30:04,493::task::495::TaskManager.Task::(_debug) Task 54542d2b-9b21-4e96-a5f6-803058f1798c: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:30:04,494::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:30:06,357::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:30:06,901::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0773576 s, 13.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:30:08,920::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:30:09,499::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0939553 s, 10.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:30:11,514::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:30:12,073::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0673833 s, 15.2 MB/s\n'; <rc> = 0 >Thread-15914353::DEBUG::2013-11-20 03:30:13,256::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914353::INFO::2013-11-20 03:30:13,257::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914353::DEBUG::2013-11-20 03:30:13,258::task::495::TaskManager.Task::(_debug) Task cb478993-87fe-4294-a3fe-52379323f2a9: moving from state init -> state preparing >Thread-15914353::DEBUG::2013-11-20 03:30:13,259::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914353::DEBUG::2013-11-20 03:30:13,259::task::495::TaskManager.Task::(_debug) Task cb478993-87fe-4294-a3fe-52379323f2a9: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914353::DEBUG::2013-11-20 03:30:13,260::task::495::TaskManager.Task::(_debug) Task cb478993-87fe-4294-a3fe-52379323f2a9: moving from state preparing -> state finished >Thread-15914353::DEBUG::2013-11-20 03:30:13,261::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914353::DEBUG::2013-11-20 03:30:13,261::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914353::DEBUG::2013-11-20 03:30:13,262::task::495::TaskManager.Task::(_debug) Task cb478993-87fe-4294-a3fe-52379323f2a9: ref 0 aborting False >Thread-15914353::INFO::2013-11-20 03:30:13,262::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914354::DEBUG::2013-11-20 03:30:13,267::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914354::INFO::2013-11-20 03:30:13,268::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914354::DEBUG::2013-11-20 03:30:13,269::task::495::TaskManager.Task::(_debug) Task 6e8bb751-fdf9-4a6f-baa9-37e7318be74a: moving from state init -> state preparing >Thread-15914354::DEBUG::2013-11-20 03:30:13,271::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a64548b6-5e57-437b-831e-d38b1f9682ee`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914354::DEBUG::2013-11-20 03:30:13,272::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914354::DEBUG::2013-11-20 03:30:13,274::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914354::DEBUG::2013-11-20 03:30:13,275::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a64548b6-5e57-437b-831e-d38b1f9682ee`::Granted request >Thread-15914354::DEBUG::2013-11-20 03:30:13,276::task::495::TaskManager.Task::(_debug) Task 6e8bb751-fdf9-4a6f-baa9-37e7318be74a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914354::DEBUG::2013-11-20 03:30:13,277::task::495::TaskManager.Task::(_debug) Task 6e8bb751-fdf9-4a6f-baa9-37e7318be74a: ref 1 aborting False >Thread-15914354::WARNING::2013-11-20 03:30:13,281::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914354::DEBUG::2013-11-20 03:30:13,282::task::495::TaskManager.Task::(_debug) Task 6e8bb751-fdf9-4a6f-baa9-37e7318be74a: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914354::DEBUG::2013-11-20 03:30:13,284::task::495::TaskManager.Task::(_debug) Task 6e8bb751-fdf9-4a6f-baa9-37e7318be74a: moving from state preparing -> state finished >Thread-15914354::DEBUG::2013-11-20 03:30:13,285::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914354::DEBUG::2013-11-20 03:30:13,286::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914354::DEBUG::2013-11-20 03:30:13,287::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914354::DEBUG::2013-11-20 03:30:13,288::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914354::DEBUG::2013-11-20 03:30:13,289::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914354::DEBUG::2013-11-20 03:30:13,289::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914354::DEBUG::2013-11-20 03:30:13,290::task::495::TaskManager.Task::(_debug) Task 6e8bb751-fdf9-4a6f-baa9-37e7318be74a: ref 0 aborting False >Thread-15914354::INFO::2013-11-20 03:30:13,291::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:30:14,087::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:30:14,634::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0770708 s, 13.3 MB/s\n'; <rc> = 0 >Thread-15914355::INFO::2013-11-20 03:30:15,672::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914355::DEBUG::2013-11-20 03:30:15,674::task::495::TaskManager.Task::(_debug) Task 3e1708d3-5d5e-463a-a191-808268c70a61: moving from state init -> state preparing >Thread-15914355::DEBUG::2013-11-20 03:30:15,674::task::495::TaskManager.Task::(_debug) Task 3e1708d3-5d5e-463a-a191-808268c70a61: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000300884246826', 'lastCheck': 1384918205.480418, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00107479095459', 'lastCheck': 1384918211.774291, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0018470287323', 'lastCheck': 1384918214.304702, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000363111495972', 'lastCheck': 1384918214.7703249, 'valid': True, 'code': 0}} >Thread-15914355::DEBUG::2013-11-20 03:30:15,675::task::495::TaskManager.Task::(_debug) Task 3e1708d3-5d5e-463a-a191-808268c70a61: moving from state preparing -> state finished >Thread-15914355::DEBUG::2013-11-20 03:30:15,676::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914355::DEBUG::2013-11-20 03:30:15,676::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914355::DEBUG::2013-11-20 03:30:15,677::task::495::TaskManager.Task::(_debug) Task 3e1708d3-5d5e-463a-a191-808268c70a61: ref 0 aborting False >Thread-15914355::INFO::2013-11-20 03:30:15,678::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000300884246826', 'lastCheck': 1384918205.480418, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00107479095459', 'lastCheck': 1384918211.774291, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0018470287323', 'lastCheck': 1384918214.304702, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000363111495972', 'lastCheck': 1384918214.7703249, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:30:16,649::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:30:17,211::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0880386 s, 11.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:30:19,223::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913546::INFO::2013-11-20 03:30:19,441::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:30:19,444::task::495::TaskManager.Task::(_debug) Task 51ba59e0-646a-4d11-902e-ba98379c3856: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:30:19,447::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`580075f0-db91-4ee9-877f-a153d6b223d2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:30:19,449::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:30:19,450::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:30:19,450::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`580075f0-db91-4ee9-877f-a153d6b223d2`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:30:19,452::task::495::TaskManager.Task::(_debug) Task 51ba59e0-646a-4d11-902e-ba98379c3856: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:30:19,452::task::495::TaskManager.Task::(_debug) Task 51ba59e0-646a-4d11-902e-ba98379c3856: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:30:19,453::task::495::TaskManager.Task::(_debug) Task 51ba59e0-646a-4d11-902e-ba98379c3856: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:30:19,454::task::495::TaskManager.Task::(_debug) Task 51ba59e0-646a-4d11-902e-ba98379c3856: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:30:19,454::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:30:19,455::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:30:19,456::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:30:19,456::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:30:19,457::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:30:19,457::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:30:19,458::task::495::TaskManager.Task::(_debug) Task 51ba59e0-646a-4d11-902e-ba98379c3856: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:30:19,459::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:30:19,460::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:30:19,460::task::495::TaskManager.Task::(_debug) Task db6a2c96-cf54-478b-82b6-99fceb578ed8: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:30:19,461::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`c78aadae-e553-4662-a8d0-a0a0027c8e5f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:30:19,462::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:30:19,462::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:30:19,463::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`c78aadae-e553-4662-a8d0-a0a0027c8e5f`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:30:19,464::task::495::TaskManager.Task::(_debug) Task db6a2c96-cf54-478b-82b6-99fceb578ed8: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:30:19,465::task::495::TaskManager.Task::(_debug) Task db6a2c96-cf54-478b-82b6-99fceb578ed8: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:30:19,466::task::495::TaskManager.Task::(_debug) Task db6a2c96-cf54-478b-82b6-99fceb578ed8: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:30:19,466::task::495::TaskManager.Task::(_debug) Task db6a2c96-cf54-478b-82b6-99fceb578ed8: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:30:19,467::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:30:19,467::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:30:19,468::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:30:19,468::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:30:19,469::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:30:19,470::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:30:19,470::task::495::TaskManager.Task::(_debug) Task db6a2c96-cf54-478b-82b6-99fceb578ed8: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:30:19,471::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:30:19,472::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:30:19,472::task::495::TaskManager.Task::(_debug) Task c4c4f2a7-c1e5-467e-bccb-f0976616f057: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:30:19,473::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`7c6ed090-cb77-40c0-a295-fd8a31e38571`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:30:19,474::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:30:19,474::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:30:19,475::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`7c6ed090-cb77-40c0-a295-fd8a31e38571`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:30:19,476::task::495::TaskManager.Task::(_debug) Task c4c4f2a7-c1e5-467e-bccb-f0976616f057: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:30:19,476::task::495::TaskManager.Task::(_debug) Task c4c4f2a7-c1e5-467e-bccb-f0976616f057: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:30:19,477::task::495::TaskManager.Task::(_debug) Task c4c4f2a7-c1e5-467e-bccb-f0976616f057: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:30:19,478::task::495::TaskManager.Task::(_debug) Task c4c4f2a7-c1e5-467e-bccb-f0976616f057: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:30:19,478::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:30:19,479::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:30:19,479::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:30:19,480::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:30:19,480::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:30:19,481::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:30:19,482::task::495::TaskManager.Task::(_debug) Task c4c4f2a7-c1e5-467e-bccb-f0976616f057: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:30:19,482::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:30:19,483::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:30:19,483::task::495::TaskManager.Task::(_debug) Task 054d8bd0-223f-4a35-b199-47682e966b36: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:30:19,485::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fe6fef9a-c27c-4bba-944e-34394fa5fe51`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:30:19,485::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:30:19,486::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:30:19,486::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fe6fef9a-c27c-4bba-944e-34394fa5fe51`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:30:19,487::task::495::TaskManager.Task::(_debug) Task 054d8bd0-223f-4a35-b199-47682e966b36: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:30:19,488::task::495::TaskManager.Task::(_debug) Task 054d8bd0-223f-4a35-b199-47682e966b36: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:30:19,488::task::495::TaskManager.Task::(_debug) Task 054d8bd0-223f-4a35-b199-47682e966b36: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:30:19,489::task::495::TaskManager.Task::(_debug) Task 054d8bd0-223f-4a35-b199-47682e966b36: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:30:19,489::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:30:19,490::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:30:19,490::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:30:19,491::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:30:19,492::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:30:19,492::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:30:19,493::task::495::TaskManager.Task::(_debug) Task 054d8bd0-223f-4a35-b199-47682e966b36: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:30:19,493::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:30:19,765::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0788895 s, 13.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:30:21,781::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:30:22,349::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0802797 s, 12.8 MB/s\n'; <rc> = 0 >Thread-15914359::DEBUG::2013-11-20 03:30:23,389::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914359::INFO::2013-11-20 03:30:23,391::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914359::DEBUG::2013-11-20 03:30:23,391::task::495::TaskManager.Task::(_debug) Task d32ba378-fdce-46e1-b611-a0e8903127cb: moving from state init -> state preparing >Thread-15914359::DEBUG::2013-11-20 03:30:23,392::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914359::DEBUG::2013-11-20 03:30:23,393::task::495::TaskManager.Task::(_debug) Task d32ba378-fdce-46e1-b611-a0e8903127cb: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914359::DEBUG::2013-11-20 03:30:23,394::task::495::TaskManager.Task::(_debug) Task d32ba378-fdce-46e1-b611-a0e8903127cb: moving from state preparing -> state finished >Thread-15914359::DEBUG::2013-11-20 03:30:23,394::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914359::DEBUG::2013-11-20 03:30:23,395::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914359::DEBUG::2013-11-20 03:30:23,396::task::495::TaskManager.Task::(_debug) Task d32ba378-fdce-46e1-b611-a0e8903127cb: ref 0 aborting False >Thread-15914359::INFO::2013-11-20 03:30:23,396::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914360::DEBUG::2013-11-20 03:30:23,408::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914360::INFO::2013-11-20 03:30:23,415::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914360::DEBUG::2013-11-20 03:30:23,419::task::495::TaskManager.Task::(_debug) Task 94245858-1af5-40fc-846e-dfda6f6e7675: moving from state init -> state preparing >Thread-15914360::DEBUG::2013-11-20 03:30:23,425::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d5a8b221-9a3e-4006-88cc-adaab73aebc0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914360::DEBUG::2013-11-20 03:30:23,427::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914360::DEBUG::2013-11-20 03:30:23,428::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914360::DEBUG::2013-11-20 03:30:23,429::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d5a8b221-9a3e-4006-88cc-adaab73aebc0`::Granted request >Thread-15914360::DEBUG::2013-11-20 03:30:23,430::task::495::TaskManager.Task::(_debug) Task 94245858-1af5-40fc-846e-dfda6f6e7675: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914360::DEBUG::2013-11-20 03:30:23,431::task::495::TaskManager.Task::(_debug) Task 94245858-1af5-40fc-846e-dfda6f6e7675: ref 1 aborting False >Thread-15914360::WARNING::2013-11-20 03:30:23,433::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914360::DEBUG::2013-11-20 03:30:23,434::task::495::TaskManager.Task::(_debug) Task 94245858-1af5-40fc-846e-dfda6f6e7675: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914360::DEBUG::2013-11-20 03:30:23,435::task::495::TaskManager.Task::(_debug) Task 94245858-1af5-40fc-846e-dfda6f6e7675: moving from state preparing -> state finished >Thread-15914360::DEBUG::2013-11-20 03:30:23,436::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914360::DEBUG::2013-11-20 03:30:23,436::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914360::DEBUG::2013-11-20 03:30:23,437::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914360::DEBUG::2013-11-20 03:30:23,438::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914360::DEBUG::2013-11-20 03:30:23,438::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914360::DEBUG::2013-11-20 03:30:23,439::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914360::DEBUG::2013-11-20 03:30:23,440::task::495::TaskManager.Task::(_debug) Task 94245858-1af5-40fc-846e-dfda6f6e7675: ref 0 aborting False >Thread-15914360::INFO::2013-11-20 03:30:23,440::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:30:24,363::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:30:24,888::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.073269 s, 14.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:30:26,903::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:30:27,436::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0738863 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:30:29,452::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913639::INFO::2013-11-20 03:30:29,666::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:30:29,668::task::495::TaskManager.Task::(_debug) Task ed99edb0-b49f-4b0d-9ff3-a3f4fa19ed8d: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:30:29,671::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`5fd76e02-fdb9-49b7-aeee-0bc549d12081`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:30:29,672::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:30:29,674::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:30:29,674::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`5fd76e02-fdb9-49b7-aeee-0bc549d12081`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:30:29,676::task::495::TaskManager.Task::(_debug) Task ed99edb0-b49f-4b0d-9ff3-a3f4fa19ed8d: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:30:29,677::task::495::TaskManager.Task::(_debug) Task ed99edb0-b49f-4b0d-9ff3-a3f4fa19ed8d: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:30:29,677::task::495::TaskManager.Task::(_debug) Task ed99edb0-b49f-4b0d-9ff3-a3f4fa19ed8d: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:30:29,678::task::495::TaskManager.Task::(_debug) Task ed99edb0-b49f-4b0d-9ff3-a3f4fa19ed8d: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:30:29,679::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:30:29,679::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:30:29,680::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:30:29,681::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:30:29,681::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:30:29,682::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:30:29,683::task::495::TaskManager.Task::(_debug) Task ed99edb0-b49f-4b0d-9ff3-a3f4fa19ed8d: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:30:29,683::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:30:29,972::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0819056 s, 12.5 MB/s\n'; <rc> = 0 >Thread-15914363::DEBUG::2013-11-20 03:30:30,106::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15914363::DEBUG::2013-11-20 03:30:30,787::clientIF::1219::vds::(fenceNode) rc 0 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = On >Done > err >Thread-15914364::DEBUG::2013-11-20 03:30:30,800::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=off,secure=,options=power_wait=4 >lanplus=1) >Thread-15914365::INFO::2013-11-20 03:30:31,078::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914365::DEBUG::2013-11-20 03:30:31,080::task::495::TaskManager.Task::(_debug) Task f37398bb-8f94-45b6-a932-1ea7e23c6d07: moving from state init -> state preparing >Thread-15914365::DEBUG::2013-11-20 03:30:31,081::task::495::TaskManager.Task::(_debug) Task f37398bb-8f94-45b6-a932-1ea7e23c6d07: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000282049179077', 'lastCheck': 1384918226.016243, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00170803070068', 'lastCheck': 1384918222.0241311, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00166201591492', 'lastCheck': 1384918224.5941899, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000417947769165', 'lastCheck': 1384918225.0157051, 'valid': True, 'code': 0}} >Thread-15914365::DEBUG::2013-11-20 03:30:31,082::task::495::TaskManager.Task::(_debug) Task f37398bb-8f94-45b6-a932-1ea7e23c6d07: moving from state preparing -> state finished >Thread-15914365::DEBUG::2013-11-20 03:30:31,082::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914365::DEBUG::2013-11-20 03:30:31,083::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914365::DEBUG::2013-11-20 03:30:31,084::task::495::TaskManager.Task::(_debug) Task f37398bb-8f94-45b6-a932-1ea7e23c6d07: ref 0 aborting False >Thread-15914365::INFO::2013-11-20 03:30:31,084::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000282049179077', 'lastCheck': 1384918226.016243, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00170803070068', 'lastCheck': 1384918222.0241311, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00166201591492', 'lastCheck': 1384918224.5941899, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000417947769165', 'lastCheck': 1384918225.0157051, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:30:31,987::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:30:32,513::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0719724 s, 14.2 MB/s\n'; <rc> = 0 >Thread-15914368::DEBUG::2013-11-20 03:30:33,536::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914368::INFO::2013-11-20 03:30:33,537::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914368::DEBUG::2013-11-20 03:30:33,538::task::495::TaskManager.Task::(_debug) Task 7feeabad-c0e0-48f5-a6a5-5fd2cb887050: moving from state init -> state preparing >Thread-15914368::DEBUG::2013-11-20 03:30:33,539::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914368::DEBUG::2013-11-20 03:30:33,540::task::495::TaskManager.Task::(_debug) Task 7feeabad-c0e0-48f5-a6a5-5fd2cb887050: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914368::DEBUG::2013-11-20 03:30:33,540::task::495::TaskManager.Task::(_debug) Task 7feeabad-c0e0-48f5-a6a5-5fd2cb887050: moving from state preparing -> state finished >Thread-15914368::DEBUG::2013-11-20 03:30:33,541::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914368::DEBUG::2013-11-20 03:30:33,541::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914368::DEBUG::2013-11-20 03:30:33,542::task::495::TaskManager.Task::(_debug) Task 7feeabad-c0e0-48f5-a6a5-5fd2cb887050: ref 0 aborting False >Thread-15914368::INFO::2013-11-20 03:30:33,543::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914369::DEBUG::2013-11-20 03:30:33,552::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914369::INFO::2013-11-20 03:30:33,553::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914369::DEBUG::2013-11-20 03:30:33,553::task::495::TaskManager.Task::(_debug) Task 3189833a-84ba-42f1-bf77-9b6712e22287: moving from state init -> state preparing >Thread-15914369::DEBUG::2013-11-20 03:30:33,555::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`efb9e2cd-f963-43a1-af1a-f1ff7fabe6ac`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914369::DEBUG::2013-11-20 03:30:33,555::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914369::DEBUG::2013-11-20 03:30:33,556::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914369::DEBUG::2013-11-20 03:30:33,557::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`efb9e2cd-f963-43a1-af1a-f1ff7fabe6ac`::Granted request >Thread-15914369::DEBUG::2013-11-20 03:30:33,558::task::495::TaskManager.Task::(_debug) Task 3189833a-84ba-42f1-bf77-9b6712e22287: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914369::DEBUG::2013-11-20 03:30:33,559::task::495::TaskManager.Task::(_debug) Task 3189833a-84ba-42f1-bf77-9b6712e22287: ref 1 aborting False >Thread-15914369::WARNING::2013-11-20 03:30:33,561::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914369::DEBUG::2013-11-20 03:30:33,562::task::495::TaskManager.Task::(_debug) Task 3189833a-84ba-42f1-bf77-9b6712e22287: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914369::DEBUG::2013-11-20 03:30:33,563::task::495::TaskManager.Task::(_debug) Task 3189833a-84ba-42f1-bf77-9b6712e22287: moving from state preparing -> state finished >Thread-15914369::DEBUG::2013-11-20 03:30:33,564::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914369::DEBUG::2013-11-20 03:30:33,564::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914369::DEBUG::2013-11-20 03:30:33,565::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914369::DEBUG::2013-11-20 03:30:33,566::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914369::DEBUG::2013-11-20 03:30:33,566::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914369::DEBUG::2013-11-20 03:30:33,567::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914369::DEBUG::2013-11-20 03:30:33,568::task::495::TaskManager.Task::(_debug) Task 3189833a-84ba-42f1-bf77-9b6712e22287: ref 0 aborting False >Thread-15914369::INFO::2013-11-20 03:30:33,568::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:30:34,531::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:30:35,018::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0742731 s, 13.8 MB/s\n'; <rc> = 0 >Thread-15914367::DEBUG::2013-11-20 03:30:36,225::clientIF::1179::vds::(waitForPid) rc 0 inp agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=off >passwd=XXXX >power_wait=4 >lanplus=1 out Powering off machine @ IPMI:192.168.100.119...Done > err >Thread-15914371::DEBUG::2013-11-20 03:30:36,323::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15914371::DEBUG::2013-11-20 03:30:36,916::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Thread-15914372::DEBUG::2013-11-20 03:30:36,923::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Dummy-15912651::DEBUG::2013-11-20 03:30:37,128::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914372::DEBUG::2013-11-20 03:30:37,739::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Dummy-15912651::DEBUG::2013-11-20 03:30:37,872::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0837742 s, 12.2 MB/s\n'; <rc> = 0 >Thread-15914374::DEBUG::2013-11-20 03:30:39,602::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Dummy-15912651::DEBUG::2013-11-20 03:30:39,909::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914374::DEBUG::2013-11-20 03:30:40,326::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Thread-15914375::DEBUG::2013-11-20 03:30:40,334::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=on,secure=,options=power_wait=4 >lanplus=1) >Dummy-15912651::DEBUG::2013-11-20 03:30:40,583::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.224642 s, 4.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:30:42,600::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914376::DEBUG::2013-11-20 03:30:42,837::clientIF::1179::vds::(waitForPid) rc 1 inp agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=on >passwd=XXXX >power_wait=4 >lanplus=1 out Powering on machine @ IPMI:192.168.100.119...Failed > err >Dummy-15912651::DEBUG::2013-11-20 03:30:43,167::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0774949 s, 13.2 MB/s\n'; <rc> = 0 >Thread-15914379::DEBUG::2013-11-20 03:30:44,111::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914379::INFO::2013-11-20 03:30:44,112::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914379::DEBUG::2013-11-20 03:30:44,113::task::495::TaskManager.Task::(_debug) Task 190125c1-231b-4cae-9f2b-0a976046fed2: moving from state init -> state preparing >Thread-15914379::DEBUG::2013-11-20 03:30:44,113::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914379::DEBUG::2013-11-20 03:30:44,114::task::495::TaskManager.Task::(_debug) Task 190125c1-231b-4cae-9f2b-0a976046fed2: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914379::DEBUG::2013-11-20 03:30:44,114::task::495::TaskManager.Task::(_debug) Task 190125c1-231b-4cae-9f2b-0a976046fed2: moving from state preparing -> state finished >Thread-15914379::DEBUG::2013-11-20 03:30:44,115::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914379::DEBUG::2013-11-20 03:30:44,115::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914379::DEBUG::2013-11-20 03:30:44,115::task::495::TaskManager.Task::(_debug) Task 190125c1-231b-4cae-9f2b-0a976046fed2: ref 0 aborting False >Thread-15914379::INFO::2013-11-20 03:30:44,116::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914380::DEBUG::2013-11-20 03:30:44,119::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914380::INFO::2013-11-20 03:30:44,120::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914380::DEBUG::2013-11-20 03:30:44,120::task::495::TaskManager.Task::(_debug) Task 1c5215fa-1ee0-4568-8d22-81852b364a8e: moving from state init -> state preparing >Thread-15914380::DEBUG::2013-11-20 03:30:44,121::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`444306b7-bb8a-427c-bd40-f77e9609c1d3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914380::DEBUG::2013-11-20 03:30:44,121::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914380::DEBUG::2013-11-20 03:30:44,122::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914380::DEBUG::2013-11-20 03:30:44,122::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`444306b7-bb8a-427c-bd40-f77e9609c1d3`::Granted request >Thread-15914380::DEBUG::2013-11-20 03:30:44,123::task::495::TaskManager.Task::(_debug) Task 1c5215fa-1ee0-4568-8d22-81852b364a8e: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914380::DEBUG::2013-11-20 03:30:44,123::task::495::TaskManager.Task::(_debug) Task 1c5215fa-1ee0-4568-8d22-81852b364a8e: ref 1 aborting False >Thread-15914380::WARNING::2013-11-20 03:30:44,124::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914380::DEBUG::2013-11-20 03:30:44,132::task::495::TaskManager.Task::(_debug) Task 1c5215fa-1ee0-4568-8d22-81852b364a8e: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914380::DEBUG::2013-11-20 03:30:44,133::task::495::TaskManager.Task::(_debug) Task 1c5215fa-1ee0-4568-8d22-81852b364a8e: moving from state preparing -> state finished >Thread-15914380::DEBUG::2013-11-20 03:30:44,133::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914380::DEBUG::2013-11-20 03:30:44,133::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914380::DEBUG::2013-11-20 03:30:44,134::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914380::DEBUG::2013-11-20 03:30:44,134::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914380::DEBUG::2013-11-20 03:30:44,135::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914380::DEBUG::2013-11-20 03:30:44,135::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914380::DEBUG::2013-11-20 03:30:44,135::task::495::TaskManager.Task::(_debug) Task 1c5215fa-1ee0-4568-8d22-81852b364a8e: ref 0 aborting False >Thread-15914380::INFO::2013-11-20 03:30:44,136::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:30:45,183::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914381::DEBUG::2013-11-20 03:30:45,769::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Dummy-15912651::DEBUG::2013-11-20 03:30:45,991::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0929709 s, 11.0 MB/s\n'; <rc> = 0 >Thread-15914382::INFO::2013-11-20 03:30:46,826::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914382::DEBUG::2013-11-20 03:30:46,827::task::495::TaskManager.Task::(_debug) Task f2f38615-a0fe-4984-9651-b08192d71a90: moving from state init -> state preparing >Thread-15914382::DEBUG::2013-11-20 03:30:46,828::task::495::TaskManager.Task::(_debug) Task f2f38615-a0fe-4984-9651-b08192d71a90: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00034499168396', 'lastCheck': 1384918246.6027889, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142502784729', 'lastCheck': 1384918242.5514531, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00155091285706', 'lastCheck': 1384918244.8458891, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00145387649536', 'lastCheck': 1384918245.548317, 'valid': True, 'code': 0}} >Thread-15914382::DEBUG::2013-11-20 03:30:46,829::task::495::TaskManager.Task::(_debug) Task f2f38615-a0fe-4984-9651-b08192d71a90: moving from state preparing -> state finished >Thread-15914382::DEBUG::2013-11-20 03:30:46,829::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914382::DEBUG::2013-11-20 03:30:46,830::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914382::DEBUG::2013-11-20 03:30:46,830::task::495::TaskManager.Task::(_debug) Task f2f38615-a0fe-4984-9651-b08192d71a90: ref 0 aborting False >Thread-15914382::INFO::2013-11-20 03:30:46,831::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00034499168396', 'lastCheck': 1384918246.6027889, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00142502784729', 'lastCheck': 1384918242.5514531, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00155091285706', 'lastCheck': 1384918244.8458891, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00145387649536', 'lastCheck': 1384918245.548317, 'valid': True, 'code': 0}} >Thread-15913695::INFO::2013-11-20 03:30:47,598::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:30:47,599::task::495::TaskManager.Task::(_debug) Task 9b4c94d9-6021-4d2b-9c29-4024d658b817: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:30:47,601::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`295729f3-d6cb-45d0-95e8-8ebaf0c65f54`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:30:47,602::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:30:47,602::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:30:47,603::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`295729f3-d6cb-45d0-95e8-8ebaf0c65f54`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:30:47,604::task::495::TaskManager.Task::(_debug) Task 9b4c94d9-6021-4d2b-9c29-4024d658b817: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:30:47,605::task::495::TaskManager.Task::(_debug) Task 9b4c94d9-6021-4d2b-9c29-4024d658b817: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:30:47,605::task::495::TaskManager.Task::(_debug) Task 9b4c94d9-6021-4d2b-9c29-4024d658b817: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:30:47,606::task::495::TaskManager.Task::(_debug) Task 9b4c94d9-6021-4d2b-9c29-4024d658b817: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:30:47,607::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:30:47,607::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:30:47,608::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:30:47,609::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:30:47,609::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:30:47,610::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:30:47,610::task::495::TaskManager.Task::(_debug) Task 9b4c94d9-6021-4d2b-9c29-4024d658b817: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:30:47,611::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:30:48,006::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914381::DEBUG::2013-11-20 03:30:48,330::clientIF::1219::vds::(fenceNode) rc 1 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Unknown >Failed > err >Thread-15914384::DEBUG::2013-11-20 03:30:48,342::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Dummy-15912651::DEBUG::2013-11-20 03:30:48,748::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.318232 s, 3.2 MB/s\n'; <rc> = 0 >Thread-15914384::DEBUG::2013-11-20 03:30:48,933::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Thread-15913658::INFO::2013-11-20 03:30:49,982::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:30:49,989::task::495::TaskManager.Task::(_debug) Task d7aa541f-aca8-4317-9ac8-66bc9223dd02: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:30:49,991::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ad33fea4-c0ec-4d4b-b23e-94d483a45473`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:30:49,992::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:30:49,993::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:30:49,994::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ad33fea4-c0ec-4d4b-b23e-94d483a45473`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:30:49,995::task::495::TaskManager.Task::(_debug) Task d7aa541f-aca8-4317-9ac8-66bc9223dd02: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:30:49,996::task::495::TaskManager.Task::(_debug) Task d7aa541f-aca8-4317-9ac8-66bc9223dd02: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:30:49,997::task::495::TaskManager.Task::(_debug) Task d7aa541f-aca8-4317-9ac8-66bc9223dd02: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:30:49,998::task::495::TaskManager.Task::(_debug) Task d7aa541f-aca8-4317-9ac8-66bc9223dd02: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:30:49,998::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:30:49,999::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:30:50,000::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:30:50,000::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:30:50,001::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:30:50,002::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:30:50,002::task::495::TaskManager.Task::(_debug) Task d7aa541f-aca8-4317-9ac8-66bc9223dd02: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:30:50,003::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:30:50,764::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:30:51,331::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0813691 s, 12.6 MB/s\n'; <rc> = 0 >Thread-15912392::INFO::2013-11-20 03:30:52,561::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:30:52,562::task::495::TaskManager.Task::(_debug) Task 2bb88f48-6f72-493c-b94c-d359b2614b39: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:30:52,566::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f38e380f-a5de-4a8a-a017-00b957796129`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:30:52,567::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:30:52,568::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:30:52,569::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f38e380f-a5de-4a8a-a017-00b957796129`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:30:52,570::task::495::TaskManager.Task::(_debug) Task 2bb88f48-6f72-493c-b94c-d359b2614b39: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:30:52,572::task::495::TaskManager.Task::(_debug) Task 2bb88f48-6f72-493c-b94c-d359b2614b39: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:30:52,572::task::495::TaskManager.Task::(_debug) Task 2bb88f48-6f72-493c-b94c-d359b2614b39: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:30:52,573::task::495::TaskManager.Task::(_debug) Task 2bb88f48-6f72-493c-b94c-d359b2614b39: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:30:52,574::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:30:52,574::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:30:52,575::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:30:52,576::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:30:52,576::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:30:52,577::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:30:52,577::task::495::TaskManager.Task::(_debug) Task 2bb88f48-6f72-493c-b94c-d359b2614b39: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:30:52,578::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:30:53,344::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:30:53,884::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0828897 s, 12.4 MB/s\n'; <rc> = 0 >Thread-15914387::DEBUG::2013-11-20 03:30:54,290::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914387::INFO::2013-11-20 03:30:54,292::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914387::DEBUG::2013-11-20 03:30:54,292::task::495::TaskManager.Task::(_debug) Task 0a8bd090-7dbb-4df9-ab7f-e131ebd28a8b: moving from state init -> state preparing >Thread-15914387::DEBUG::2013-11-20 03:30:54,293::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914387::DEBUG::2013-11-20 03:30:54,294::task::495::TaskManager.Task::(_debug) Task 0a8bd090-7dbb-4df9-ab7f-e131ebd28a8b: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914387::DEBUG::2013-11-20 03:30:54,295::task::495::TaskManager.Task::(_debug) Task 0a8bd090-7dbb-4df9-ab7f-e131ebd28a8b: moving from state preparing -> state finished >Thread-15914387::DEBUG::2013-11-20 03:30:54,295::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914387::DEBUG::2013-11-20 03:30:54,296::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914387::DEBUG::2013-11-20 03:30:54,296::task::495::TaskManager.Task::(_debug) Task 0a8bd090-7dbb-4df9-ab7f-e131ebd28a8b: ref 0 aborting False >Thread-15914387::INFO::2013-11-20 03:30:54,297::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914388::DEBUG::2013-11-20 03:30:54,313::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914388::INFO::2013-11-20 03:30:54,314::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914388::DEBUG::2013-11-20 03:30:54,314::task::495::TaskManager.Task::(_debug) Task 7d8ea776-acee-492e-b59e-9e7cdf8339d1: moving from state init -> state preparing >Thread-15914388::DEBUG::2013-11-20 03:30:54,316::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a3a7163e-f065-4f3e-9a67-3c6f8b6b0083`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914388::DEBUG::2013-11-20 03:30:54,317::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914388::DEBUG::2013-11-20 03:30:54,317::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914388::DEBUG::2013-11-20 03:30:54,318::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a3a7163e-f065-4f3e-9a67-3c6f8b6b0083`::Granted request >Thread-15914388::DEBUG::2013-11-20 03:30:54,319::task::495::TaskManager.Task::(_debug) Task 7d8ea776-acee-492e-b59e-9e7cdf8339d1: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914388::DEBUG::2013-11-20 03:30:54,320::task::495::TaskManager.Task::(_debug) Task 7d8ea776-acee-492e-b59e-9e7cdf8339d1: ref 1 aborting False >Thread-15914388::WARNING::2013-11-20 03:30:54,322::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914388::DEBUG::2013-11-20 03:30:54,323::task::495::TaskManager.Task::(_debug) Task 7d8ea776-acee-492e-b59e-9e7cdf8339d1: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914388::DEBUG::2013-11-20 03:30:54,324::task::495::TaskManager.Task::(_debug) Task 7d8ea776-acee-492e-b59e-9e7cdf8339d1: moving from state preparing -> state finished >Thread-15914388::DEBUG::2013-11-20 03:30:54,325::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914388::DEBUG::2013-11-20 03:30:54,325::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914388::DEBUG::2013-11-20 03:30:54,326::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914388::DEBUG::2013-11-20 03:30:54,327::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914388::DEBUG::2013-11-20 03:30:54,327::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914388::DEBUG::2013-11-20 03:30:54,328::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914388::DEBUG::2013-11-20 03:30:54,329::task::495::TaskManager.Task::(_debug) Task 7d8ea776-acee-492e-b59e-9e7cdf8339d1: ref 0 aborting False >Thread-15914388::INFO::2013-11-20 03:30:54,329::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:30:55,900::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:30:56,471::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0788241 s, 13.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:30:58,485::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914390::DEBUG::2013-11-20 03:30:58,988::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Dummy-15912651::DEBUG::2013-11-20 03:30:59,315::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.330806 s, 3.1 MB/s\n'; <rc> = 0 >Thread-15914390::DEBUG::2013-11-20 03:30:59,565::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Thread-15914392::DEBUG::2013-11-20 03:30:59,571::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15914392::DEBUG::2013-11-20 03:31:00,159::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Dummy-15912651::DEBUG::2013-11-20 03:31:01,332::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:31:01,873::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0825349 s, 12.4 MB/s\n'; <rc> = 0 >Thread-15914393::INFO::2013-11-20 03:31:02,247::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914393::DEBUG::2013-11-20 03:31:02,248::task::495::TaskManager.Task::(_debug) Task a31c2986-60f9-46a7-93b1-9a05a29e52ec: moving from state init -> state preparing >Thread-15914393::DEBUG::2013-11-20 03:31:02,249::task::495::TaskManager.Task::(_debug) Task a31c2986-60f9-46a7-93b1-9a05a29e52ec: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000333070755005', 'lastCheck': 1384918256.61498, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00151586532593', 'lastCheck': 1384918252.573221, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00178909301758', 'lastCheck': 1384918254.9975531, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000470876693726', 'lastCheck': 1384918255.563375, 'valid': True, 'code': 0}} >Thread-15914393::DEBUG::2013-11-20 03:31:02,253::task::495::TaskManager.Task::(_debug) Task a31c2986-60f9-46a7-93b1-9a05a29e52ec: moving from state preparing -> state finished >Thread-15914393::DEBUG::2013-11-20 03:31:02,254::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914393::DEBUG::2013-11-20 03:31:02,255::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914393::DEBUG::2013-11-20 03:31:02,255::task::495::TaskManager.Task::(_debug) Task a31c2986-60f9-46a7-93b1-9a05a29e52ec: ref 0 aborting False >Thread-15914393::INFO::2013-11-20 03:31:02,256::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000333070755005', 'lastCheck': 1384918256.61498, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00151586532593', 'lastCheck': 1384918252.573221, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00178909301758', 'lastCheck': 1384918254.9975531, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000470876693726', 'lastCheck': 1384918255.563375, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:31:03,895::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:31:04,447::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0904328 s, 11.3 MB/s\n'; <rc> = 0 >Thread-15914395::DEBUG::2013-11-20 03:31:04,514::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914395::INFO::2013-11-20 03:31:04,515::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914395::DEBUG::2013-11-20 03:31:04,516::task::495::TaskManager.Task::(_debug) Task 76b43534-5ac5-4f0b-abcb-b56f9121e169: moving from state init -> state preparing >Thread-15914395::DEBUG::2013-11-20 03:31:04,517::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914395::DEBUG::2013-11-20 03:31:04,517::task::495::TaskManager.Task::(_debug) Task 76b43534-5ac5-4f0b-abcb-b56f9121e169: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914395::DEBUG::2013-11-20 03:31:04,518::task::495::TaskManager.Task::(_debug) Task 76b43534-5ac5-4f0b-abcb-b56f9121e169: moving from state preparing -> state finished >Thread-15914395::DEBUG::2013-11-20 03:31:04,519::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914395::DEBUG::2013-11-20 03:31:04,519::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914395::DEBUG::2013-11-20 03:31:04,520::task::495::TaskManager.Task::(_debug) Task 76b43534-5ac5-4f0b-abcb-b56f9121e169: ref 0 aborting False >Thread-15914395::INFO::2013-11-20 03:31:04,520::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914396::DEBUG::2013-11-20 03:31:04,530::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914396::INFO::2013-11-20 03:31:04,531::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914396::DEBUG::2013-11-20 03:31:04,532::task::495::TaskManager.Task::(_debug) Task 8a0577ee-9c97-4aa9-ae2b-204a9feeb69c: moving from state init -> state preparing >Thread-15914396::DEBUG::2013-11-20 03:31:04,533::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5783af83-e659-4682-9666-4d0c1d8a602b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914396::DEBUG::2013-11-20 03:31:04,534::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914396::DEBUG::2013-11-20 03:31:04,534::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914396::DEBUG::2013-11-20 03:31:04,535::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5783af83-e659-4682-9666-4d0c1d8a602b`::Granted request >Thread-15914396::DEBUG::2013-11-20 03:31:04,536::task::495::TaskManager.Task::(_debug) Task 8a0577ee-9c97-4aa9-ae2b-204a9feeb69c: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914396::DEBUG::2013-11-20 03:31:04,537::task::495::TaskManager.Task::(_debug) Task 8a0577ee-9c97-4aa9-ae2b-204a9feeb69c: ref 1 aborting False >Thread-15914396::WARNING::2013-11-20 03:31:04,539::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914396::DEBUG::2013-11-20 03:31:04,540::task::495::TaskManager.Task::(_debug) Task 8a0577ee-9c97-4aa9-ae2b-204a9feeb69c: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914396::DEBUG::2013-11-20 03:31:04,540::task::495::TaskManager.Task::(_debug) Task 8a0577ee-9c97-4aa9-ae2b-204a9feeb69c: moving from state preparing -> state finished >Thread-15914396::DEBUG::2013-11-20 03:31:04,541::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914396::DEBUG::2013-11-20 03:31:04,542::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914396::DEBUG::2013-11-20 03:31:04,542::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914396::DEBUG::2013-11-20 03:31:04,543::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914396::DEBUG::2013-11-20 03:31:04,543::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914396::DEBUG::2013-11-20 03:31:04,544::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914396::DEBUG::2013-11-20 03:31:04,545::task::495::TaskManager.Task::(_debug) Task 8a0577ee-9c97-4aa9-ae2b-204a9feeb69c: ref 0 aborting False >Thread-15914396::INFO::2013-11-20 03:31:04,545::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913518::INFO::2013-11-20 03:31:05,875::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:31:05,876::task::495::TaskManager.Task::(_debug) Task 36e4eed0-d706-4d80-82cc-5e071ddb6a33: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:31:05,878::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`82107db2-854b-4f8d-b0d4-30f8e360448a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:31:05,879::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:31:05,880::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:31:05,881::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`82107db2-854b-4f8d-b0d4-30f8e360448a`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:31:05,882::task::495::TaskManager.Task::(_debug) Task 36e4eed0-d706-4d80-82cc-5e071ddb6a33: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:31:05,883::task::495::TaskManager.Task::(_debug) Task 36e4eed0-d706-4d80-82cc-5e071ddb6a33: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:31:05,884::task::495::TaskManager.Task::(_debug) Task 36e4eed0-d706-4d80-82cc-5e071ddb6a33: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:31:05,885::task::495::TaskManager.Task::(_debug) Task 36e4eed0-d706-4d80-82cc-5e071ddb6a33: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:31:05,886::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:31:05,887::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:31:05,888::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:31:05,889::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:31:05,889::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:31:05,890::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:31:05,891::task::495::TaskManager.Task::(_debug) Task 36e4eed0-d706-4d80-82cc-5e071ddb6a33: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:31:05,892::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:31:05,893::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:31:05,894::task::495::TaskManager.Task::(_debug) Task 7ec9d58e-7625-4e53-ae30-90398a839b5d: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:31:05,895::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`bff64e7c-799e-446f-922d-6b412b3bf1df`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:31:05,896::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:31:05,897::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:31:05,898::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`bff64e7c-799e-446f-922d-6b412b3bf1df`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:31:05,899::task::495::TaskManager.Task::(_debug) Task 7ec9d58e-7625-4e53-ae30-90398a839b5d: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:31:05,899::task::495::TaskManager.Task::(_debug) Task 7ec9d58e-7625-4e53-ae30-90398a839b5d: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:31:05,900::task::495::TaskManager.Task::(_debug) Task 7ec9d58e-7625-4e53-ae30-90398a839b5d: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:31:05,900::task::495::TaskManager.Task::(_debug) Task 7ec9d58e-7625-4e53-ae30-90398a839b5d: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:31:05,901::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:31:05,902::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:31:05,903::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:31:05,904::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:31:05,905::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:31:05,905::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:31:05,906::task::495::TaskManager.Task::(_debug) Task 7ec9d58e-7625-4e53-ae30-90398a839b5d: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:31:05,907::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:31:06,463::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:31:06,975::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0792617 s, 12.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:31:08,988::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:31:09,509::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0779901 s, 13.1 MB/s\n'; <rc> = 0 >Thread-15914399::DEBUG::2013-11-20 03:31:10,212::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15914399::DEBUG::2013-11-20 03:31:10,828::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Thread-15914400::DEBUG::2013-11-20 03:31:10,835::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15914400::DEBUG::2013-11-20 03:31:11,448::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Dummy-15912651::DEBUG::2013-11-20 03:31:11,524::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:31:12,039::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0571554 s, 17.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:31:14,052::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:31:14,605::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0852773 s, 12.0 MB/s\n'; <rc> = 0 >Thread-15914402::DEBUG::2013-11-20 03:31:14,712::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914402::INFO::2013-11-20 03:31:14,714::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914402::DEBUG::2013-11-20 03:31:14,715::task::495::TaskManager.Task::(_debug) Task c88f22b0-eab2-4208-b526-77e1c3eac0d5: moving from state init -> state preparing >Thread-15914402::DEBUG::2013-11-20 03:31:14,716::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914402::DEBUG::2013-11-20 03:31:14,717::task::495::TaskManager.Task::(_debug) Task c88f22b0-eab2-4208-b526-77e1c3eac0d5: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914402::DEBUG::2013-11-20 03:31:14,717::task::495::TaskManager.Task::(_debug) Task c88f22b0-eab2-4208-b526-77e1c3eac0d5: moving from state preparing -> state finished >Thread-15914402::DEBUG::2013-11-20 03:31:14,718::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914402::DEBUG::2013-11-20 03:31:14,719::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914402::DEBUG::2013-11-20 03:31:14,719::task::495::TaskManager.Task::(_debug) Task c88f22b0-eab2-4208-b526-77e1c3eac0d5: ref 0 aborting False >Thread-15914402::INFO::2013-11-20 03:31:14,720::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914403::DEBUG::2013-11-20 03:31:14,725::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914403::INFO::2013-11-20 03:31:14,726::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914403::DEBUG::2013-11-20 03:31:14,727::task::495::TaskManager.Task::(_debug) Task 07ca339b-94d5-48aa-8beb-ec8dac5ded0e: moving from state init -> state preparing >Thread-15914403::DEBUG::2013-11-20 03:31:14,728::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b9d3b10f-f4e3-4eab-ac2b-90d61d94dadd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914403::DEBUG::2013-11-20 03:31:14,729::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914403::DEBUG::2013-11-20 03:31:14,729::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914403::DEBUG::2013-11-20 03:31:14,730::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b9d3b10f-f4e3-4eab-ac2b-90d61d94dadd`::Granted request >Thread-15914403::DEBUG::2013-11-20 03:31:14,731::task::495::TaskManager.Task::(_debug) Task 07ca339b-94d5-48aa-8beb-ec8dac5ded0e: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914403::DEBUG::2013-11-20 03:31:14,732::task::495::TaskManager.Task::(_debug) Task 07ca339b-94d5-48aa-8beb-ec8dac5ded0e: ref 1 aborting False >Thread-15914403::WARNING::2013-11-20 03:31:14,734::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914403::DEBUG::2013-11-20 03:31:14,741::task::495::TaskManager.Task::(_debug) Task 07ca339b-94d5-48aa-8beb-ec8dac5ded0e: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914403::DEBUG::2013-11-20 03:31:14,743::task::495::TaskManager.Task::(_debug) Task 07ca339b-94d5-48aa-8beb-ec8dac5ded0e: moving from state preparing -> state finished >Thread-15914403::DEBUG::2013-11-20 03:31:14,744::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914403::DEBUG::2013-11-20 03:31:14,746::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914403::DEBUG::2013-11-20 03:31:14,746::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914403::DEBUG::2013-11-20 03:31:14,747::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914403::DEBUG::2013-11-20 03:31:14,748::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914403::DEBUG::2013-11-20 03:31:14,749::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914403::DEBUG::2013-11-20 03:31:14,751::task::495::TaskManager.Task::(_debug) Task 07ca339b-94d5-48aa-8beb-ec8dac5ded0e: ref 0 aborting False >Thread-15914403::INFO::2013-11-20 03:31:14,754::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:31:16,620::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:31:17,194::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0844752 s, 12.1 MB/s\n'; <rc> = 0 >Thread-15914405::INFO::2013-11-20 03:31:17,803::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914405::DEBUG::2013-11-20 03:31:17,804::task::495::TaskManager.Task::(_debug) Task 95d5a659-f938-4620-80f5-e6b5418c5054: moving from state init -> state preparing >Thread-15914405::DEBUG::2013-11-20 03:31:17,805::task::495::TaskManager.Task::(_debug) Task 95d5a659-f938-4620-80f5-e6b5418c5054: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000521898269653', 'lastCheck': 1384918276.8574729, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0127351284027', 'lastCheck': 1384918272.8105929, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0097188949585', 'lastCheck': 1384918275.454607, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000421047210693', 'lastCheck': 1384918275.7921989, 'valid': True, 'code': 0}} >Thread-15914405::DEBUG::2013-11-20 03:31:17,807::task::495::TaskManager.Task::(_debug) Task 95d5a659-f938-4620-80f5-e6b5418c5054: moving from state preparing -> state finished >Thread-15914405::DEBUG::2013-11-20 03:31:17,815::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914405::DEBUG::2013-11-20 03:31:17,816::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914405::DEBUG::2013-11-20 03:31:17,817::task::495::TaskManager.Task::(_debug) Task 95d5a659-f938-4620-80f5-e6b5418c5054: ref 0 aborting False >Thread-15914405::INFO::2013-11-20 03:31:17,818::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000521898269653', 'lastCheck': 1384918276.8574729, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0127351284027', 'lastCheck': 1384918272.8105929, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0097188949585', 'lastCheck': 1384918275.454607, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000421047210693', 'lastCheck': 1384918275.7921989, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:31:19,210::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:31:19,757::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0721018 s, 14.2 MB/s\n'; <rc> = 0 >Thread-15913546::INFO::2013-11-20 03:31:20,283::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:31:20,286::task::495::TaskManager.Task::(_debug) Task d37c81d3-a870-4c46-b416-444eead4d7ae: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:31:20,291::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`7a9facd1-543f-4b51-8b5c-14d95f2a2736`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:31:20,295::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:31:20,297::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:31:20,298::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`7a9facd1-543f-4b51-8b5c-14d95f2a2736`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:31:20,300::task::495::TaskManager.Task::(_debug) Task d37c81d3-a870-4c46-b416-444eead4d7ae: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:31:20,300::task::495::TaskManager.Task::(_debug) Task d37c81d3-a870-4c46-b416-444eead4d7ae: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:31:20,302::task::495::TaskManager.Task::(_debug) Task d37c81d3-a870-4c46-b416-444eead4d7ae: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:31:20,302::task::495::TaskManager.Task::(_debug) Task d37c81d3-a870-4c46-b416-444eead4d7ae: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:31:20,303::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:31:20,304::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:31:20,304::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:31:20,305::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:31:20,306::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:31:20,306::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:31:20,307::task::495::TaskManager.Task::(_debug) Task d37c81d3-a870-4c46-b416-444eead4d7ae: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:31:20,308::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:31:20,308::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:31:20,309::task::495::TaskManager.Task::(_debug) Task 0753ebfe-d71f-48b0-83d7-9033e66b6e20: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:31:20,310::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`3f2efdc6-c4e9-4172-acb4-f28c2ba23ee2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:31:20,311::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:31:20,311::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:31:20,312::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`3f2efdc6-c4e9-4172-acb4-f28c2ba23ee2`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:31:20,313::task::495::TaskManager.Task::(_debug) Task 0753ebfe-d71f-48b0-83d7-9033e66b6e20: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:31:20,314::task::495::TaskManager.Task::(_debug) Task 0753ebfe-d71f-48b0-83d7-9033e66b6e20: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:31:20,314::task::495::TaskManager.Task::(_debug) Task 0753ebfe-d71f-48b0-83d7-9033e66b6e20: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:31:20,315::task::495::TaskManager.Task::(_debug) Task 0753ebfe-d71f-48b0-83d7-9033e66b6e20: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:31:20,315::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:31:20,316::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:31:20,317::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:31:20,318::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:31:20,318::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:31:20,319::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:31:20,320::task::495::TaskManager.Task::(_debug) Task 0753ebfe-d71f-48b0-83d7-9033e66b6e20: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:31:20,320::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:31:20,321::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:31:20,321::task::495::TaskManager.Task::(_debug) Task f1fcc48c-9ff3-41d4-a844-16ac6b780775: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:31:20,323::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`71f003b4-6b57-4a6c-bb7a-f4f3291429c3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:31:20,323::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:31:20,324::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:31:20,325::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`71f003b4-6b57-4a6c-bb7a-f4f3291429c3`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:31:20,326::task::495::TaskManager.Task::(_debug) Task f1fcc48c-9ff3-41d4-a844-16ac6b780775: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:31:20,327::task::495::TaskManager.Task::(_debug) Task f1fcc48c-9ff3-41d4-a844-16ac6b780775: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:31:20,328::task::495::TaskManager.Task::(_debug) Task f1fcc48c-9ff3-41d4-a844-16ac6b780775: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:31:20,329::task::495::TaskManager.Task::(_debug) Task f1fcc48c-9ff3-41d4-a844-16ac6b780775: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:31:20,330::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:31:20,331::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:31:20,331::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:31:20,332::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:31:20,333::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:31:20,333::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:31:20,334::task::495::TaskManager.Task::(_debug) Task f1fcc48c-9ff3-41d4-a844-16ac6b780775: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:31:20,335::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:31:20,335::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:31:20,336::task::495::TaskManager.Task::(_debug) Task 7d2672b2-64a3-4dd6-b31c-f7a1f406be3a: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:31:20,337::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`a9af02ab-a1a0-4ee1-9386-530ab174f8d7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:31:20,338::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:31:20,339::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:31:20,339::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`a9af02ab-a1a0-4ee1-9386-530ab174f8d7`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:31:20,340::task::495::TaskManager.Task::(_debug) Task 7d2672b2-64a3-4dd6-b31c-f7a1f406be3a: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:31:20,340::task::495::TaskManager.Task::(_debug) Task 7d2672b2-64a3-4dd6-b31c-f7a1f406be3a: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:31:20,341::task::495::TaskManager.Task::(_debug) Task 7d2672b2-64a3-4dd6-b31c-f7a1f406be3a: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:31:20,342::task::495::TaskManager.Task::(_debug) Task 7d2672b2-64a3-4dd6-b31c-f7a1f406be3a: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:31:20,342::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:31:20,343::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:31:20,343::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:31:20,344::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:31:20,344::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:31:20,345::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:31:20,346::task::495::TaskManager.Task::(_debug) Task 7d2672b2-64a3-4dd6-b31c-f7a1f406be3a: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:31:20,346::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15914408::DEBUG::2013-11-20 03:31:21,523::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Dummy-15912651::DEBUG::2013-11-20 03:31:21,779::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914408::DEBUG::2013-11-20 03:31:22,275::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Thread-15914409::DEBUG::2013-11-20 03:31:22,282::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Dummy-15912651::DEBUG::2013-11-20 03:31:22,594::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.283409 s, 3.6 MB/s\n'; <rc> = 0 >Thread-15914409::DEBUG::2013-11-20 03:31:22,893::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Dummy-15912651::DEBUG::2013-11-20 03:31:24,612::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914411::DEBUG::2013-11-20 03:31:24,867::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914411::INFO::2013-11-20 03:31:24,869::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914411::DEBUG::2013-11-20 03:31:24,869::task::495::TaskManager.Task::(_debug) Task c3667b52-cf0b-40dd-be6f-a5fbf93ea9f5: moving from state init -> state preparing >Thread-15914411::DEBUG::2013-11-20 03:31:24,870::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914411::DEBUG::2013-11-20 03:31:24,871::task::495::TaskManager.Task::(_debug) Task c3667b52-cf0b-40dd-be6f-a5fbf93ea9f5: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914411::DEBUG::2013-11-20 03:31:24,871::task::495::TaskManager.Task::(_debug) Task c3667b52-cf0b-40dd-be6f-a5fbf93ea9f5: moving from state preparing -> state finished >Thread-15914411::DEBUG::2013-11-20 03:31:24,872::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914411::DEBUG::2013-11-20 03:31:24,872::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914411::DEBUG::2013-11-20 03:31:24,873::task::495::TaskManager.Task::(_debug) Task c3667b52-cf0b-40dd-be6f-a5fbf93ea9f5: ref 0 aborting False >Thread-15914411::INFO::2013-11-20 03:31:24,874::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914412::DEBUG::2013-11-20 03:31:24,879::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914412::INFO::2013-11-20 03:31:24,880::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914412::DEBUG::2013-11-20 03:31:24,880::task::495::TaskManager.Task::(_debug) Task a697726f-7163-4ace-a82d-7f402f9607cb: moving from state init -> state preparing >Thread-15914412::DEBUG::2013-11-20 03:31:24,882::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6bc2d0a8-ab0f-4a09-abc7-9754e3c5bd64`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914412::DEBUG::2013-11-20 03:31:24,883::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914412::DEBUG::2013-11-20 03:31:24,884::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914412::DEBUG::2013-11-20 03:31:24,884::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6bc2d0a8-ab0f-4a09-abc7-9754e3c5bd64`::Granted request >Thread-15914412::DEBUG::2013-11-20 03:31:24,885::task::495::TaskManager.Task::(_debug) Task a697726f-7163-4ace-a82d-7f402f9607cb: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914412::DEBUG::2013-11-20 03:31:24,886::task::495::TaskManager.Task::(_debug) Task a697726f-7163-4ace-a82d-7f402f9607cb: ref 1 aborting False >Thread-15914412::WARNING::2013-11-20 03:31:24,888::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914412::DEBUG::2013-11-20 03:31:24,889::task::495::TaskManager.Task::(_debug) Task a697726f-7163-4ace-a82d-7f402f9607cb: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914412::DEBUG::2013-11-20 03:31:24,890::task::495::TaskManager.Task::(_debug) Task a697726f-7163-4ace-a82d-7f402f9607cb: moving from state preparing -> state finished >Thread-15914412::DEBUG::2013-11-20 03:31:24,891::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914412::DEBUG::2013-11-20 03:31:24,892::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914412::DEBUG::2013-11-20 03:31:24,892::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914412::DEBUG::2013-11-20 03:31:24,893::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914412::DEBUG::2013-11-20 03:31:24,894::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914412::DEBUG::2013-11-20 03:31:24,894::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914412::DEBUG::2013-11-20 03:31:24,895::task::495::TaskManager.Task::(_debug) Task a697726f-7163-4ace-a82d-7f402f9607cb: ref 0 aborting False >Thread-15914412::INFO::2013-11-20 03:31:24,896::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:31:25,140::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0860026 s, 11.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:31:27,157::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:31:27,651::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0806644 s, 12.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:31:29,667::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:31:30,226::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0848797 s, 12.1 MB/s\n'; <rc> = 0 >Thread-15913639::INFO::2013-11-20 03:31:30,479::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:31:30,490::task::495::TaskManager.Task::(_debug) Task 83c728ab-7e58-4c9f-9965-abe38c84cd94: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:31:30,493::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`da5d4a27-ed0d-4ab1-95a4-03435da54d65`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:31:30,494::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:31:30,495::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:31:30,496::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`da5d4a27-ed0d-4ab1-95a4-03435da54d65`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:31:30,497::task::495::TaskManager.Task::(_debug) Task 83c728ab-7e58-4c9f-9965-abe38c84cd94: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:31:30,497::task::495::TaskManager.Task::(_debug) Task 83c728ab-7e58-4c9f-9965-abe38c84cd94: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:31:30,498::task::495::TaskManager.Task::(_debug) Task 83c728ab-7e58-4c9f-9965-abe38c84cd94: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:31:30,499::task::495::TaskManager.Task::(_debug) Task 83c728ab-7e58-4c9f-9965-abe38c84cd94: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:31:30,500::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:31:30,500::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:31:30,501::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:31:30,502::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:31:30,502::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:31:30,503::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:31:30,503::task::495::TaskManager.Task::(_debug) Task 83c728ab-7e58-4c9f-9965-abe38c84cd94: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:31:30,504::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:31:32,240::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:31:32,700::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0622011 s, 16.5 MB/s\n'; <rc> = 0 >Thread-15914415::DEBUG::2013-11-20 03:31:32,928::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15914416::INFO::2013-11-20 03:31:33,134::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914416::DEBUG::2013-11-20 03:31:33,149::task::495::TaskManager.Task::(_debug) Task c8191942-d32d-4074-938d-c3c4c2f5db43: moving from state init -> state preparing >Thread-15914416::DEBUG::2013-11-20 03:31:33,150::task::495::TaskManager.Task::(_debug) Task c8191942-d32d-4074-938d-c3c4c2f5db43: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000313997268677', 'lastCheck': 1384918287.065321, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00115394592285', 'lastCheck': 1384918293.1484239, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00141477584839', 'lastCheck': 1384918285.4756269, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000412940979004', 'lastCheck': 1384918286.056818, 'valid': True, 'code': 0}} >Thread-15914416::DEBUG::2013-11-20 03:31:33,150::task::495::TaskManager.Task::(_debug) Task c8191942-d32d-4074-938d-c3c4c2f5db43: moving from state preparing -> state finished >Thread-15914416::DEBUG::2013-11-20 03:31:33,151::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914416::DEBUG::2013-11-20 03:31:33,151::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914416::DEBUG::2013-11-20 03:31:33,151::task::495::TaskManager.Task::(_debug) Task c8191942-d32d-4074-938d-c3c4c2f5db43: ref 0 aborting False >Thread-15914416::INFO::2013-11-20 03:31:33,152::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000313997268677', 'lastCheck': 1384918287.065321, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00115394592285', 'lastCheck': 1384918293.1484239, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00141477584839', 'lastCheck': 1384918285.4756269, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000412940979004', 'lastCheck': 1384918286.056818, 'valid': True, 'code': 0}} >Thread-15914415::DEBUG::2013-11-20 03:31:33,479::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Thread-15914418::DEBUG::2013-11-20 03:31:33,485::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15914418::DEBUG::2013-11-20 03:31:34,088::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Dummy-15912651::DEBUG::2013-11-20 03:31:34,715::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914419::DEBUG::2013-11-20 03:31:35,033::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914419::INFO::2013-11-20 03:31:35,035::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914419::DEBUG::2013-11-20 03:31:35,037::task::495::TaskManager.Task::(_debug) Task 5dd916a2-66af-4603-9b68-3a3e8e6ebfad: moving from state init -> state preparing >Thread-15914419::DEBUG::2013-11-20 03:31:35,039::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914419::DEBUG::2013-11-20 03:31:35,044::task::495::TaskManager.Task::(_debug) Task 5dd916a2-66af-4603-9b68-3a3e8e6ebfad: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914419::DEBUG::2013-11-20 03:31:35,048::task::495::TaskManager.Task::(_debug) Task 5dd916a2-66af-4603-9b68-3a3e8e6ebfad: moving from state preparing -> state finished >Thread-15914419::DEBUG::2013-11-20 03:31:35,053::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914419::DEBUG::2013-11-20 03:31:35,055::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914419::DEBUG::2013-11-20 03:31:35,056::task::495::TaskManager.Task::(_debug) Task 5dd916a2-66af-4603-9b68-3a3e8e6ebfad: ref 0 aborting False >Thread-15914419::INFO::2013-11-20 03:31:35,056::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914420::DEBUG::2013-11-20 03:31:35,062::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914420::INFO::2013-11-20 03:31:35,063::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914420::DEBUG::2013-11-20 03:31:35,063::task::495::TaskManager.Task::(_debug) Task 1eac4fe0-7fd8-44d3-9a52-5977605e4eda: moving from state init -> state preparing >Thread-15914420::DEBUG::2013-11-20 03:31:35,065::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`22721561-ca8b-40ce-8abe-aab501dd37a6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914420::DEBUG::2013-11-20 03:31:35,065::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914420::DEBUG::2013-11-20 03:31:35,066::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914420::DEBUG::2013-11-20 03:31:35,067::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`22721561-ca8b-40ce-8abe-aab501dd37a6`::Granted request >Thread-15914420::DEBUG::2013-11-20 03:31:35,068::task::495::TaskManager.Task::(_debug) Task 1eac4fe0-7fd8-44d3-9a52-5977605e4eda: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914420::DEBUG::2013-11-20 03:31:35,069::task::495::TaskManager.Task::(_debug) Task 1eac4fe0-7fd8-44d3-9a52-5977605e4eda: ref 1 aborting False >Thread-15914420::WARNING::2013-11-20 03:31:35,071::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914420::DEBUG::2013-11-20 03:31:35,073::task::495::TaskManager.Task::(_debug) Task 1eac4fe0-7fd8-44d3-9a52-5977605e4eda: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914420::DEBUG::2013-11-20 03:31:35,074::task::495::TaskManager.Task::(_debug) Task 1eac4fe0-7fd8-44d3-9a52-5977605e4eda: moving from state preparing -> state finished >Thread-15914420::DEBUG::2013-11-20 03:31:35,074::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914420::DEBUG::2013-11-20 03:31:35,075::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914420::DEBUG::2013-11-20 03:31:35,076::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914420::DEBUG::2013-11-20 03:31:35,076::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914420::DEBUG::2013-11-20 03:31:35,077::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914420::DEBUG::2013-11-20 03:31:35,078::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914420::DEBUG::2013-11-20 03:31:35,078::task::495::TaskManager.Task::(_debug) Task 1eac4fe0-7fd8-44d3-9a52-5977605e4eda: ref 0 aborting False >Thread-15914420::INFO::2013-11-20 03:31:35,079::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115215433728', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:31:35,216::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0540513 s, 18.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:31:37,227::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:31:37,771::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0746346 s, 13.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:31:39,787::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:31:40,331::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0814821 s, 12.6 MB/s\n'; <rc> = 0 >Thread-15914423::DEBUG::2013-11-20 03:31:42,341::clientIF::54::vds::(wrapper) [192.168.100.135]::call create with ({'bridge': 'rhevm', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.2.0', 'vmId': '5f587d5c-d4b4-4242-b049-ede331c7f8b4', 'transparentHugePages': 'true', 'spiceSslCipherSuite': 'DEFAULT', 'cpuType': 'Westmere', 'smp': '2', 'macAddr': '00:1a:4a:a8:64:17', 'boot': 'd', 'custom': {}, 'vmType': 'kvm', 'memSize': 2048, 'smpCoresPerSocket': '1', 'vmName': 'Taipei', 'spiceMonitors': '1', 'nice': '0', 'drives': [{'index': '0', 'domainID': '6b892067-ac7e-4039-a7be-5b7387525b16', 'format': 'cow', 'boot': 'true', 'volumeID': 'd7028f7c-0551-4bbe-a907-6f804cc9bd54', 'imageID': '143cb26d-de08-4d30-89cf-aee10245b66a', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}], 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'smartcardEnable': 'false', 'nicModel': 'pv', 'keyboardLayout': 'en-us', 'kvmEnable': 'true', 'timeOffset': '25201', 'spiceSecureChannels': 'smain,sinputs', 'display': 'qxl'},) {} >Dummy-15912651::DEBUG::2013-11-20 03:31:42,538::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914423::INFO::2013-11-20 03:31:42,872::clientIF::523::vds::(_getNetworkIp) network None: using 0 >Thread-15914423::INFO::2013-11-20 03:31:42,874::clientIF::656::vds::(create) vmContainerLock acquired by vm 5f587d5c-d4b4-4242-b049-ede331c7f8b4 >Thread-15914424::DEBUG::2013-11-20 03:31:42,881::vm::342::vm.Vm::(_startUnderlyingVm) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Start >Thread-15914423::DEBUG::2013-11-20 03:31:42,881::clientIF::669::vds::(create) Total desktops after creation of 5f587d5c-d4b4-4242-b049-ede331c7f8b4 is 7 >Thread-15914424::DEBUG::2013-11-20 03:31:42,882::vm::346::vm.Vm::(_startUnderlyingVm) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::_ongoingCreations acquired >Thread-15914423::DEBUG::2013-11-20 03:31:42,883::clientIF::59::vds::(wrapper) return create with {'status': {'message': 'Done', 'code': 0}, 'vmList': {'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'bridge': 'rhevm', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.2.0', 'vmId': '5f587d5c-d4b4-4242-b049-ede331c7f8b4', 'pid': '0', 'drives': [{'index': '0', 'domainID': '6b892067-ac7e-4039-a7be-5b7387525b16', 'format': 'cow', 'boot': 'true', 'volumeID': 'd7028f7c-0551-4bbe-a907-6f804cc9bd54', 'imageID': '143cb26d-de08-4d30-89cf-aee10245b66a', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'virtio'}], 'transparentHugePages': 'true', 'keyboardLayout': 'en-us', 'displaySecurePort': '-1', 'displayPort': '-1', 'cpuType': 'Westmere', 'custom': {}, 'clientIp': '', 'nicModel': 'pv', 'macAddr': '00:1a:4a:a8:64:17', 'kvmEnable': 'true', 'smartcardEnable': 'false', 'boot': 'd', 'smp': '2', 'vmType': 'kvm', 'spiceSslCipherSuite': 'DEFAULT', 'status': 'WaitForLaunch', 'timeOffset': '25201', 'memSize': 2048, 'displayIp': '0', 'spiceSecureChannels': 'smain,sinputs', 'smpCoresPerSocket': '1', 'vmName': 'Taipei', 'display': 'qxl', 'spiceMonitors': '1', 'nice': '0'}} >Thread-15914424::INFO::2013-11-20 03:31:42,884::libvirtvm::1061::vm.Vm::(_run) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::VM wrapper has started >Thread-15914424::INFO::2013-11-20 03:31:42,887::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=143cb26d-de08-4d30-89cf-aee10245b66a volUUID=d7028f7c-0551-4bbe-a907-6f804cc9bd54) >Thread-15914424::DEBUG::2013-11-20 03:31:42,888::task::495::TaskManager.Task::(_debug) Task 5519e9ba-e7d4-4d63-9d74-6e4597e59b92: moving from state init -> state preparing >Thread-15914424::DEBUG::2013-11-20 03:31:42,891::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d49132b5-dd02-4912-9add-49eba2780a32`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1690' at 'public_prepareVolume' >Thread-15914424::DEBUG::2013-11-20 03:31:42,892::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914424::DEBUG::2013-11-20 03:31:42,895::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914424::DEBUG::2013-11-20 03:31:42,895::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d49132b5-dd02-4912-9add-49eba2780a32`::Granted request >Thread-15914424::DEBUG::2013-11-20 03:31:42,896::task::495::TaskManager.Task::(_debug) Task 5519e9ba-e7d4-4d63-9d74-6e4597e59b92: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914424::DEBUG::2013-11-20 03:31:42,897::task::495::TaskManager.Task::(_debug) Task 5519e9ba-e7d4-4d63-9d74-6e4597e59b92: ref 1 aborting False >Thread-15914424::DEBUG::2013-11-20 03:31:42,898::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.143cb26d-de08-4d30-89cf-aee10245b66a`ReqID=`5ecd9ec3-f997-422a-9d33-40c5ce7f3568`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1695' at 'public_prepareVolume' >Thread-15914424::DEBUG::2013-11-20 03:31:42,899::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.143cb26d-de08-4d30-89cf-aee10245b66a' for lock type 'exclusive' >Thread-15914424::DEBUG::2013-11-20 03:31:42,900::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15914424::DEBUG::2013-11-20 03:31:42,902::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:31:43,397::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.166475 s, 6.2 MB/s\n'; <rc> = 0 >Thread-15914424::DEBUG::2013-11-20 03:31:44,098::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914427::DEBUG::2013-11-20 03:31:44,348::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15914424::DEBUG::2013-11-20 03:31:44,350::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15914424::DEBUG::2013-11-20 03:31:44,562::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=14 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15914424::DEBUG::2013-11-20 03:31:45,105::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000177243 s, 2.9 MB/s\n'; <rc> = 0 >Thread-15914424::DEBUG::2013-11-20 03:31:45,106::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000177243 s, 2.9 MB/s'], size: 512 >Thread-15914424::DEBUG::2013-11-20 03:31:45,107::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=14 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15914427::DEBUG::2013-11-20 03:31:45,324::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Thread-15914428::DEBUG::2013-11-20 03:31:45,332::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914428::INFO::2013-11-20 03:31:45,334::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914428::DEBUG::2013-11-20 03:31:45,335::task::495::TaskManager.Task::(_debug) Task 6d58f57e-09a8-4031-9e58-d9ad3e7a7c54: moving from state init -> state preparing >Thread-15914428::DEBUG::2013-11-20 03:31:45,342::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914429::DEBUG::2013-11-20 03:31:45,357::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15914428::DEBUG::2013-11-20 03:31:45,358::task::495::TaskManager.Task::(_debug) Task 6d58f57e-09a8-4031-9e58-d9ad3e7a7c54: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914428::DEBUG::2013-11-20 03:31:45,360::task::495::TaskManager.Task::(_debug) Task 6d58f57e-09a8-4031-9e58-d9ad3e7a7c54: moving from state preparing -> state finished >Dummy-15912651::DEBUG::2013-11-20 03:31:45,675::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914428::DEBUG::2013-11-20 03:31:45,693::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914428::DEBUG::2013-11-20 03:31:45,923::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914428::DEBUG::2013-11-20 03:31:45,931::task::495::TaskManager.Task::(_debug) Task 6d58f57e-09a8-4031-9e58-d9ad3e7a7c54: ref 0 aborting False >Thread-15914428::INFO::2013-11-20 03:31:45,933::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914424::DEBUG::2013-11-20 03:31:45,935::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000194874 s, 2.6 MB/s\n'; <rc> = 0 >Thread-15914424::DEBUG::2013-11-20 03:31:45,941::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000194874 s, 2.6 MB/s'], size: 512 >Thread-15914424::INFO::2013-11-20 03:31:45,942::image::340::Storage.Image::(getChain) sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 imgUUID=143cb26d-de08-4d30-89cf-aee10245b66a chain=[<storage.blockVolume.BlockVolume instance at 0x7f48ac1a7638>] >Thread-15914424::DEBUG::2013-11-20 03:31:45,944::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --autobackup n --available y 6b892067-ac7e-4039-a7be-5b7387525b16/d7028f7c-0551-4bbe-a907-6f804cc9bd54' (cwd None) >Thread-15914429::DEBUG::2013-11-20 03:31:46,167::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Thread-15914430::DEBUG::2013-11-20 03:31:46,181::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914430::INFO::2013-11-20 03:31:46,184::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914430::DEBUG::2013-11-20 03:31:46,185::task::495::TaskManager.Task::(_debug) Task 9341f498-0d8f-4ef2-a20c-5a80d1a31c4b: moving from state init -> state preparing >Thread-15914430::DEBUG::2013-11-20 03:31:46,193::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a4145c34-75c9-47b4-a46a-0a1e74d2f39c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914430::DEBUG::2013-11-20 03:31:46,194::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914430::DEBUG::2013-11-20 03:31:46,196::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914430::DEBUG::2013-11-20 03:31:46,197::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a4145c34-75c9-47b4-a46a-0a1e74d2f39c`::Granted request >Thread-15914430::DEBUG::2013-11-20 03:31:46,199::task::495::TaskManager.Task::(_debug) Task 9341f498-0d8f-4ef2-a20c-5a80d1a31c4b: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914430::DEBUG::2013-11-20 03:31:46,200::task::495::TaskManager.Task::(_debug) Task 9341f498-0d8f-4ef2-a20c-5a80d1a31c4b: ref 1 aborting False >Thread-15914430::WARNING::2013-11-20 03:31:46,204::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914430::DEBUG::2013-11-20 03:31:46,206::task::495::TaskManager.Task::(_debug) Task 9341f498-0d8f-4ef2-a20c-5a80d1a31c4b: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914430::DEBUG::2013-11-20 03:31:46,207::task::495::TaskManager.Task::(_debug) Task 9341f498-0d8f-4ef2-a20c-5a80d1a31c4b: moving from state preparing -> state finished >Thread-15914430::DEBUG::2013-11-20 03:31:46,208::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914430::DEBUG::2013-11-20 03:31:46,209::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914430::DEBUG::2013-11-20 03:31:46,210::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914430::DEBUG::2013-11-20 03:31:46,212::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914430::DEBUG::2013-11-20 03:31:46,212::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914430::DEBUG::2013-11-20 03:31:46,214::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914430::DEBUG::2013-11-20 03:31:46,215::task::495::TaskManager.Task::(_debug) Task 9341f498-0d8f-4ef2-a20c-5a80d1a31c4b: ref 0 aborting False >Thread-15914430::INFO::2013-11-20 03:31:46,216::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:31:46,359::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.18917 s, 5.4 MB/s\n'; <rc> = 0 >Thread-15914424::DEBUG::2013-11-20 03:31:46,831::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914424::DEBUG::2013-11-20 03:31:46,840::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15914424::DEBUG::2013-11-20 03:31:46,841::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15914424::DEBUG::2013-11-20 03:31:46,842::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.d7028f7c-0551-4bbe-a907-6f804cc9bd54`ReqID=`e45cc610-c6bf-4fd3-a194-bc5cb0578e3d`::Request was made in '/usr/share/vdsm/storage/resourceFactories.py' line '149' at '__getResourceCandidatesList' >Thread-15914424::DEBUG::2013-11-20 03:31:46,842::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.d7028f7c-0551-4bbe-a907-6f804cc9bd54' for lock type 'exclusive' >Thread-15914424::DEBUG::2013-11-20 03:31:46,843::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.d7028f7c-0551-4bbe-a907-6f804cc9bd54' is free. Now locking as 'exclusive' (1 active user) >Thread-15914424::DEBUG::2013-11-20 03:31:46,844::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.d7028f7c-0551-4bbe-a907-6f804cc9bd54`ReqID=`e45cc610-c6bf-4fd3-a194-bc5cb0578e3d`::Granted request >Thread-15914424::DEBUG::2013-11-20 03:31:46,845::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.143cb26d-de08-4d30-89cf-aee10245b66a' is free. Now locking as 'exclusive' (1 active user) >Thread-15914424::DEBUG::2013-11-20 03:31:46,845::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.143cb26d-de08-4d30-89cf-aee10245b66a`ReqID=`5ecd9ec3-f997-422a-9d33-40c5ce7f3568`::Granted request >Thread-15914424::DEBUG::2013-11-20 03:31:46,847::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15914424::DEBUG::2013-11-20 03:31:46,848::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15914424::DEBUG::2013-11-20 03:31:47,862::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914424::DEBUG::2013-11-20 03:31:48,141::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15914424::DEBUG::2013-11-20 03:31:48,143::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --refresh 6b892067-ac7e-4039-a7be-5b7387525b16/d7028f7c-0551-4bbe-a907-6f804cc9bd54' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:31:48,372::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913695::INFO::2013-11-20 03:31:48,724::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:31:48,727::task::495::TaskManager.Task::(_debug) Task 52c4adbb-7281-45db-993d-60ba4e323a68: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:31:48,729::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c13c804b-3df4-473d-84c4-80a252739090`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:31:48,730::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:31:48,732::resourceManager::481::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' found in shared state and queue is empty, Joining current shared lock (2 active users) >Thread-15913695::DEBUG::2013-11-20 03:31:48,732::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c13c804b-3df4-473d-84c4-80a252739090`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:31:48,734::task::495::TaskManager.Task::(_debug) Task 52c4adbb-7281-45db-993d-60ba4e323a68: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:31:48,735::task::495::TaskManager.Task::(_debug) Task 52c4adbb-7281-45db-993d-60ba4e323a68: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:31:48,736::task::495::TaskManager.Task::(_debug) Task 52c4adbb-7281-45db-993d-60ba4e323a68: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:31:48,736::task::495::TaskManager.Task::(_debug) Task 52c4adbb-7281-45db-993d-60ba4e323a68: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:31:48,737::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:31:48,738::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:31:48,738::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:31:48,739::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (1 active users) >Thread-15913695::DEBUG::2013-11-20 03:31:48,740::task::495::TaskManager.Task::(_debug) Task 52c4adbb-7281-45db-993d-60ba4e323a68: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:31:48,741::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:31:49,023::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0691001 s, 14.8 MB/s\n'; <rc> = 0 >Thread-15914424::DEBUG::2013-11-20 03:31:49,084::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914424::DEBUG::2013-11-20 03:31:49,085::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15914424::DEBUG::2013-11-20 03:31:49,086::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15914424::INFO::2013-11-20 03:31:49,087::volume::517::Storage.Volume::(prepare) Volume: preparing volume 6b892067-ac7e-4039-a7be-5b7387525b16/d7028f7c-0551-4bbe-a907-6f804cc9bd54 >Thread-15914424::DEBUG::2013-11-20 03:31:49,087::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15914424::DEBUG::2013-11-20 03:31:49,089::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15914432::INFO::2013-11-20 03:31:49,355::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914432::DEBUG::2013-11-20 03:31:49,356::task::495::TaskManager.Task::(_debug) Task fee4631b-ee4c-46ec-9458-437a6bf8290a: moving from state init -> state preparing >Thread-15914432::DEBUG::2013-11-20 03:31:49,357::task::495::TaskManager.Task::(_debug) Task fee4631b-ee4c-46ec-9458-437a6bf8290a: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000355005264282', 'lastCheck': 1384918307.326561, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00123500823975', 'lastCheck': 1384918303.2946169, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.232423067093', 'lastCheck': 1384918305.928488, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000983953475952', 'lastCheck': 1384918306.324789, 'valid': True, 'code': 0}} >Thread-15914432::DEBUG::2013-11-20 03:31:49,358::task::495::TaskManager.Task::(_debug) Task fee4631b-ee4c-46ec-9458-437a6bf8290a: moving from state preparing -> state finished >Thread-15914432::DEBUG::2013-11-20 03:31:49,359::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914432::DEBUG::2013-11-20 03:31:49,360::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914432::DEBUG::2013-11-20 03:31:49,360::task::495::TaskManager.Task::(_debug) Task fee4631b-ee4c-46ec-9458-437a6bf8290a: ref 0 aborting False >Thread-15914432::INFO::2013-11-20 03:31:49,361::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000355005264282', 'lastCheck': 1384918307.326561, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00123500823975', 'lastCheck': 1384918303.2946169, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.232423067093', 'lastCheck': 1384918305.928488, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000983953475952', 'lastCheck': 1384918306.324789, 'valid': True, 'code': 0}} >Thread-15914424::DEBUG::2013-11-20 03:31:50,045::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914424::DEBUG::2013-11-20 03:31:50,318::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15914424::DEBUG::2013-11-20 03:31:50,319::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=14 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15914424::DEBUG::2013-11-20 03:31:50,687::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000160435 s, 3.2 MB/s\n'; <rc> = 0 >Thread-15914424::DEBUG::2013-11-20 03:31:50,689::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000160435 s, 3.2 MB/s'], size: 512 >Thread-15914424::DEBUG::2013-11-20 03:31:50,702::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=14 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:31:51,045::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913658::INFO::2013-11-20 03:31:51,046::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:31:51,049::task::495::TaskManager.Task::(_debug) Task 1a25a783-755d-4037-b177-c6ff2f517a99: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:31:51,276::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`4ec2bfd8-afc0-4d2a-b8a1-c000ce9793b4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:31:51,280::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:31:51,285::resourceManager::481::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' found in shared state and queue is empty, Joining current shared lock (2 active users) >Thread-15914424::DEBUG::2013-11-20 03:31:51,285::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000242659 s, 2.1 MB/s\n'; <rc> = 0 >Thread-15913658::DEBUG::2013-11-20 03:31:51,286::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`4ec2bfd8-afc0-4d2a-b8a1-c000ce9793b4`::Granted request >Thread-15914424::DEBUG::2013-11-20 03:31:51,288::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000242659 s, 2.1 MB/s'], size: 512 >Thread-15914424::DEBUG::2013-11-20 03:31:51,291::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.d7028f7c-0551-4bbe-a907-6f804cc9bd54`ReqID=`238d00d7-7ec0-4531-9380-600f51363cd6`::Request was made in '/usr/share/vdsm/storage/volume.py' line '533' at 'prepare' >Thread-15914424::DEBUG::2013-11-20 03:31:51,292::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.d7028f7c-0551-4bbe-a907-6f804cc9bd54' for lock type 'exclusive' >Thread-15913658::DEBUG::2013-11-20 03:31:51,293::task::495::TaskManager.Task::(_debug) Task 1a25a783-755d-4037-b177-c6ff2f517a99: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:31:51,295::task::495::TaskManager.Task::(_debug) Task 1a25a783-755d-4037-b177-c6ff2f517a99: ref 1 aborting False >Thread-15914424::DEBUG::2013-11-20 03:31:51,295::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.d7028f7c-0551-4bbe-a907-6f804cc9bd54' is free. Now locking as 'exclusive' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:31:51,296::task::495::TaskManager.Task::(_debug) Task 1a25a783-755d-4037-b177-c6ff2f517a99: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15914424::DEBUG::2013-11-20 03:31:51,297::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.d7028f7c-0551-4bbe-a907-6f804cc9bd54`ReqID=`238d00d7-7ec0-4531-9380-600f51363cd6`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:31:51,298::task::495::TaskManager.Task::(_debug) Task 1a25a783-755d-4037-b177-c6ff2f517a99: moving from state preparing -> state finished >Thread-15914424::DEBUG::2013-11-20 03:31:51,300::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=14 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15913658::DEBUG::2013-11-20 03:31:51,301::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:31:51,302::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:31:51,628::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:31:51,632::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (1 active users) >Thread-15913658::DEBUG::2013-11-20 03:31:51,634::task::495::TaskManager.Task::(_debug) Task 1a25a783-755d-4037-b177-c6ff2f517a99: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:31:51,635::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:31:51,795::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.165426 s, 6.2 MB/s\n'; <rc> = 0 >Thread-15914424::DEBUG::2013-11-20 03:31:51,835::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000196598 s, 2.6 MB/s\n'; <rc> = 0 >Thread-15914424::DEBUG::2013-11-20 03:31:51,837::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000196598 s, 2.6 MB/s'], size: 512 >Thread-15914424::DEBUG::2013-11-20 03:31:51,839::task::495::TaskManager.Task::(_debug) Task 5519e9ba-e7d4-4d63-9d74-6e4597e59b92: finished: None >Thread-15914424::DEBUG::2013-11-20 03:31:51,839::task::495::TaskManager.Task::(_debug) Task 5519e9ba-e7d4-4d63-9d74-6e4597e59b92: moving from state preparing -> state finished >Thread-15914424::DEBUG::2013-11-20 03:31:51,840::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914424::DEBUG::2013-11-20 03:31:51,841::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914424::DEBUG::2013-11-20 03:31:51,841::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914424::DEBUG::2013-11-20 03:31:51,842::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914424::DEBUG::2013-11-20 03:31:51,843::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914424::DEBUG::2013-11-20 03:31:51,843::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914424::DEBUG::2013-11-20 03:31:51,844::task::495::TaskManager.Task::(_debug) Task 5519e9ba-e7d4-4d63-9d74-6e4597e59b92: ref 0 aborting False >Thread-15914424::INFO::2013-11-20 03:31:51,845::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15914424::INFO::2013-11-20 03:31:51,846::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=143cb26d-de08-4d30-89cf-aee10245b66a volUUID=d7028f7c-0551-4bbe-a907-6f804cc9bd54) >Thread-15914424::DEBUG::2013-11-20 03:31:51,846::task::495::TaskManager.Task::(_debug) Task ef558dc3-c146-4080-9a5a-c1b22ffcadba: moving from state init -> state preparing >Thread-15914424::DEBUG::2013-11-20 03:31:51,848::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c39f7d48-4e4d-4383-be57-13f0011ac095`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1666' at 'public_getVolumePath' >Thread-15914424::DEBUG::2013-11-20 03:31:51,848::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914424::DEBUG::2013-11-20 03:31:51,849::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914424::DEBUG::2013-11-20 03:31:51,850::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c39f7d48-4e4d-4383-be57-13f0011ac095`::Granted request >Thread-15914424::DEBUG::2013-11-20 03:31:51,851::task::495::TaskManager.Task::(_debug) Task ef558dc3-c146-4080-9a5a-c1b22ffcadba: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914424::DEBUG::2013-11-20 03:31:51,851::task::495::TaskManager.Task::(_debug) Task ef558dc3-c146-4080-9a5a-c1b22ffcadba: ref 1 aborting False >Thread-15914424::DEBUG::2013-11-20 03:31:51,853::task::495::TaskManager.Task::(_debug) Task ef558dc3-c146-4080-9a5a-c1b22ffcadba: finished: {'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/143cb26d-de08-4d30-89cf-aee10245b66a/d7028f7c-0551-4bbe-a907-6f804cc9bd54'} >Thread-15914424::DEBUG::2013-11-20 03:31:51,853::task::495::TaskManager.Task::(_debug) Task ef558dc3-c146-4080-9a5a-c1b22ffcadba: moving from state preparing -> state finished >Thread-15914424::DEBUG::2013-11-20 03:31:51,854::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914424::DEBUG::2013-11-20 03:31:51,854::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914424::DEBUG::2013-11-20 03:31:51,855::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914424::DEBUG::2013-11-20 03:31:51,856::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914424::DEBUG::2013-11-20 03:31:51,856::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914424::DEBUG::2013-11-20 03:31:51,857::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914424::DEBUG::2013-11-20 03:31:51,857::task::495::TaskManager.Task::(_debug) Task ef558dc3-c146-4080-9a5a-c1b22ffcadba: ref 0 aborting False >Thread-15914424::INFO::2013-11-20 03:31:51,858::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, Return response: {'status': {'message': 'OK', 'code': 0}, 'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/143cb26d-de08-4d30-89cf-aee10245b66a/d7028f7c-0551-4bbe-a907-6f804cc9bd54'} >Thread-15914424::INFO::2013-11-20 03:31:51,860::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=143cb26d-de08-4d30-89cf-aee10245b66a volUUID=d7028f7c-0551-4bbe-a907-6f804cc9bd54) >Thread-15914424::DEBUG::2013-11-20 03:31:51,860::task::495::TaskManager.Task::(_debug) Task 2193c045-1ee5-4c10-8464-10bdaa4e0b2a: moving from state init -> state preparing >Thread-15914424::DEBUG::2013-11-20 03:31:51,862::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`826160ca-9ff5-4cb7-b4a0-ae0afa37c68e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914424::DEBUG::2013-11-20 03:31:51,862::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914424::DEBUG::2013-11-20 03:31:51,863::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914424::DEBUG::2013-11-20 03:31:51,864::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`826160ca-9ff5-4cb7-b4a0-ae0afa37c68e`::Granted request >Thread-15914424::DEBUG::2013-11-20 03:31:51,864::task::495::TaskManager.Task::(_debug) Task 2193c045-1ee5-4c10-8464-10bdaa4e0b2a: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914424::DEBUG::2013-11-20 03:31:51,865::task::495::TaskManager.Task::(_debug) Task 2193c045-1ee5-4c10-8464-10bdaa4e0b2a: ref 1 aborting False >Thread-15914424::DEBUG::2013-11-20 03:31:51,866::task::495::TaskManager.Task::(_debug) Task 2193c045-1ee5-4c10-8464-10bdaa4e0b2a: finished: {'truesize': '56908316672', 'apparentsize': '56908316672'} >Thread-15914424::DEBUG::2013-11-20 03:31:51,866::task::495::TaskManager.Task::(_debug) Task 2193c045-1ee5-4c10-8464-10bdaa4e0b2a: moving from state preparing -> state finished >Thread-15914424::DEBUG::2013-11-20 03:31:51,867::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914424::DEBUG::2013-11-20 03:31:51,867::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914424::DEBUG::2013-11-20 03:31:51,868::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914424::DEBUG::2013-11-20 03:31:51,869::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914424::DEBUG::2013-11-20 03:31:51,869::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914424::DEBUG::2013-11-20 03:31:51,870::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914424::DEBUG::2013-11-20 03:31:51,871::task::495::TaskManager.Task::(_debug) Task 2193c045-1ee5-4c10-8464-10bdaa4e0b2a: ref 0 aborting False >Thread-15914424::INFO::2013-11-20 03:31:51,871::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '56908316672', 'apparentsize': '56908316672'} >Thread-15914424::INFO::2013-11-20 03:31:51,872::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16) >Thread-15914424::DEBUG::2013-11-20 03:31:51,873::task::495::TaskManager.Task::(_debug) Task 657f9c47-bc40-467c-8d17-4d9e6212cbf6: moving from state init -> state preparing >Thread-15914424::INFO::2013-11-20 03:31:51,873::blockSD::674::Storage.StorageDomain::(validate) sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 >Thread-15914424::DEBUG::2013-11-20 03:31:51,874::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15914424::DEBUG::2013-11-20 03:31:51,875::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15914424::DEBUG::2013-11-20 03:31:51,875::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15914424::DEBUG::2013-11-20 03:31:51,877::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15914424::DEBUG::2013-11-20 03:31:51,877::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15914424::DEBUG::2013-11-20 03:31:51,879::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15914424::DEBUG::2013-11-20 03:31:52,521::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914424::DEBUG::2013-11-20 03:31:52,525::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15914424::DEBUG::2013-11-20 03:31:52,526::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'_SHA_CKSUM=716ea7492d9134a4b13de27ac61a336ad474727c', u'POOL_SPM_ID=2', u'POOL_SPM_LVER=287'] >Thread-15914424::DEBUG::2013-11-20 03:31:52,529::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f3340c23-9b6e-4a35-b978-248c4d65d03e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1297' at 'public_getStorageDomainInfo' >Thread-15914424::DEBUG::2013-11-20 03:31:52,530::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914424::DEBUG::2013-11-20 03:31:52,531::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914424::DEBUG::2013-11-20 03:31:52,531::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f3340c23-9b6e-4a35-b978-248c4d65d03e`::Granted request >Thread-15914424::DEBUG::2013-11-20 03:31:52,532::task::495::TaskManager.Task::(_debug) Task 657f9c47-bc40-467c-8d17-4d9e6212cbf6: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914424::DEBUG::2013-11-20 03:31:52,533::task::495::TaskManager.Task::(_debug) Task 657f9c47-bc40-467c-8d17-4d9e6212cbf6: ref 1 aborting False >Thread-15914424::WARNING::2013-11-20 03:31:52,536::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914424::DEBUG::2013-11-20 03:31:52,537::task::495::TaskManager.Task::(_debug) Task 657f9c47-bc40-467c-8d17-4d9e6212cbf6: finished: {'info': {'uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'vguuid': 'aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', 'lver': 287, 'state': 'OK', 'version': '2', 'role': 'Master', 'master_ver': 1, 'spm_id': 2, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'OS'}} >Thread-15914424::DEBUG::2013-11-20 03:31:52,537::task::495::TaskManager.Task::(_debug) Task 657f9c47-bc40-467c-8d17-4d9e6212cbf6: moving from state preparing -> state finished >Thread-15914424::DEBUG::2013-11-20 03:31:52,538::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914424::DEBUG::2013-11-20 03:31:52,539::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914424::DEBUG::2013-11-20 03:31:52,539::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914424::DEBUG::2013-11-20 03:31:52,540::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914424::DEBUG::2013-11-20 03:31:52,541::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914424::DEBUG::2013-11-20 03:31:52,541::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914424::DEBUG::2013-11-20 03:31:52,542::task::495::TaskManager.Task::(_debug) Task 657f9c47-bc40-467c-8d17-4d9e6212cbf6: ref 0 aborting False >Thread-15914424::INFO::2013-11-20 03:31:52,543::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'vguuid': 'aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', 'lver': 287, 'state': 'OK', 'version': '2', 'role': 'Master', 'master_ver': 1, 'spm_id': 2, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'OS'}} >Thread-15914424::DEBUG::2013-11-20 03:31:52,558::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/libexec/vdsm/hooks/before_vm_start/50_vhostmd' (cwd None) >Thread-15914424::DEBUG::2013-11-20 03:31:53,032::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914424::INFO::2013-11-20 03:31:53,034::hooks::51::root::(_runHooksDir) >Thread-15914424::DEBUG::2013-11-20 03:31:53,035::libvirtvm::1077::vm.Vm::(_run) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::<?xml version="1.0" encoding="utf-8"?> ><domain type="kvm"> > <name>Taipei</name> > <uuid>5f587d5c-d4b4-4242-b049-ede331c7f8b4</uuid> > <memory>2097152</memory> > <currentMemory>2097152</currentMemory> > <vcpu>2</vcpu> > <devices> > <disk device="disk" type="block"> > <source dev="/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/143cb26d-de08-4d30-89cf-aee10245b66a/d7028f7c-0551-4bbe-a907-6f804cc9bd54"/> > <target bus="virtio" dev="vda"/> > <serial>30-89cf-aee10245b66a</serial> > <driver cache="none" error_policy="stop" io="native" name="qemu" type="qcow2"/> > </disk> > <disk device="cdrom" type="file"> > <source file="/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO"/> > <target bus="ide" dev="hdc"/> > </disk> > <controller index="0" ports="16" type="virtio-serial"/> > <channel type="unix"> > <target name="com.redhat.rhevm.vdsm" type="virtio"/> > <source mode="bind" path="/var/lib/libvirt/qemu/channels/Taipei.com.redhat.rhevm.vdsm"/> > </channel> > <memballoon model="none"/> > <interface type="bridge"> > <mac address="00:1a:4a:a8:64:17"/> > <model type="virtio"/> > <source bridge="rhevm"/> > </interface> > <input bus="ps2" type="mouse"/> > <video> > <model heads="1" type="qxl" vram="65536"/> > </video> > <channel type="spicevmc"> > <target name="com.redhat.spice.0" type="virtio"/> > </channel> > <graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice"> > <channel mode="secure" name="main"/> > <channel mode="secure" name="inputs"/> > </graphics> > </devices> > <os> > <type arch="x86_64" machine="rhel6.2.0">hvm</type> > <boot dev="cdrom"/> > <smbios mode="sysinfo"/> > </os> > <sysinfo type="smbios"> > <system> > <entry name="manufacturer">Red Hat</entry> > <entry name="product">RHEV Hypervisor</entry> > <entry name="version">6.3-20121012.0.el6_3</entry> > <entry name="serial">30343536-3138-5A43-4A32-333630373442_AC:16:2D:76:EB:F0</entry> > <entry name="uuid">5f587d5c-d4b4-4242-b049-ede331c7f8b4</entry> > </system> > </sysinfo> > <clock adjustment="25201" offset="variable"> > <timer name="rtc" tickpolicy="catchup"/> > </clock> > <features> > <acpi/> > </features> > <cpu match="exact"> > <model>Westmere</model> > <topology cores="1" sockets="2" threads="1"/> > </cpu> ></domain> > >Thread-15912392::INFO::2013-11-20 03:31:53,626::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:31:53,627::task::495::TaskManager.Task::(_debug) Task 51c1f67f-7634-4cf3-b094-0b1e8ff3a18f: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:31:53,630::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`1ca04b10-2ac7-455c-a078-78a75abd5794`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:31:53,631::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:31:53,632::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:31:53,633::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`1ca04b10-2ac7-455c-a078-78a75abd5794`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:31:53,634::task::495::TaskManager.Task::(_debug) Task 51c1f67f-7634-4cf3-b094-0b1e8ff3a18f: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:31:53,635::task::495::TaskManager.Task::(_debug) Task 51c1f67f-7634-4cf3-b094-0b1e8ff3a18f: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:31:53,636::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15912392::DEBUG::2013-11-20 03:31:53,638::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:31:53,867::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:31:54,553::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.081357 s, 12.6 MB/s\n'; <rc> = 0 >Thread-15912392::DEBUG::2013-11-20 03:31:54,597::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-20 03:31:54,722::libvirtvm::1401::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::event Started detail 0 opaque None >Thread-15912392::DEBUG::2013-11-20 03:31:54,893::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15912392::DEBUG::2013-11-20 03:31:54,894::task::495::TaskManager.Task::(_debug) Task 51c1f67f-7634-4cf3-b094-0b1e8ff3a18f: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:31:54,895::task::495::TaskManager.Task::(_debug) Task 51c1f67f-7634-4cf3-b094-0b1e8ff3a18f: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:31:54,905::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:31:54,906::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:31:54,907::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:31:54,917::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:31:54,918::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:31:54,919::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914424::DEBUG::2013-11-20 03:31:54,929::utils::307::vm.Vm::(start) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Start statistics collection >Thread-15912392::DEBUG::2013-11-20 03:31:54,930::task::495::TaskManager.Task::(_debug) Task 51c1f67f-7634-4cf3-b094-0b1e8ff3a18f: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:31:54,931::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15914435::DEBUG::2013-11-20 03:31:54,931::utils::336::vm.Vm::(run) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Stats thread started >Thread-15914436::DEBUG::2013-11-20 03:31:54,933::guestIF::47::root::(_connect) Attempting connection to /var/lib/libvirt/qemu/channels/Taipei.com.redhat.rhevm.vdsm >Thread-15914436::DEBUG::2013-11-20 03:31:54,934::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /usr/share/vdsm/prepare-vmchannel /var/lib/libvirt/qemu/channels/Taipei.com.redhat.rhevm.vdsm' (cwd None) >Thread-15914424::WARNING::2013-11-20 03:31:55,117::libvirtvm::1124::vm.Vm::(_readPauseCode) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::_readPauseCode unsupported by libvirt vm >Thread-15914435::INFO::2013-11-20 03:31:55,121::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=143cb26d-de08-4d30-89cf-aee10245b66a volUUID=d7028f7c-0551-4bbe-a907-6f804cc9bd54) >Thread-15914424::DEBUG::2013-11-20 03:31:55,124::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /usr/share/vdsm/get-vm-pid Taipei' (cwd None) >Thread-15914435::DEBUG::2013-11-20 03:31:55,125::task::495::TaskManager.Task::(_debug) Task cd431d70-a263-458f-9ecc-ff807bdfb990: moving from state init -> state preparing >Thread-15914435::DEBUG::2013-11-20 03:31:55,128::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`31156151-a9bd-4cbb-a954-e87d484abfb3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914435::DEBUG::2013-11-20 03:31:55,406::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914435::DEBUG::2013-11-20 03:31:55,420::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914435::DEBUG::2013-11-20 03:31:55,422::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`31156151-a9bd-4cbb-a954-e87d484abfb3`::Granted request >Thread-15914435::DEBUG::2013-11-20 03:31:55,424::task::495::TaskManager.Task::(_debug) Task cd431d70-a263-458f-9ecc-ff807bdfb990: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914435::DEBUG::2013-11-20 03:31:55,425::task::495::TaskManager.Task::(_debug) Task cd431d70-a263-458f-9ecc-ff807bdfb990: ref 1 aborting False >Thread-15914435::DEBUG::2013-11-20 03:31:55,426::task::495::TaskManager.Task::(_debug) Task cd431d70-a263-458f-9ecc-ff807bdfb990: finished: {'truesize': '56908316672', 'apparentsize': '56908316672'} >Thread-15914435::DEBUG::2013-11-20 03:31:55,427::task::495::TaskManager.Task::(_debug) Task cd431d70-a263-458f-9ecc-ff807bdfb990: moving from state preparing -> state finished >Thread-15914435::DEBUG::2013-11-20 03:31:55,428::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914435::DEBUG::2013-11-20 03:31:55,428::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914435::DEBUG::2013-11-20 03:31:55,429::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914435::DEBUG::2013-11-20 03:31:55,430::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914435::DEBUG::2013-11-20 03:31:55,430::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914435::DEBUG::2013-11-20 03:31:55,431::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914435::DEBUG::2013-11-20 03:31:55,432::task::495::TaskManager.Task::(_debug) Task cd431d70-a263-458f-9ecc-ff807bdfb990: ref 0 aborting False >Thread-15914435::INFO::2013-11-20 03:31:55,432::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '56908316672', 'apparentsize': '56908316672'} >Thread-15914436::DEBUG::2013-11-20 03:31:55,679::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914436::DEBUG::2013-11-20 03:31:55,681::guestIF::53::root::(_connect) connected to /var/lib/libvirt/qemu/channels/Taipei.com.redhat.rhevm.vdsm >Thread-15914424::DEBUG::2013-11-20 03:31:55,696::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914424::DEBUG::2013-11-20 03:31:55,708::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /sbin/service ksmtuned retune' (cwd None) >Thread-15914424::DEBUG::2013-11-20 03:31:56,157::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914424::DEBUG::2013-11-20 03:31:56,158::vm::358::vm.Vm::(_startUnderlyingVm) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::_ongoingCreations released >Thread-15914438::DEBUG::2013-11-20 03:31:56,221::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15914439::DEBUG::2013-11-20 03:31:56,412::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914439::INFO::2013-11-20 03:31:56,416::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914439::DEBUG::2013-11-20 03:31:56,417::task::495::TaskManager.Task::(_debug) Task 8c11591e-0b44-4ab4-abcd-edee7f4d545c: moving from state init -> state preparing >Thread-15914439::DEBUG::2013-11-20 03:31:56,418::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914439::DEBUG::2013-11-20 03:31:56,419::task::495::TaskManager.Task::(_debug) Task 8c11591e-0b44-4ab4-abcd-edee7f4d545c: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914439::DEBUG::2013-11-20 03:31:56,420::task::495::TaskManager.Task::(_debug) Task 8c11591e-0b44-4ab4-abcd-edee7f4d545c: moving from state preparing -> state finished >Thread-15914439::DEBUG::2013-11-20 03:31:56,421::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914439::DEBUG::2013-11-20 03:31:56,421::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914439::DEBUG::2013-11-20 03:31:56,422::task::495::TaskManager.Task::(_debug) Task 8c11591e-0b44-4ab4-abcd-edee7f4d545c: ref 0 aborting False >Thread-15914439::INFO::2013-11-20 03:31:56,423::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914440::DEBUG::2013-11-20 03:31:56,428::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914440::INFO::2013-11-20 03:31:56,440::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914440::DEBUG::2013-11-20 03:31:56,441::task::495::TaskManager.Task::(_debug) Task 8a0d50b5-7818-4c1a-b262-0c6539d8d2d6: moving from state init -> state preparing >Thread-15914440::DEBUG::2013-11-20 03:31:56,443::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`69df2f61-00cc-4ecd-9982-e8926bc08f79`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914440::DEBUG::2013-11-20 03:31:56,444::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914440::DEBUG::2013-11-20 03:31:56,445::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914440::DEBUG::2013-11-20 03:31:56,445::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`69df2f61-00cc-4ecd-9982-e8926bc08f79`::Granted request >Thread-15914440::DEBUG::2013-11-20 03:31:56,446::task::495::TaskManager.Task::(_debug) Task 8a0d50b5-7818-4c1a-b262-0c6539d8d2d6: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914440::DEBUG::2013-11-20 03:31:56,447::task::495::TaskManager.Task::(_debug) Task 8a0d50b5-7818-4c1a-b262-0c6539d8d2d6: ref 1 aborting False >Thread-15914440::WARNING::2013-11-20 03:31:56,449::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914440::DEBUG::2013-11-20 03:31:56,450::task::495::TaskManager.Task::(_debug) Task 8a0d50b5-7818-4c1a-b262-0c6539d8d2d6: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914440::DEBUG::2013-11-20 03:31:56,451::task::495::TaskManager.Task::(_debug) Task 8a0d50b5-7818-4c1a-b262-0c6539d8d2d6: moving from state preparing -> state finished >Thread-15914440::DEBUG::2013-11-20 03:31:56,452::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914440::DEBUG::2013-11-20 03:31:56,452::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914440::DEBUG::2013-11-20 03:31:56,453::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914440::DEBUG::2013-11-20 03:31:56,454::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914440::DEBUG::2013-11-20 03:31:56,454::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914440::DEBUG::2013-11-20 03:31:56,455::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914440::DEBUG::2013-11-20 03:31:56,455::task::495::TaskManager.Task::(_debug) Task 8a0d50b5-7818-4c1a-b262-0c6539d8d2d6: ref 0 aborting False >Thread-15914440::INFO::2013-11-20 03:31:56,456::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:31:56,569::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914438::DEBUG::2013-11-20 03:31:57,010::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Thread-15914441::DEBUG::2013-11-20 03:31:57,019::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15914442::DEBUG::2013-11-20 03:31:57,246::clientIF::54::vds::(wrapper) [192.168.100.135]::call create with ({'bridge': 'rhevm', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.2.0', 'vmId': '163a3426-5ec5-46a5-b8f1-af06fc5d7e7d', 'transparentHugePages': 'true', 'spiceSslCipherSuite': 'DEFAULT', 'cpuType': 'Westmere', 'smp': '4', 'macAddr': '00:1a:4a:a8:64:15', 'boot': 'c', 'custom': {}, 'vmType': 'kvm', 'memSize': 4096, 'smpCoresPerSocket': '1', 'vmName': 'Luanda', 'spiceMonitors': '1', 'nice': '0', 'drives': [{'index': '0', 'domainID': '6b892067-ac7e-4039-a7be-5b7387525b16', 'format': 'raw', 'volumeID': 'b9cffdbe-152e-41ce-ae5d-08db7c08454e', 'imageID': '5f930ef3-763f-4294-a62a-6d52df220f8c', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'ide'}], 'smartcardEnable': 'false', 'nicModel': 'pv', 'keyboardLayout': 'en-us', 'kvmEnable': 'true', 'pitReinjection': 'false', 'timeOffset': '0', 'spiceSecureChannels': 'smain,sinputs', 'display': 'qxl'},) {} >Dummy-15912651::DEBUG::2013-11-20 03:31:57,703::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.699069 s, 1.5 MB/s\n'; <rc> = 0 >Thread-15914442::INFO::2013-11-20 03:31:57,820::clientIF::523::vds::(_getNetworkIp) network None: using 0 >Thread-15914442::INFO::2013-11-20 03:31:57,821::clientIF::656::vds::(create) vmContainerLock acquired by vm 163a3426-5ec5-46a5-b8f1-af06fc5d7e7d >Thread-15914443::DEBUG::2013-11-20 03:31:57,825::vm::342::vm.Vm::(_startUnderlyingVm) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Start >Thread-15914442::DEBUG::2013-11-20 03:31:57,826::clientIF::669::vds::(create) Total desktops after creation of 163a3426-5ec5-46a5-b8f1-af06fc5d7e7d is 8 >Thread-15914443::DEBUG::2013-11-20 03:31:57,827::vm::346::vm.Vm::(_startUnderlyingVm) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::_ongoingCreations acquired >Thread-15914442::DEBUG::2013-11-20 03:31:57,828::clientIF::59::vds::(wrapper) return create with {'status': {'message': 'Done', 'code': 0}, 'vmList': {'status': 'WaitForLaunch', 'bridge': 'rhevm', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.2.0', 'vmId': '163a3426-5ec5-46a5-b8f1-af06fc5d7e7d', 'pid': '0', 'drives': [{'index': '0', 'domainID': '6b892067-ac7e-4039-a7be-5b7387525b16', 'format': 'raw', 'volumeID': 'b9cffdbe-152e-41ce-ae5d-08db7c08454e', 'imageID': '5f930ef3-763f-4294-a62a-6d52df220f8c', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'ide'}], 'transparentHugePages': 'true', 'keyboardLayout': 'en-us', 'displayPort': '-1', 'displaySecurePort': '-1', 'spiceSslCipherSuite': 'DEFAULT', 'cpuType': 'Westmere', 'custom': {}, 'clientIp': '', 'nicModel': 'pv', 'macAddr': '00:1a:4a:a8:64:15', 'kvmEnable': 'true', 'pitReinjection': 'false', 'boot': 'c', 'smp': '4', 'vmType': 'kvm', 'smartcardEnable': 'false', 'timeOffset': '0', 'memSize': 4096, 'displayIp': '0', 'spiceSecureChannels': 'smain,sinputs', 'smpCoresPerSocket': '1', 'vmName': 'Luanda', 'display': 'qxl', 'spiceMonitors': '1', 'nice': '0'}} >Thread-15914443::INFO::2013-11-20 03:31:57,829::libvirtvm::1061::vm.Vm::(_run) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::VM wrapper has started >Thread-15914443::INFO::2013-11-20 03:31:57,833::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=5f930ef3-763f-4294-a62a-6d52df220f8c volUUID=b9cffdbe-152e-41ce-ae5d-08db7c08454e) >Thread-15914443::DEBUG::2013-11-20 03:31:57,834::task::495::TaskManager.Task::(_debug) Task 5970dcff-dc75-4fcf-ab88-4fcf11a7ce28: moving from state init -> state preparing >Thread-15914443::DEBUG::2013-11-20 03:31:57,836::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b1ad435a-87cb-4308-90fa-644c87281444`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1690' at 'public_prepareVolume' >Thread-15914443::DEBUG::2013-11-20 03:31:57,836::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914443::DEBUG::2013-11-20 03:31:57,837::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914443::DEBUG::2013-11-20 03:31:57,838::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b1ad435a-87cb-4308-90fa-644c87281444`::Granted request >Thread-15914443::DEBUG::2013-11-20 03:31:57,838::task::495::TaskManager.Task::(_debug) Task 5970dcff-dc75-4fcf-ab88-4fcf11a7ce28: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914443::DEBUG::2013-11-20 03:31:57,839::task::495::TaskManager.Task::(_debug) Task 5970dcff-dc75-4fcf-ab88-4fcf11a7ce28: ref 1 aborting False >Thread-15914443::DEBUG::2013-11-20 03:31:57,839::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.5f930ef3-763f-4294-a62a-6d52df220f8c`ReqID=`56675b6f-aa9a-47ba-9c23-9942c3884454`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1695' at 'public_prepareVolume' >Thread-15914443::DEBUG::2013-11-20 03:31:57,840::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.5f930ef3-763f-4294-a62a-6d52df220f8c' for lock type 'exclusive' >Thread-15914443::DEBUG::2013-11-20 03:31:57,841::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15914443::DEBUG::2013-11-20 03:31:57,841::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-42838::DEBUG::2013-11-20 03:31:58,004::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15914441::DEBUG::2013-11-20 03:31:58,007::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Thread-15914443::DEBUG::2013-11-20 03:31:58,753::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-42838::DEBUG::2013-11-20 03:31:58,960::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914443::DEBUG::2013-11-20 03:31:59,061::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15914446::DEBUG::2013-11-20 03:31:59,064::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('5f587d5c-d4b4-4242-b049-ede331c7f8b4',) {} >Thread-15914443::DEBUG::2013-11-20 03:31:59,066::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=31 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15914446::DEBUG::2013-11-20 03:31:59,067::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::CPU stats not available >Thread-15914446::DEBUG::2013-11-20 03:31:59,068::libvirtvm::202::vm.Vm::(_getNetworkStats) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Network stats not available >Thread-15914446::DEBUG::2013-11-20 03:31:59,272::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk stats not available >Thread-15914446::DEBUG::2013-11-20 03:31:59,274::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk latency not available >Thread-15914446::DEBUG::2013-11-20 03:31:59,275::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5523', 'displayIp': '0', 'displayPort': u'5912', 'session': 'Unknown', 'displaySecurePort': u'5913', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet7': {'state': 'unknown', 'macAddr': u'00:1a:4a:a8:64:17', 'speed': '1000', 'name': u'vnet7'}}, 'vmId': '5f587d5c-d4b4-4242-b049-ede331c7f8b4', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '56908316672', 'writeLatency': '0', 'imageID': '143cb26d-de08-4d30-89cf-aee10245b66a', 'flushLatency': '0', 'truesize': '56908316672'}}, 'boot': 'd', 'statsAge': '1.46', 'cpuIdle': '100.00', 'elapsedTime': '16', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '25201', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15914443::DEBUG::2013-11-20 03:31:59,463::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.00718333 s, 71.3 kB/s\n'; <rc> = 0 >Thread-15914443::DEBUG::2013-11-20 03:31:59,464::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.00718333 s, 71.3 kB/s'], size: 512 >Thread-15914443::DEBUG::2013-11-20 03:31:59,465::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=31 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:31:59,719::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914443::DEBUG::2013-11-20 03:31:59,952::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.0001905 s, 2.7 MB/s\n'; <rc> = 0 >Thread-15914443::DEBUG::2013-11-20 03:31:59,955::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.0001905 s, 2.7 MB/s'], size: 512 >Thread-15914443::INFO::2013-11-20 03:31:59,956::image::340::Storage.Image::(getChain) sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 imgUUID=5f930ef3-763f-4294-a62a-6d52df220f8c chain=[<storage.blockVolume.BlockVolume instance at 0x7f48ac6029e0>] >Thread-15914443::DEBUG::2013-11-20 03:31:59,957::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --autobackup n --available y 6b892067-ac7e-4039-a7be-5b7387525b16/b9cffdbe-152e-41ce-ae5d-08db7c08454e' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:32:00,394::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0899939 s, 11.4 MB/s\n'; <rc> = 0 >Thread-15914443::DEBUG::2013-11-20 03:32:00,773::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914443::DEBUG::2013-11-20 03:32:00,774::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15914443::DEBUG::2013-11-20 03:32:00,775::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15914443::DEBUG::2013-11-20 03:32:00,776::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.b9cffdbe-152e-41ce-ae5d-08db7c08454e`ReqID=`e416d3d3-2558-4cf8-a0e5-9e1d171d8fce`::Request was made in '/usr/share/vdsm/storage/resourceFactories.py' line '149' at '__getResourceCandidatesList' >Thread-15914443::DEBUG::2013-11-20 03:32:00,776::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.b9cffdbe-152e-41ce-ae5d-08db7c08454e' for lock type 'exclusive' >Thread-15914443::DEBUG::2013-11-20 03:32:00,777::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.b9cffdbe-152e-41ce-ae5d-08db7c08454e' is free. Now locking as 'exclusive' (1 active user) >Thread-15914443::DEBUG::2013-11-20 03:32:00,777::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.b9cffdbe-152e-41ce-ae5d-08db7c08454e`ReqID=`e416d3d3-2558-4cf8-a0e5-9e1d171d8fce`::Granted request >Thread-15914443::DEBUG::2013-11-20 03:32:00,778::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.5f930ef3-763f-4294-a62a-6d52df220f8c' is free. Now locking as 'exclusive' (1 active user) >Thread-15914443::DEBUG::2013-11-20 03:32:00,778::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.5f930ef3-763f-4294-a62a-6d52df220f8c`ReqID=`56675b6f-aa9a-47ba-9c23-9942c3884454`::Granted request >Thread-15914443::DEBUG::2013-11-20 03:32:00,779::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15914443::DEBUG::2013-11-20 03:32:00,780::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15914443::DEBUG::2013-11-20 03:32:01,501::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914443::DEBUG::2013-11-20 03:32:01,739::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15914443::DEBUG::2013-11-20 03:32:01,742::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --refresh 6b892067-ac7e-4039-a7be-5b7387525b16/b9cffdbe-152e-41ce-ae5d-08db7c08454e' (cwd None) >Thread-15914443::DEBUG::2013-11-20 03:32:02,340::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914443::DEBUG::2013-11-20 03:32:02,350::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15914443::DEBUG::2013-11-20 03:32:02,357::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15914443::INFO::2013-11-20 03:32:02,358::volume::517::Storage.Volume::(prepare) Volume: preparing volume 6b892067-ac7e-4039-a7be-5b7387525b16/b9cffdbe-152e-41ce-ae5d-08db7c08454e >Thread-15914443::DEBUG::2013-11-20 03:32:02,359::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15914443::DEBUG::2013-11-20 03:32:02,360::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:32:02,540::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914448::DEBUG::2013-11-20 03:32:02,804::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('5f587d5c-d4b4-4242-b049-ede331c7f8b4',) {} >Thread-15914448::DEBUG::2013-11-20 03:32:02,805::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::CPU stats not available >Thread-15914448::DEBUG::2013-11-20 03:32:02,806::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk stats not available >Thread-15914448::DEBUG::2013-11-20 03:32:02,806::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk latency not available >Thread-15914448::DEBUG::2013-11-20 03:32:02,807::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5523', 'displayIp': '0', 'displayPort': u'5912', 'session': 'Unknown', 'displaySecurePort': u'5913', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet7': {'macAddr': u'00:1a:4a:a8:64:17', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet7'}}, 'vmId': '5f587d5c-d4b4-4242-b049-ede331c7f8b4', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '56908316672', 'writeLatency': '0', 'imageID': '143cb26d-de08-4d30-89cf-aee10245b66a', 'flushLatency': '0', 'truesize': '56908316672'}}, 'boot': 'd', 'statsAge': '1.13', 'cpuIdle': '100.00', 'elapsedTime': '19', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '25201', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:32:03,007::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0476825 s, 21.5 MB/s\n'; <rc> = 0 >Thread-15914443::DEBUG::2013-11-20 03:32:03,060::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914443::DEBUG::2013-11-20 03:32:03,248::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15914443::DEBUG::2013-11-20 03:32:03,249::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=31 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15914443::DEBUG::2013-11-20 03:32:03,606::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000206212 s, 2.5 MB/s\n'; <rc> = 0 >Thread-15914443::DEBUG::2013-11-20 03:32:03,608::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000206212 s, 2.5 MB/s'], size: 512 >Thread-15914443::DEBUG::2013-11-20 03:32:03,609::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=31 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15914443::DEBUG::2013-11-20 03:32:04,025::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000221991 s, 2.3 MB/s\n'; <rc> = 0 >Thread-15914443::DEBUG::2013-11-20 03:32:04,026::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000221991 s, 2.3 MB/s'], size: 512 >Thread-15914443::DEBUG::2013-11-20 03:32:04,029::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.b9cffdbe-152e-41ce-ae5d-08db7c08454e`ReqID=`7e87b0f3-e582-4035-9590-420f72cc1c9e`::Request was made in '/usr/share/vdsm/storage/volume.py' line '533' at 'prepare' >Thread-15914443::DEBUG::2013-11-20 03:32:04,030::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.b9cffdbe-152e-41ce-ae5d-08db7c08454e' for lock type 'exclusive' >Thread-15914443::DEBUG::2013-11-20 03:32:04,031::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.b9cffdbe-152e-41ce-ae5d-08db7c08454e' is free. Now locking as 'exclusive' (1 active user) >Thread-15914443::DEBUG::2013-11-20 03:32:04,032::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.b9cffdbe-152e-41ce-ae5d-08db7c08454e`ReqID=`7e87b0f3-e582-4035-9590-420f72cc1c9e`::Granted request >Thread-15914443::DEBUG::2013-11-20 03:32:04,034::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=31 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15914443::DEBUG::2013-11-20 03:32:04,696::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000219516 s, 2.3 MB/s\n'; <rc> = 0 >Thread-15914443::DEBUG::2013-11-20 03:32:04,697::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000219516 s, 2.3 MB/s'], size: 512 >Thread-15914443::DEBUG::2013-11-20 03:32:04,699::task::495::TaskManager.Task::(_debug) Task 5970dcff-dc75-4fcf-ab88-4fcf11a7ce28: finished: None >Thread-15914443::DEBUG::2013-11-20 03:32:04,699::task::495::TaskManager.Task::(_debug) Task 5970dcff-dc75-4fcf-ab88-4fcf11a7ce28: moving from state preparing -> state finished >Thread-15914443::DEBUG::2013-11-20 03:32:04,700::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914443::DEBUG::2013-11-20 03:32:04,701::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914443::DEBUG::2013-11-20 03:32:04,702::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914443::DEBUG::2013-11-20 03:32:04,702::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914443::DEBUG::2013-11-20 03:32:04,703::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914443::DEBUG::2013-11-20 03:32:04,704::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914443::DEBUG::2013-11-20 03:32:04,705::task::495::TaskManager.Task::(_debug) Task 5970dcff-dc75-4fcf-ab88-4fcf11a7ce28: ref 0 aborting False >Thread-15914443::INFO::2013-11-20 03:32:04,705::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15914443::INFO::2013-11-20 03:32:04,706::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=5f930ef3-763f-4294-a62a-6d52df220f8c volUUID=b9cffdbe-152e-41ce-ae5d-08db7c08454e) >Thread-15914443::DEBUG::2013-11-20 03:32:04,707::task::495::TaskManager.Task::(_debug) Task a7c76b9a-b0ef-495c-a422-a99ea841f61f: moving from state init -> state preparing >Thread-15914443::DEBUG::2013-11-20 03:32:04,708::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`576f44dc-0036-4c83-97ae-929801b51f5c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1666' at 'public_getVolumePath' >Thread-15914443::DEBUG::2013-11-20 03:32:04,709::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914443::DEBUG::2013-11-20 03:32:04,710::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914443::DEBUG::2013-11-20 03:32:04,710::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`576f44dc-0036-4c83-97ae-929801b51f5c`::Granted request >Thread-15914443::DEBUG::2013-11-20 03:32:04,711::task::495::TaskManager.Task::(_debug) Task a7c76b9a-b0ef-495c-a422-a99ea841f61f: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914443::DEBUG::2013-11-20 03:32:04,712::task::495::TaskManager.Task::(_debug) Task a7c76b9a-b0ef-495c-a422-a99ea841f61f: ref 1 aborting False >Thread-15914443::DEBUG::2013-11-20 03:32:04,713::task::495::TaskManager.Task::(_debug) Task a7c76b9a-b0ef-495c-a422-a99ea841f61f: finished: {'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/5f930ef3-763f-4294-a62a-6d52df220f8c/b9cffdbe-152e-41ce-ae5d-08db7c08454e'} >Thread-15914443::DEBUG::2013-11-20 03:32:04,714::task::495::TaskManager.Task::(_debug) Task a7c76b9a-b0ef-495c-a422-a99ea841f61f: moving from state preparing -> state finished >Thread-15914443::DEBUG::2013-11-20 03:32:04,715::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914443::DEBUG::2013-11-20 03:32:04,715::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914443::DEBUG::2013-11-20 03:32:04,716::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914443::DEBUG::2013-11-20 03:32:04,717::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914443::DEBUG::2013-11-20 03:32:04,717::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914443::DEBUG::2013-11-20 03:32:04,718::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914443::DEBUG::2013-11-20 03:32:04,718::task::495::TaskManager.Task::(_debug) Task a7c76b9a-b0ef-495c-a422-a99ea841f61f: ref 0 aborting False >Thread-15914443::INFO::2013-11-20 03:32:04,719::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, Return response: {'status': {'message': 'OK', 'code': 0}, 'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/5f930ef3-763f-4294-a62a-6d52df220f8c/b9cffdbe-152e-41ce-ae5d-08db7c08454e'} >Thread-15914443::INFO::2013-11-20 03:32:04,720::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=5f930ef3-763f-4294-a62a-6d52df220f8c volUUID=b9cffdbe-152e-41ce-ae5d-08db7c08454e) >Thread-15914443::DEBUG::2013-11-20 03:32:04,721::task::495::TaskManager.Task::(_debug) Task 5c6b5979-8662-4799-a435-847e6c3b2e52: moving from state init -> state preparing >Thread-15914443::DEBUG::2013-11-20 03:32:04,722::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a820f8d5-0db7-49ad-b1b0-4b1f81b47d17`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914443::DEBUG::2013-11-20 03:32:04,723::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914443::DEBUG::2013-11-20 03:32:04,723::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914443::DEBUG::2013-11-20 03:32:04,724::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a820f8d5-0db7-49ad-b1b0-4b1f81b47d17`::Granted request >Thread-15914443::DEBUG::2013-11-20 03:32:04,725::task::495::TaskManager.Task::(_debug) Task 5c6b5979-8662-4799-a435-847e6c3b2e52: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914443::DEBUG::2013-11-20 03:32:04,726::task::495::TaskManager.Task::(_debug) Task 5c6b5979-8662-4799-a435-847e6c3b2e52: ref 1 aborting False >Thread-15914443::DEBUG::2013-11-20 03:32:04,726::task::495::TaskManager.Task::(_debug) Task 5c6b5979-8662-4799-a435-847e6c3b2e52: finished: {'truesize': '53821308928', 'apparentsize': '53821308928'} >Thread-15914443::DEBUG::2013-11-20 03:32:04,727::task::495::TaskManager.Task::(_debug) Task 5c6b5979-8662-4799-a435-847e6c3b2e52: moving from state preparing -> state finished >Thread-15914443::DEBUG::2013-11-20 03:32:04,728::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914443::DEBUG::2013-11-20 03:32:04,728::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914443::DEBUG::2013-11-20 03:32:04,729::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914443::DEBUG::2013-11-20 03:32:04,729::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914443::DEBUG::2013-11-20 03:32:04,730::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914443::DEBUG::2013-11-20 03:32:04,731::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914443::DEBUG::2013-11-20 03:32:04,731::task::495::TaskManager.Task::(_debug) Task 5c6b5979-8662-4799-a435-847e6c3b2e52: ref 0 aborting False >Thread-15914443::INFO::2013-11-20 03:32:04,732::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '53821308928', 'apparentsize': '53821308928'} >Thread-15914443::INFO::2013-11-20 03:32:04,733::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16) >Thread-15914443::DEBUG::2013-11-20 03:32:04,733::task::495::TaskManager.Task::(_debug) Task 11f6d703-c803-435c-9fce-deb9f74961eb: moving from state init -> state preparing >Thread-15914443::INFO::2013-11-20 03:32:04,734::blockSD::674::Storage.StorageDomain::(validate) sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 >Thread-15914443::DEBUG::2013-11-20 03:32:04,735::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15914443::DEBUG::2013-11-20 03:32:04,736::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15914443::DEBUG::2013-11-20 03:32:04,736::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15914443::DEBUG::2013-11-20 03:32:04,738::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15914443::DEBUG::2013-11-20 03:32:04,739::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15914443::DEBUG::2013-11-20 03:32:04,740::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:32:05,018::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:32:05,816::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0762156 s, 13.4 MB/s\n'; <rc> = 0 >Thread-15914449::INFO::2013-11-20 03:32:05,835::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914449::DEBUG::2013-11-20 03:32:05,836::task::495::TaskManager.Task::(_debug) Task 1236ef9e-63c7-4b59-8ba7-43a808ea1c6c: moving from state init -> state preparing >Thread-15914449::DEBUG::2013-11-20 03:32:05,836::task::495::TaskManager.Task::(_debug) Task 1236ef9e-63c7-4b59-8ba7-43a808ea1c6c: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00133514404297', 'lastCheck': 1384918319.0651929, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00243592262268', 'lastCheck': 1384918324.952491, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00214195251465', 'lastCheck': 1384918316.4166529, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000575065612793', 'lastCheck': 1384918316.920346, 'valid': True, 'code': 0}} >Thread-15914449::DEBUG::2013-11-20 03:32:05,838::task::495::TaskManager.Task::(_debug) Task 1236ef9e-63c7-4b59-8ba7-43a808ea1c6c: moving from state preparing -> state finished >Thread-15914449::DEBUG::2013-11-20 03:32:05,838::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914449::DEBUG::2013-11-20 03:32:05,839::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914449::DEBUG::2013-11-20 03:32:05,840::task::495::TaskManager.Task::(_debug) Task 1236ef9e-63c7-4b59-8ba7-43a808ea1c6c: ref 0 aborting False >Thread-15914449::INFO::2013-11-20 03:32:05,840::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00133514404297', 'lastCheck': 1384918319.0651929, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00243592262268', 'lastCheck': 1384918324.952491, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00214195251465', 'lastCheck': 1384918316.4166529, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000575065612793', 'lastCheck': 1384918316.920346, 'valid': True, 'code': 0}} >Thread-15914443::DEBUG::2013-11-20 03:32:05,842::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914443::DEBUG::2013-11-20 03:32:05,845::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15914443::DEBUG::2013-11-20 03:32:05,846::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'_SHA_CKSUM=716ea7492d9134a4b13de27ac61a336ad474727c', u'POOL_SPM_ID=2', u'POOL_SPM_LVER=287'] >Thread-15914443::DEBUG::2013-11-20 03:32:05,849::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a1d0f8cd-d5a4-46d8-9356-f128ae225d9f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1297' at 'public_getStorageDomainInfo' >Thread-15914443::DEBUG::2013-11-20 03:32:05,850::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914443::DEBUG::2013-11-20 03:32:05,851::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914443::DEBUG::2013-11-20 03:32:05,856::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a1d0f8cd-d5a4-46d8-9356-f128ae225d9f`::Granted request >Thread-15914443::DEBUG::2013-11-20 03:32:05,857::task::495::TaskManager.Task::(_debug) Task 11f6d703-c803-435c-9fce-deb9f74961eb: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914443::DEBUG::2013-11-20 03:32:05,857::task::495::TaskManager.Task::(_debug) Task 11f6d703-c803-435c-9fce-deb9f74961eb: ref 1 aborting False >Thread-15914443::WARNING::2013-11-20 03:32:05,859::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914443::DEBUG::2013-11-20 03:32:05,860::task::495::TaskManager.Task::(_debug) Task 11f6d703-c803-435c-9fce-deb9f74961eb: finished: {'info': {'uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'vguuid': 'aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', 'lver': 287, 'state': 'OK', 'version': '2', 'role': 'Master', 'master_ver': 1, 'spm_id': 2, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'OS'}} >Thread-15914443::DEBUG::2013-11-20 03:32:05,860::task::495::TaskManager.Task::(_debug) Task 11f6d703-c803-435c-9fce-deb9f74961eb: moving from state preparing -> state finished >Thread-15914443::DEBUG::2013-11-20 03:32:05,861::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914443::DEBUG::2013-11-20 03:32:05,861::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914443::DEBUG::2013-11-20 03:32:05,862::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914443::DEBUG::2013-11-20 03:32:05,863::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914443::DEBUG::2013-11-20 03:32:05,864::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914450::DEBUG::2013-11-20 03:32:05,866::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::CPU stats not available >Thread-15914443::DEBUG::2013-11-20 03:32:05,867::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914450::DEBUG::2013-11-20 03:32:05,867::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk stats not available >Thread-15914450::DEBUG::2013-11-20 03:32:05,868::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk latency not available >Thread-15914443::DEBUG::2013-11-20 03:32:05,868::task::495::TaskManager.Task::(_debug) Task 11f6d703-c803-435c-9fce-deb9f74961eb: ref 0 aborting False >Thread-15914443::INFO::2013-11-20 03:32:05,876::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'vguuid': 'aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', 'lver': 287, 'state': 'OK', 'version': '2', 'role': 'Master', 'master_ver': 1, 'spm_id': 2, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'OS'}} >Thread-15914443::DEBUG::2013-11-20 03:32:05,888::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/libexec/vdsm/hooks/before_vm_start/50_vhostmd' (cwd None) >Thread-15914443::DEBUG::2013-11-20 03:32:06,360::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914443::INFO::2013-11-20 03:32:06,361::hooks::51::root::(_runHooksDir) >Thread-15914443::DEBUG::2013-11-20 03:32:06,362::libvirtvm::1077::vm.Vm::(_run) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::<?xml version="1.0" encoding="utf-8"?> ><domain type="kvm"> > <name>Luanda</name> > <uuid>163a3426-5ec5-46a5-b8f1-af06fc5d7e7d</uuid> > <memory>4194304</memory> > <currentMemory>4194304</currentMemory> > <vcpu>4</vcpu> > <devices> > <disk device="disk" type="block"> > <source dev="/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/5f930ef3-763f-4294-a62a-6d52df220f8c/b9cffdbe-152e-41ce-ae5d-08db7c08454e"/> > <target bus="ide" dev="hda"/> > <serial>94-a62a-6d52df220f8c</serial> > <driver cache="none" error_policy="stop" io="native" name="qemu" type="raw"/> > </disk> > <disk device="cdrom" type="file"> > <target bus="ide" dev="hdc"/> > </disk> > <controller index="0" ports="16" type="virtio-serial"/> > <channel type="unix"> > <target name="com.redhat.rhevm.vdsm" type="virtio"/> > <source mode="bind" path="/var/lib/libvirt/qemu/channels/Luanda.com.redhat.rhevm.vdsm"/> > </channel> > <memballoon model="none"/> > <interface type="bridge"> > <mac address="00:1a:4a:a8:64:15"/> > <model type="virtio"/> > <source bridge="rhevm"/> > </interface> > <input bus="ps2" type="mouse"/> > <video> > <model heads="1" type="qxl" vram="65536"/> > </video> > <channel type="spicevmc"> > <target name="com.redhat.spice.0" type="virtio"/> > </channel> > <graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice"> > <channel mode="secure" name="main"/> > <channel mode="secure" name="inputs"/> > </graphics> > </devices> > <os> > <type arch="x86_64" machine="rhel6.2.0">hvm</type> > <boot dev="hd"/> > <smbios mode="sysinfo"/> > </os> > <sysinfo type="smbios"> > <system> > <entry name="manufacturer">Red Hat</entry> > <entry name="product">RHEV Hypervisor</entry> > <entry name="version">6.3-20121012.0.el6_3</entry> > <entry name="serial">30343536-3138-5A43-4A32-333630373442_AC:16:2D:76:EB:F0</entry> > <entry name="uuid">163a3426-5ec5-46a5-b8f1-af06fc5d7e7d</entry> > </system> > </sysinfo> > <clock adjustment="0" offset="variable"> > <timer name="rtc" tickpolicy="catchup"/> > </clock> > <features> > <acpi/> > </features> > <cpu match="exact"> > <model>Westmere</model> > <topology cores="1" sockets="4" threads="1"/> > </cpu> ></domain> > >Thread-15914451::DEBUG::2013-11-20 03:32:06,627::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914451::INFO::2013-11-20 03:32:06,628::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914451::DEBUG::2013-11-20 03:32:06,628::task::495::TaskManager.Task::(_debug) Task 06e9c810-27e2-4c4c-9aa3-f8ff2592c356: moving from state init -> state preparing >Thread-15914451::DEBUG::2013-11-20 03:32:06,629::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914451::DEBUG::2013-11-20 03:32:06,629::task::495::TaskManager.Task::(_debug) Task 06e9c810-27e2-4c4c-9aa3-f8ff2592c356: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914451::DEBUG::2013-11-20 03:32:06,630::task::495::TaskManager.Task::(_debug) Task 06e9c810-27e2-4c4c-9aa3-f8ff2592c356: moving from state preparing -> state finished >Thread-15914451::DEBUG::2013-11-20 03:32:06,630::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914451::DEBUG::2013-11-20 03:32:06,630::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914451::DEBUG::2013-11-20 03:32:06,631::task::495::TaskManager.Task::(_debug) Task 06e9c810-27e2-4c4c-9aa3-f8ff2592c356: ref 0 aborting False >Thread-15914451::INFO::2013-11-20 03:32:06,631::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914452::DEBUG::2013-11-20 03:32:06,635::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914452::INFO::2013-11-20 03:32:06,636::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914452::DEBUG::2013-11-20 03:32:06,636::task::495::TaskManager.Task::(_debug) Task 4369c542-a6a8-4434-880c-429150255f40: moving from state init -> state preparing >Thread-15914452::DEBUG::2013-11-20 03:32:06,637::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b0d176f7-af44-4730-ad79-ba61b40ffd63`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914452::DEBUG::2013-11-20 03:32:06,637::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914452::DEBUG::2013-11-20 03:32:06,638::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914452::DEBUG::2013-11-20 03:32:06,638::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b0d176f7-af44-4730-ad79-ba61b40ffd63`::Granted request >Thread-15914452::DEBUG::2013-11-20 03:32:06,639::task::495::TaskManager.Task::(_debug) Task 4369c542-a6a8-4434-880c-429150255f40: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914452::DEBUG::2013-11-20 03:32:06,639::task::495::TaskManager.Task::(_debug) Task 4369c542-a6a8-4434-880c-429150255f40: ref 1 aborting False >Thread-15914452::WARNING::2013-11-20 03:32:06,641::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914452::DEBUG::2013-11-20 03:32:06,642::task::495::TaskManager.Task::(_debug) Task 4369c542-a6a8-4434-880c-429150255f40: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914452::DEBUG::2013-11-20 03:32:06,642::task::495::TaskManager.Task::(_debug) Task 4369c542-a6a8-4434-880c-429150255f40: moving from state preparing -> state finished >Thread-15914452::DEBUG::2013-11-20 03:32:06,643::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914452::DEBUG::2013-11-20 03:32:06,643::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914452::DEBUG::2013-11-20 03:32:06,644::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914452::DEBUG::2013-11-20 03:32:06,644::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914452::DEBUG::2013-11-20 03:32:06,644::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914452::DEBUG::2013-11-20 03:32:06,645::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914452::DEBUG::2013-11-20 03:32:06,645::task::495::TaskManager.Task::(_debug) Task 4369c542-a6a8-4434-880c-429150255f40: ref 0 aborting False >Thread-15914452::INFO::2013-11-20 03:32:06,646::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >libvirtEventLoop::DEBUG::2013-11-20 03:32:07,339::libvirtvm::1401::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::event Started detail 0 opaque None >Thread-15914443::DEBUG::2013-11-20 03:32:07,376::utils::307::vm.Vm::(start) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Start statistics collection >Thread-15914453::DEBUG::2013-11-20 03:32:07,377::utils::336::vm.Vm::(run) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Stats thread started >Thread-15914454::DEBUG::2013-11-20 03:32:07,378::guestIF::47::root::(_connect) Attempting connection to /var/lib/libvirt/qemu/channels/Luanda.com.redhat.rhevm.vdsm >Thread-15914453::INFO::2013-11-20 03:32:07,379::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=5f930ef3-763f-4294-a62a-6d52df220f8c volUUID=b9cffdbe-152e-41ce-ae5d-08db7c08454e) >Thread-15914454::DEBUG::2013-11-20 03:32:07,380::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /usr/share/vdsm/prepare-vmchannel /var/lib/libvirt/qemu/channels/Luanda.com.redhat.rhevm.vdsm' (cwd None) >Thread-15914453::DEBUG::2013-11-20 03:32:07,380::task::495::TaskManager.Task::(_debug) Task d1b40b70-69bc-450c-93a4-44c03326fbab: moving from state init -> state preparing >Thread-15914453::DEBUG::2013-11-20 03:32:07,382::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e57d86e2-4734-4082-898f-c060122ada2f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914443::WARNING::2013-11-20 03:32:07,546::libvirtvm::1124::vm.Vm::(_readPauseCode) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::_readPauseCode unsupported by libvirt vm >Thread-15914453::DEBUG::2013-11-20 03:32:07,575::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914443::DEBUG::2013-11-20 03:32:07,587::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /usr/share/vdsm/get-vm-pid Luanda' (cwd None) >Thread-15914453::DEBUG::2013-11-20 03:32:07,595::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Dummy-15912651::DEBUG::2013-11-20 03:32:07,863::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914453::DEBUG::2013-11-20 03:32:07,866::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e57d86e2-4734-4082-898f-c060122ada2f`::Granted request >Thread-15914454::DEBUG::2013-11-20 03:32:07,870::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914453::DEBUG::2013-11-20 03:32:07,871::task::495::TaskManager.Task::(_debug) Task d1b40b70-69bc-450c-93a4-44c03326fbab: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914454::DEBUG::2013-11-20 03:32:08,148::guestIF::53::root::(_connect) connected to /var/lib/libvirt/qemu/channels/Luanda.com.redhat.rhevm.vdsm >Thread-15914453::DEBUG::2013-11-20 03:32:08,149::task::495::TaskManager.Task::(_debug) Task d1b40b70-69bc-450c-93a4-44c03326fbab: ref 1 aborting False >Thread-15914453::DEBUG::2013-11-20 03:32:08,152::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15914443::DEBUG::2013-11-20 03:32:08,155::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914453::DEBUG::2013-11-20 03:32:08,156::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15913518::INFO::2013-11-20 03:32:08,463::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:32:08,470::task::495::TaskManager.Task::(_debug) Task 639b875a-f2d9-40bb-a186-31e5cf75d3cf: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:32:08,473::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`71fc05db-135a-4181-b992-23ea4527cd64`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:32:08,474::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:32:08,476::resourceManager::481::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' found in shared state and queue is empty, Joining current shared lock (2 active users) >Thread-15913518::DEBUG::2013-11-20 03:32:08,476::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`71fc05db-135a-4181-b992-23ea4527cd64`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:32:08,479::task::495::TaskManager.Task::(_debug) Task 639b875a-f2d9-40bb-a186-31e5cf75d3cf: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:32:08,480::task::495::TaskManager.Task::(_debug) Task 639b875a-f2d9-40bb-a186-31e5cf75d3cf: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:32:08,481::lvm::480::OperationMutex::(_reloadlvs) Got the operational mutex >Thread-15913518::DEBUG::2013-11-20 03:32:08,482::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15914443::DEBUG::2013-11-20 03:32:08,483::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /sbin/service ksmtuned retune' (cwd None) >Thread-15914455::DEBUG::2013-11-20 03:32:09,063::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15914443::DEBUG::2013-11-20 03:32:09,382::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914443::DEBUG::2013-11-20 03:32:09,401::vm::358::vm.Vm::(_startUnderlyingVm) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::_ongoingCreations released >Thread-15914457::DEBUG::2013-11-20 03:32:09,431::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('5f587d5c-d4b4-4242-b049-ede331c7f8b4',) {} >Thread-15914457::DEBUG::2013-11-20 03:32:09,436::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::CPU stats not available >Thread-15914457::DEBUG::2013-11-20 03:32:09,437::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk stats not available >Thread-15914457::DEBUG::2013-11-20 03:32:09,440::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk latency not available >Thread-15914457::DEBUG::2013-11-20 03:32:09,442::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5523', 'displayIp': '0', 'displayPort': u'5912', 'session': 'Unknown', 'displaySecurePort': u'5913', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet7': {'macAddr': u'00:1a:4a:a8:64:17', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet7'}}, 'vmId': '5f587d5c-d4b4-4242-b049-ede331c7f8b4', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '56908316672', 'writeLatency': '0', 'imageID': '143cb26d-de08-4d30-89cf-aee10245b66a', 'flushLatency': '0', 'truesize': '56908316672'}}, 'boot': 'd', 'statsAge': '1.57', 'cpuIdle': '100.00', 'elapsedTime': '26', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '25201', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15914453::DEBUG::2013-11-20 03:32:09,510::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913518::DEBUG::2013-11-20 03:32:09,652::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914453::DEBUG::2013-11-20 03:32:09,848::task::495::TaskManager.Task::(_debug) Task d1b40b70-69bc-450c-93a4-44c03326fbab: finished: {'truesize': '53821308928', 'apparentsize': '53821308928'} >Thread-15914455::DEBUG::2013-11-20 03:32:09,861::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Thread-15914453::DEBUG::2013-11-20 03:32:09,874::task::495::TaskManager.Task::(_debug) Task d1b40b70-69bc-450c-93a4-44c03326fbab: moving from state preparing -> state finished >Thread-15914453::DEBUG::2013-11-20 03:32:09,947::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914458::DEBUG::2013-11-20 03:32:09,961::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15914453::DEBUG::2013-11-20 03:32:10,010::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914453::DEBUG::2013-11-20 03:32:10,332::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:32:10,336::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913518::DEBUG::2013-11-20 03:32:10,340::task::495::TaskManager.Task::(_debug) Task 639b875a-f2d9-40bb-a186-31e5cf75d3cf: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15914453::DEBUG::2013-11-20 03:32:10,340::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (1 active users) >Thread-15913518::DEBUG::2013-11-20 03:32:10,341::task::495::TaskManager.Task::(_debug) Task 639b875a-f2d9-40bb-a186-31e5cf75d3cf: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:32:10,343::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914453::DEBUG::2013-11-20 03:32:10,344::task::495::TaskManager.Task::(_debug) Task d1b40b70-69bc-450c-93a4-44c03326fbab: ref 0 aborting False >Thread-15913518::DEBUG::2013-11-20 03:32:10,345::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914453::INFO::2013-11-20 03:32:10,346::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '53821308928', 'apparentsize': '53821308928'} >Thread-15913518::DEBUG::2013-11-20 03:32:10,348::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:32:10,349::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:32:10,350::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:32:10,351::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:32:10,352::task::495::TaskManager.Task::(_debug) Task 639b875a-f2d9-40bb-a186-31e5cf75d3cf: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:32:10,353::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:32:10,355::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:32:10,356::task::495::TaskManager.Task::(_debug) Task f2b117d7-da7f-4e67-bd21-96ac61458849: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:32:10,359::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a691a3de-58e3-4e9c-9f68-02dd9a5af275`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:32:10,360::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:32:10,363::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:32:10,364::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a691a3de-58e3-4e9c-9f68-02dd9a5af275`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:32:10,366::task::495::TaskManager.Task::(_debug) Task f2b117d7-da7f-4e67-bd21-96ac61458849: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:32:10,367::task::495::TaskManager.Task::(_debug) Task f2b117d7-da7f-4e67-bd21-96ac61458849: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:32:10,368::task::495::TaskManager.Task::(_debug) Task f2b117d7-da7f-4e67-bd21-96ac61458849: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:32:10,368::task::495::TaskManager.Task::(_debug) Task f2b117d7-da7f-4e67-bd21-96ac61458849: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:32:10,369::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:32:10,370::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:32:10,371::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:32:10,372::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:32:10,373::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:32:10,374::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:32:10,375::task::495::TaskManager.Task::(_debug) Task f2b117d7-da7f-4e67-bd21-96ac61458849: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:32:10,376::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:32:10,451::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 1.9687 s, 520 kB/s\n'; <rc> = 0 >Thread-15914458::DEBUG::2013-11-20 03:32:11,079::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Dummy-15912651::DEBUG::2013-11-20 03:32:12,461::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914460::DEBUG::2013-11-20 03:32:12,463::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('5f587d5c-d4b4-4242-b049-ede331c7f8b4',) {} >Thread-15914460::DEBUG::2013-11-20 03:32:12,642::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk stats not available >Thread-15914460::DEBUG::2013-11-20 03:32:12,643::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk latency not available >Thread-15914460::DEBUG::2013-11-20 03:32:12,644::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5523', 'displayIp': '0', 'displayPort': u'5912', 'session': 'Unknown', 'displaySecurePort': u'5913', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet7': {'macAddr': u'00:1a:4a:a8:64:17', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet7'}}, 'vmId': '5f587d5c-d4b4-4242-b049-ede331c7f8b4', 'monitorResponse': '0', 'cpuUser': '81.89', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '56908316672', 'writeLatency': '0', 'imageID': '143cb26d-de08-4d30-89cf-aee10245b66a', 'flushLatency': '0', 'truesize': '56908316672'}}, 'boot': 'd', 'statsAge': '0.33', 'cpuIdle': '18.11', 'elapsedTime': '29', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '25201', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15914461::DEBUG::2013-11-20 03:32:12,651::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('163a3426-5ec5-46a5-b8f1-af06fc5d7e7d',) {} >Thread-15914461::DEBUG::2013-11-20 03:32:12,652::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::CPU stats not available >Thread-15914461::DEBUG::2013-11-20 03:32:12,653::libvirtvm::202::vm.Vm::(_getNetworkStats) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Network stats not available >Thread-15914461::DEBUG::2013-11-20 03:32:12,653::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk stats not available >Thread-15914461::DEBUG::2013-11-20 03:32:12,654::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk latency not available >Thread-15914461::DEBUG::2013-11-20 03:32:12,654::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5851', 'displayIp': '0', 'displayPort': u'5914', 'session': 'Unknown', 'displaySecurePort': u'5915', 'timeOffset': '0', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet8': {'state': 'unknown', 'macAddr': u'00:1a:4a:a8:64:15', 'speed': '1000', 'name': u'vnet8'}}, 'vmId': '163a3426-5ec5-46a5-b8f1-af06fc5d7e7d', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'hda': {'readLatency': '0', 'apparentsize': '53821308928', 'writeLatency': '0', 'imageID': '5f930ef3-763f-4294-a62a-6d52df220f8c', 'flushLatency': '0', 'truesize': '53821308928'}}, 'boot': 'c', 'statsAge': '0.29', 'cpuIdle': '100.00', 'elapsedTime': '14', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:32:12,955::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.05525 s, 18.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:32:14,972::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:32:15,882::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0789659 s, 13.0 MB/s\n'; <rc> = 0 >Thread-15914463::DEBUG::2013-11-20 03:32:15,968::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('5f587d5c-d4b4-4242-b049-ede331c7f8b4',) {} >Thread-15914463::DEBUG::2013-11-20 03:32:15,969::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk stats not available >Thread-15914463::DEBUG::2013-11-20 03:32:15,969::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk latency not available >Thread-15914463::DEBUG::2013-11-20 03:32:15,970::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5523', 'displayIp': '0', 'displayPort': u'5912', 'session': 'Unknown', 'displaySecurePort': u'5913', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet7': {'macAddr': u'00:1a:4a:a8:64:17', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet7'}}, 'vmId': '5f587d5c-d4b4-4242-b049-ede331c7f8b4', 'monitorResponse': '0', 'cpuUser': '81.89', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '56908316672', 'writeLatency': '0', 'imageID': '143cb26d-de08-4d30-89cf-aee10245b66a', 'flushLatency': '0', 'truesize': '56908316672'}}, 'boot': 'd', 'statsAge': '1.61', 'cpuIdle': '18.11', 'elapsedTime': '33', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '25201', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15914464::DEBUG::2013-11-20 03:32:15,975::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('163a3426-5ec5-46a5-b8f1-af06fc5d7e7d',) {} >Thread-15914464::DEBUG::2013-11-20 03:32:15,975::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::CPU stats not available >Thread-15914464::DEBUG::2013-11-20 03:32:15,976::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk stats not available >Thread-15914464::DEBUG::2013-11-20 03:32:15,977::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk latency not available >Thread-15914464::DEBUG::2013-11-20 03:32:15,978::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5851', 'displayIp': '0', 'displayPort': u'5914', 'session': 'Unknown', 'displaySecurePort': u'5915', 'timeOffset': '0', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet8': {'macAddr': u'00:1a:4a:a8:64:15', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet8'}}, 'vmId': '163a3426-5ec5-46a5-b8f1-af06fc5d7e7d', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'hda': {'readLatency': '0', 'apparentsize': '53821308928', 'writeLatency': '0', 'imageID': '5f930ef3-763f-4294-a62a-6d52df220f8c', 'flushLatency': '0', 'truesize': '53821308928'}}, 'boot': 'c', 'statsAge': '0.61', 'cpuIdle': '100.00', 'elapsedTime': '18', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15914465::DEBUG::2013-11-20 03:32:16,861::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914465::INFO::2013-11-20 03:32:16,862::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914465::DEBUG::2013-11-20 03:32:16,863::task::495::TaskManager.Task::(_debug) Task 77d54c13-52d5-49f7-afe7-8ad1a1f4cb1a: moving from state init -> state preparing >Thread-15914465::DEBUG::2013-11-20 03:32:16,863::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914465::DEBUG::2013-11-20 03:32:16,864::task::495::TaskManager.Task::(_debug) Task 77d54c13-52d5-49f7-afe7-8ad1a1f4cb1a: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914465::DEBUG::2013-11-20 03:32:16,864::task::495::TaskManager.Task::(_debug) Task 77d54c13-52d5-49f7-afe7-8ad1a1f4cb1a: moving from state preparing -> state finished >Thread-15914465::DEBUG::2013-11-20 03:32:16,865::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914465::DEBUG::2013-11-20 03:32:16,865::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914465::DEBUG::2013-11-20 03:32:16,865::task::495::TaskManager.Task::(_debug) Task 77d54c13-52d5-49f7-afe7-8ad1a1f4cb1a: ref 0 aborting False >Thread-15914465::INFO::2013-11-20 03:32:16,866::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914466::DEBUG::2013-11-20 03:32:16,871::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914466::INFO::2013-11-20 03:32:16,871::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914466::DEBUG::2013-11-20 03:32:16,872::task::495::TaskManager.Task::(_debug) Task d023d428-aaee-4340-948e-60569e6441f8: moving from state init -> state preparing >Thread-15914466::DEBUG::2013-11-20 03:32:16,873::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`aa5b793f-5224-403d-8276-14ee46515617`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914466::DEBUG::2013-11-20 03:32:16,873::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914466::DEBUG::2013-11-20 03:32:16,874::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914466::DEBUG::2013-11-20 03:32:16,875::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`aa5b793f-5224-403d-8276-14ee46515617`::Granted request >Thread-15914466::DEBUG::2013-11-20 03:32:16,876::task::495::TaskManager.Task::(_debug) Task d023d428-aaee-4340-948e-60569e6441f8: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914466::DEBUG::2013-11-20 03:32:16,876::task::495::TaskManager.Task::(_debug) Task d023d428-aaee-4340-948e-60569e6441f8: ref 1 aborting False >Thread-15914466::WARNING::2013-11-20 03:32:16,878::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914466::DEBUG::2013-11-20 03:32:16,879::task::495::TaskManager.Task::(_debug) Task d023d428-aaee-4340-948e-60569e6441f8: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914466::DEBUG::2013-11-20 03:32:16,879::task::495::TaskManager.Task::(_debug) Task d023d428-aaee-4340-948e-60569e6441f8: moving from state preparing -> state finished >Thread-15914466::DEBUG::2013-11-20 03:32:16,879::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914466::DEBUG::2013-11-20 03:32:16,880::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914466::DEBUG::2013-11-20 03:32:16,880::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914466::DEBUG::2013-11-20 03:32:16,881::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914466::DEBUG::2013-11-20 03:32:16,881::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914466::DEBUG::2013-11-20 03:32:16,881::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914466::DEBUG::2013-11-20 03:32:16,882::task::495::TaskManager.Task::(_debug) Task d023d428-aaee-4340-948e-60569e6441f8: ref 0 aborting False >Thread-15914466::INFO::2013-11-20 03:32:16,882::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:32:17,899::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:32:18,458::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0862256 s, 11.9 MB/s\n'; <rc> = 0 >Thread-15914468::DEBUG::2013-11-20 03:32:19,002::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('5f587d5c-d4b4-4242-b049-ede331c7f8b4',) {} >Thread-15914468::DEBUG::2013-11-20 03:32:19,003::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk stats not available >Thread-15914468::DEBUG::2013-11-20 03:32:19,004::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk latency not available >Thread-15914468::DEBUG::2013-11-20 03:32:19,005::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5523', 'displayIp': '0', 'displayPort': u'5912', 'session': 'Unknown', 'displaySecurePort': u'5913', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet7': {'macAddr': u'00:1a:4a:a8:64:17', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet7'}}, 'vmId': '5f587d5c-d4b4-4242-b049-ede331c7f8b4', 'monitorResponse': '0', 'cpuUser': '81.89', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '56908316672', 'writeLatency': '0', 'imageID': '143cb26d-de08-4d30-89cf-aee10245b66a', 'flushLatency': '0', 'truesize': '56908316672'}}, 'boot': 'd', 'statsAge': '2.59', 'cpuIdle': '18.11', 'elapsedTime': '36', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '25201', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15914469::DEBUG::2013-11-20 03:32:19,011::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('163a3426-5ec5-46a5-b8f1-af06fc5d7e7d',) {} >Thread-15914469::DEBUG::2013-11-20 03:32:19,012::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::CPU stats not available >Thread-15914469::DEBUG::2013-11-20 03:32:19,013::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk stats not available >Thread-15914469::DEBUG::2013-11-20 03:32:19,013::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk latency not available >Thread-15914469::DEBUG::2013-11-20 03:32:19,014::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5851', 'displayIp': '0', 'displayPort': u'5914', 'session': 'Unknown', 'displaySecurePort': u'5915', 'timeOffset': '0', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet8': {'macAddr': u'00:1a:4a:a8:64:15', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet8'}}, 'vmId': '163a3426-5ec5-46a5-b8f1-af06fc5d7e7d', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'hda': {'readLatency': '0', 'apparentsize': '53821308928', 'writeLatency': '0', 'imageID': '5f930ef3-763f-4294-a62a-6d52df220f8c', 'flushLatency': '0', 'truesize': '53821308928'}}, 'boot': 'c', 'statsAge': '0.64', 'cpuIdle': '100.00', 'elapsedTime': '21', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:32:20,475::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:32:21,012::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0492122 s, 20.8 MB/s\n'; <rc> = 0 >Thread-15914470::DEBUG::2013-11-20 03:32:21,132::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15913546::INFO::2013-11-20 03:32:21,137::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:32:21,139::task::495::TaskManager.Task::(_debug) Task 20d527f3-79bc-45f9-9939-76665de188ad: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:32:21,369::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`f58ac0e2-c6ad-4c0b-9a3b-bc42b75ce6f5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:32:21,371::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:32:21,372::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:32:21,372::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`f58ac0e2-c6ad-4c0b-9a3b-bc42b75ce6f5`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:32:21,373::task::495::TaskManager.Task::(_debug) Task 20d527f3-79bc-45f9-9939-76665de188ad: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:32:21,374::task::495::TaskManager.Task::(_debug) Task 20d527f3-79bc-45f9-9939-76665de188ad: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:32:21,375::task::495::TaskManager.Task::(_debug) Task 20d527f3-79bc-45f9-9939-76665de188ad: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:32:21,375::task::495::TaskManager.Task::(_debug) Task 20d527f3-79bc-45f9-9939-76665de188ad: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:32:21,376::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:32:21,376::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:32:21,377::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:32:21,377::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:32:21,377::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:32:21,378::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:32:21,378::task::495::TaskManager.Task::(_debug) Task 20d527f3-79bc-45f9-9939-76665de188ad: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:32:21,378::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:32:21,379::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:32:21,380::task::495::TaskManager.Task::(_debug) Task 51816d1a-badc-4a2b-9bf1-7bc68bb8d102: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:32:21,381::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`af74b6eb-9a79-4348-888c-3f625ba62182`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:32:21,381::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:32:21,382::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:32:21,382::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`af74b6eb-9a79-4348-888c-3f625ba62182`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:32:21,383::task::495::TaskManager.Task::(_debug) Task 51816d1a-badc-4a2b-9bf1-7bc68bb8d102: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:32:21,383::task::495::TaskManager.Task::(_debug) Task 51816d1a-badc-4a2b-9bf1-7bc68bb8d102: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:32:21,384::task::495::TaskManager.Task::(_debug) Task 51816d1a-badc-4a2b-9bf1-7bc68bb8d102: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:32:21,384::task::495::TaskManager.Task::(_debug) Task 51816d1a-badc-4a2b-9bf1-7bc68bb8d102: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:32:21,385::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:32:21,385::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:32:21,385::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:32:21,386::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:32:21,386::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:32:21,386::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:32:21,387::task::495::TaskManager.Task::(_debug) Task 51816d1a-badc-4a2b-9bf1-7bc68bb8d102: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:32:21,387::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:32:21,388::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:32:21,388::task::495::TaskManager.Task::(_debug) Task 61d1471f-3043-4ff5-b81d-24c8594ae2ac: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:32:21,389::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`b6c1ecbc-3c98-46e4-977d-ed766400a9a1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:32:21,389::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:32:21,389::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:32:21,390::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`b6c1ecbc-3c98-46e4-977d-ed766400a9a1`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:32:21,390::task::495::TaskManager.Task::(_debug) Task 61d1471f-3043-4ff5-b81d-24c8594ae2ac: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:32:21,391::task::495::TaskManager.Task::(_debug) Task 61d1471f-3043-4ff5-b81d-24c8594ae2ac: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:32:21,391::task::495::TaskManager.Task::(_debug) Task 61d1471f-3043-4ff5-b81d-24c8594ae2ac: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:32:21,391::task::495::TaskManager.Task::(_debug) Task 61d1471f-3043-4ff5-b81d-24c8594ae2ac: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:32:21,392::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:32:21,392::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:32:21,392::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:32:21,393::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:32:21,393::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:32:21,393::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:32:21,394::task::495::TaskManager.Task::(_debug) Task 61d1471f-3043-4ff5-b81d-24c8594ae2ac: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:32:21,394::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:32:21,395::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:32:21,395::task::495::TaskManager.Task::(_debug) Task ea18323e-d475-4f29-9a4c-8253aa1f6179: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:32:21,396::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`c7b16ad3-8de1-4449-b6f2-5b06bdebaa52`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:32:21,396::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:32:21,396::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:32:21,397::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`c7b16ad3-8de1-4449-b6f2-5b06bdebaa52`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:32:21,397::task::495::TaskManager.Task::(_debug) Task ea18323e-d475-4f29-9a4c-8253aa1f6179: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:32:21,398::task::495::TaskManager.Task::(_debug) Task ea18323e-d475-4f29-9a4c-8253aa1f6179: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:32:21,398::task::495::TaskManager.Task::(_debug) Task ea18323e-d475-4f29-9a4c-8253aa1f6179: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:32:21,398::task::495::TaskManager.Task::(_debug) Task ea18323e-d475-4f29-9a4c-8253aa1f6179: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:32:21,399::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:32:21,399::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:32:21,399::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:32:21,400::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:32:21,400::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:32:21,400::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:32:21,401::task::495::TaskManager.Task::(_debug) Task ea18323e-d475-4f29-9a4c-8253aa1f6179: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:32:21,401::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15914470::DEBUG::2013-11-20 03:32:21,818::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Thread-15914471::DEBUG::2013-11-20 03:32:21,824::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15914472::INFO::2013-11-20 03:32:22,104::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914472::DEBUG::2013-11-20 03:32:22,121::task::495::TaskManager.Task::(_debug) Task 21b1da7c-d2ef-4eb6-8bef-a2a14bd5cdb3: moving from state init -> state preparing >Thread-15914472::DEBUG::2013-11-20 03:32:22,123::task::495::TaskManager.Task::(_debug) Task 21b1da7c-d2ef-4eb6-8bef-a2a14bd5cdb3: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000314950942993', 'lastCheck': 1384918340.0713339, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00159001350403', 'lastCheck': 1384918335.3766799, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00109195709229', 'lastCheck': 1384918337.3727779, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00300216674805', 'lastCheck': 1384918338.136579, 'valid': True, 'code': 0}} >Thread-15914472::DEBUG::2013-11-20 03:32:22,154::task::495::TaskManager.Task::(_debug) Task 21b1da7c-d2ef-4eb6-8bef-a2a14bd5cdb3: moving from state preparing -> state finished >Thread-15914472::DEBUG::2013-11-20 03:32:22,166::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914472::DEBUG::2013-11-20 03:32:22,172::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914472::DEBUG::2013-11-20 03:32:22,181::task::495::TaskManager.Task::(_debug) Task 21b1da7c-d2ef-4eb6-8bef-a2a14bd5cdb3: ref 0 aborting False >Thread-15914472::INFO::2013-11-20 03:32:22,190::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000314950942993', 'lastCheck': 1384918340.0713339, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00159001350403', 'lastCheck': 1384918335.3766799, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00109195709229', 'lastCheck': 1384918337.3727779, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00300216674805', 'lastCheck': 1384918338.136579, 'valid': True, 'code': 0}} >Thread-15914473::DEBUG::2013-11-20 03:32:22,336::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::CPU stats not available >Thread-15914473::DEBUG::2013-11-20 03:32:22,348::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk stats not available >Thread-15914473::DEBUG::2013-11-20 03:32:22,355::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk latency not available >Thread-15914473::DEBUG::2013-11-20 03:32:22,379::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk stats not available >Thread-15914473::DEBUG::2013-11-20 03:32:22,386::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk latency not available >Thread-15914471::DEBUG::2013-11-20 03:32:22,457::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Dummy-15912651::DEBUG::2013-11-20 03:32:23,023::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:32:24,046::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0465804 s, 22.0 MB/s\n'; <rc> = 0 >Thread-15914475::DEBUG::2013-11-20 03:32:25,733::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('5f587d5c-d4b4-4242-b049-ede331c7f8b4',) {} >Thread-15914475::DEBUG::2013-11-20 03:32:25,745::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk stats not available >Thread-15914475::DEBUG::2013-11-20 03:32:25,762::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk latency not available >Thread-15914475::DEBUG::2013-11-20 03:32:25,775::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5523', 'displayIp': '0', 'displayPort': u'5912', 'session': 'Unknown', 'displaySecurePort': u'5913', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet7': {'macAddr': u'00:1a:4a:a8:64:17', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet7'}}, 'vmId': '5f587d5c-d4b4-4242-b049-ede331c7f8b4', 'monitorResponse': '0', 'cpuUser': '81.89', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '56908316672', 'writeLatency': '0', 'imageID': '143cb26d-de08-4d30-89cf-aee10245b66a', 'flushLatency': '0', 'truesize': '56908316672'}}, 'boot': 'd', 'statsAge': '2.36', 'cpuIdle': '18.11', 'elapsedTime': '42', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '25201', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15914476::DEBUG::2013-11-20 03:32:25,835::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('163a3426-5ec5-46a5-b8f1-af06fc5d7e7d',) {} >Thread-15914476::DEBUG::2013-11-20 03:32:25,837::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk stats not available >Thread-15914476::DEBUG::2013-11-20 03:32:25,838::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk latency not available >Thread-15914476::DEBUG::2013-11-20 03:32:25,838::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5851', 'displayIp': '0', 'displayPort': u'5914', 'session': 'Unknown', 'displaySecurePort': u'5915', 'timeOffset': '0', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet8': {'macAddr': u'00:1a:4a:a8:64:15', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet8'}}, 'vmId': '163a3426-5ec5-46a5-b8f1-af06fc5d7e7d', 'monitorResponse': '0', 'cpuUser': '59.86', 'disks': {u'hda': {'readLatency': '0', 'apparentsize': '53821308928', 'writeLatency': '0', 'imageID': '5f930ef3-763f-4294-a62a-6d52df220f8c', 'flushLatency': '0', 'truesize': '53821308928'}}, 'boot': 'c', 'statsAge': '0.45', 'cpuIdle': '40.14', 'elapsedTime': '28', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:32:26,056::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:32:26,608::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0927764 s, 11.0 MB/s\n'; <rc> = 0 >Thread-15914477::DEBUG::2013-11-20 03:32:27,039::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914477::INFO::2013-11-20 03:32:27,040::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914477::DEBUG::2013-11-20 03:32:27,041::task::495::TaskManager.Task::(_debug) Task b25a48b1-5c97-4369-8139-02e5895f06f9: moving from state init -> state preparing >Thread-15914477::DEBUG::2013-11-20 03:32:27,041::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914477::DEBUG::2013-11-20 03:32:27,042::task::495::TaskManager.Task::(_debug) Task b25a48b1-5c97-4369-8139-02e5895f06f9: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914477::DEBUG::2013-11-20 03:32:27,042::task::495::TaskManager.Task::(_debug) Task b25a48b1-5c97-4369-8139-02e5895f06f9: moving from state preparing -> state finished >Thread-15914477::DEBUG::2013-11-20 03:32:27,043::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914477::DEBUG::2013-11-20 03:32:27,043::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914477::DEBUG::2013-11-20 03:32:27,044::task::495::TaskManager.Task::(_debug) Task b25a48b1-5c97-4369-8139-02e5895f06f9: ref 0 aborting False >Thread-15914477::INFO::2013-11-20 03:32:27,044::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914478::DEBUG::2013-11-20 03:32:27,048::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914478::INFO::2013-11-20 03:32:27,049::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914478::DEBUG::2013-11-20 03:32:27,050::task::495::TaskManager.Task::(_debug) Task d87a59dd-021e-4d39-8006-81bfa6c29951: moving from state init -> state preparing >Thread-15914478::DEBUG::2013-11-20 03:32:27,051::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b21d3a7e-a6a5-42b6-9d6c-65dd3b7cb4c6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914478::DEBUG::2013-11-20 03:32:27,052::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914478::DEBUG::2013-11-20 03:32:27,053::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914478::DEBUG::2013-11-20 03:32:27,053::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b21d3a7e-a6a5-42b6-9d6c-65dd3b7cb4c6`::Granted request >Thread-15914478::DEBUG::2013-11-20 03:32:27,054::task::495::TaskManager.Task::(_debug) Task d87a59dd-021e-4d39-8006-81bfa6c29951: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914478::DEBUG::2013-11-20 03:32:27,055::task::495::TaskManager.Task::(_debug) Task d87a59dd-021e-4d39-8006-81bfa6c29951: ref 1 aborting False >Thread-15914478::WARNING::2013-11-20 03:32:27,057::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914478::DEBUG::2013-11-20 03:32:27,058::task::495::TaskManager.Task::(_debug) Task d87a59dd-021e-4d39-8006-81bfa6c29951: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914478::DEBUG::2013-11-20 03:32:27,058::task::495::TaskManager.Task::(_debug) Task d87a59dd-021e-4d39-8006-81bfa6c29951: moving from state preparing -> state finished >Thread-15914478::DEBUG::2013-11-20 03:32:27,059::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914478::DEBUG::2013-11-20 03:32:27,059::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914478::DEBUG::2013-11-20 03:32:27,060::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914478::DEBUG::2013-11-20 03:32:27,061::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914478::DEBUG::2013-11-20 03:32:27,061::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914478::DEBUG::2013-11-20 03:32:27,062::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914478::DEBUG::2013-11-20 03:32:27,063::task::495::TaskManager.Task::(_debug) Task d87a59dd-021e-4d39-8006-81bfa6c29951: ref 0 aborting False >Thread-15914478::INFO::2013-11-20 03:32:27,063::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:32:28,621::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914480::DEBUG::2013-11-20 03:32:28,891::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('5f587d5c-d4b4-4242-b049-ede331c7f8b4',) {} >Thread-15914480::DEBUG::2013-11-20 03:32:28,893::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk stats not available >Thread-15914480::DEBUG::2013-11-20 03:32:28,894::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk latency not available >Thread-15914480::DEBUG::2013-11-20 03:32:28,895::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5523', 'displayIp': '0', 'displayPort': u'5912', 'session': 'Unknown', 'displaySecurePort': u'5913', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet7': {'macAddr': u'00:1a:4a:a8:64:17', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet7'}}, 'vmId': '5f587d5c-d4b4-4242-b049-ede331c7f8b4', 'monitorResponse': '0', 'cpuUser': '188.92', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '56908316672', 'writeLatency': '0', 'imageID': '143cb26d-de08-4d30-89cf-aee10245b66a', 'flushLatency': '0', 'truesize': '56908316672'}}, 'boot': 'd', 'statsAge': '0.86', 'cpuIdle': '0.00', 'elapsedTime': '46', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '25201', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15914481::DEBUG::2013-11-20 03:32:28,901::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('163a3426-5ec5-46a5-b8f1-af06fc5d7e7d',) {} >Thread-15914481::DEBUG::2013-11-20 03:32:28,902::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk stats not available >Thread-15914481::DEBUG::2013-11-20 03:32:28,902::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk latency not available >Thread-15914481::DEBUG::2013-11-20 03:32:28,903::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5851', 'displayIp': '0', 'displayPort': u'5914', 'session': 'Unknown', 'displaySecurePort': u'5915', 'timeOffset': '0', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet8': {'macAddr': u'00:1a:4a:a8:64:15', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet8'}}, 'vmId': '163a3426-5ec5-46a5-b8f1-af06fc5d7e7d', 'monitorResponse': '0', 'cpuUser': '59.86', 'disks': {u'hda': {'readLatency': '0', 'apparentsize': '53821308928', 'writeLatency': '0', 'imageID': '5f930ef3-763f-4294-a62a-6d52df220f8c', 'flushLatency': '0', 'truesize': '53821308928'}}, 'boot': 'c', 'statsAge': '0.50', 'cpuIdle': '40.14', 'elapsedTime': '31', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:32:29,236::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0862576 s, 11.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:32:31,251::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914483::DEBUG::2013-11-20 03:32:31,940::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('5f587d5c-d4b4-4242-b049-ede331c7f8b4',) {} >Thread-15914483::DEBUG::2013-11-20 03:32:31,941::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk stats not available >Thread-15914483::DEBUG::2013-11-20 03:32:31,941::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk latency not available >Thread-15914483::DEBUG::2013-11-20 03:32:31,942::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5523', 'displayIp': '0', 'displayPort': u'5912', 'session': 'Unknown', 'displaySecurePort': u'5913', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet7': {'macAddr': u'00:1a:4a:a8:64:17', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet7'}}, 'vmId': '5f587d5c-d4b4-4242-b049-ede331c7f8b4', 'monitorResponse': '0', 'cpuUser': '188.92', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '56908316672', 'writeLatency': '0', 'imageID': '143cb26d-de08-4d30-89cf-aee10245b66a', 'flushLatency': '0', 'truesize': '56908316672'}}, 'boot': 'd', 'statsAge': '1.69', 'cpuIdle': '0.00', 'elapsedTime': '49', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '25201', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15914484::DEBUG::2013-11-20 03:32:31,947::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('163a3426-5ec5-46a5-b8f1-af06fc5d7e7d',) {} >Thread-15914484::DEBUG::2013-11-20 03:32:31,947::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk stats not available >Thread-15914484::DEBUG::2013-11-20 03:32:31,948::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk latency not available >Thread-15914484::DEBUG::2013-11-20 03:32:31,948::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5851', 'displayIp': '0', 'displayPort': u'5914', 'session': 'Unknown', 'displaySecurePort': u'5915', 'timeOffset': '0', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet8': {'macAddr': u'00:1a:4a:a8:64:15', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet8'}}, 'vmId': '163a3426-5ec5-46a5-b8f1-af06fc5d7e7d', 'monitorResponse': '0', 'cpuUser': '59.86', 'disks': {u'hda': {'readLatency': '0', 'apparentsize': '53821308928', 'writeLatency': '0', 'imageID': '5f930ef3-763f-4294-a62a-6d52df220f8c', 'flushLatency': '0', 'truesize': '53821308928'}}, 'boot': 'c', 'statsAge': '1.55', 'cpuIdle': '40.14', 'elapsedTime': '34', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913639::INFO::2013-11-20 03:32:32,309::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:32:32,310::task::495::TaskManager.Task::(_debug) Task 76ebffb5-efa6-4a66-a655-adff85194c91: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:32:32,313::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`8a43c8ca-be1b-4087-977c-2995c2ebbd25`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:32:32,314::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:32:32,315::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:32:32,316::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`8a43c8ca-be1b-4087-977c-2995c2ebbd25`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:32:32,329::task::495::TaskManager.Task::(_debug) Task 76ebffb5-efa6-4a66-a655-adff85194c91: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:32:32,330::task::495::TaskManager.Task::(_debug) Task 76ebffb5-efa6-4a66-a655-adff85194c91: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:32:32,331::task::495::TaskManager.Task::(_debug) Task 76ebffb5-efa6-4a66-a655-adff85194c91: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:32:32,332::task::495::TaskManager.Task::(_debug) Task 76ebffb5-efa6-4a66-a655-adff85194c91: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:32:32,333::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:32:32,334::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:32:32,335::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:32:32,336::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:32:32,338::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:32:32,338::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:32:32,340::task::495::TaskManager.Task::(_debug) Task 76ebffb5-efa6-4a66-a655-adff85194c91: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:32:32,341::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:32:32,506::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.126347 s, 8.1 MB/s\n'; <rc> = 0 >Thread-15914485::DEBUG::2013-11-20 03:32:32,538::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15914485::DEBUG::2013-11-20 03:32:33,650::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Thread-15914486::DEBUG::2013-11-20 03:32:33,655::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15914486::DEBUG::2013-11-20 03:32:34,409::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Dummy-15912651::DEBUG::2013-11-20 03:32:34,521::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914488::DEBUG::2013-11-20 03:32:34,976::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('5f587d5c-d4b4-4242-b049-ede331c7f8b4',) {} >Thread-15914488::DEBUG::2013-11-20 03:32:34,978::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk stats not available >Thread-15914488::DEBUG::2013-11-20 03:32:34,978::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk latency not available >Thread-15914488::DEBUG::2013-11-20 03:32:34,979::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5523', 'displayIp': '0', 'displayPort': u'5912', 'session': 'Unknown', 'displaySecurePort': u'5913', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet7': {'macAddr': u'00:1a:4a:a8:64:17', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet7'}}, 'vmId': '5f587d5c-d4b4-4242-b049-ede331c7f8b4', 'monitorResponse': '0', 'cpuUser': '188.92', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '56908316672', 'writeLatency': '0', 'imageID': '143cb26d-de08-4d30-89cf-aee10245b66a', 'flushLatency': '0', 'truesize': '56908316672'}}, 'boot': 'd', 'statsAge': '0.11', 'cpuIdle': '0.00', 'elapsedTime': '52', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '25201', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15914489::DEBUG::2013-11-20 03:32:34,985::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('163a3426-5ec5-46a5-b8f1-af06fc5d7e7d',) {} >Thread-15914489::DEBUG::2013-11-20 03:32:34,986::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk stats not available >Thread-15914489::DEBUG::2013-11-20 03:32:34,987::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk latency not available >Thread-15914489::DEBUG::2013-11-20 03:32:34,988::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5851', 'displayIp': '0', 'displayPort': u'5914', 'session': 'Unknown', 'displaySecurePort': u'5915', 'timeOffset': '0', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet8': {'macAddr': u'00:1a:4a:a8:64:15', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet8'}}, 'vmId': '163a3426-5ec5-46a5-b8f1-af06fc5d7e7d', 'monitorResponse': '0', 'cpuUser': '59.86', 'disks': {u'hda': {'readLatency': '0', 'apparentsize': '53821308928', 'writeLatency': '0', 'imageID': '5f930ef3-763f-4294-a62a-6d52df220f8c', 'flushLatency': '0', 'truesize': '53821308928'}}, 'boot': 'c', 'statsAge': '0.58', 'cpuIdle': '40.14', 'elapsedTime': '37', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:32:35,305::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0501967 s, 20.4 MB/s\n'; <rc> = 0 >Thread-15914490::DEBUG::2013-11-20 03:32:37,185::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914490::INFO::2013-11-20 03:32:37,187::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914490::DEBUG::2013-11-20 03:32:37,187::task::495::TaskManager.Task::(_debug) Task 631708b4-98d4-4832-86cc-3f5bb04bdb74: moving from state init -> state preparing >Thread-15914490::DEBUG::2013-11-20 03:32:37,188::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914490::DEBUG::2013-11-20 03:32:37,189::task::495::TaskManager.Task::(_debug) Task 631708b4-98d4-4832-86cc-3f5bb04bdb74: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914490::DEBUG::2013-11-20 03:32:37,190::task::495::TaskManager.Task::(_debug) Task 631708b4-98d4-4832-86cc-3f5bb04bdb74: moving from state preparing -> state finished >Thread-15914490::DEBUG::2013-11-20 03:32:37,190::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914490::DEBUG::2013-11-20 03:32:37,191::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914490::DEBUG::2013-11-20 03:32:37,192::task::495::TaskManager.Task::(_debug) Task 631708b4-98d4-4832-86cc-3f5bb04bdb74: ref 0 aborting False >Thread-15914490::INFO::2013-11-20 03:32:37,192::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914491::DEBUG::2013-11-20 03:32:37,197::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914491::INFO::2013-11-20 03:32:37,197::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914491::DEBUG::2013-11-20 03:32:37,198::task::495::TaskManager.Task::(_debug) Task 7882378a-b52c-4c8c-9924-0afbc74d492e: moving from state init -> state preparing >Thread-15914491::DEBUG::2013-11-20 03:32:37,199::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d494dd46-7ebe-4744-9d3a-0d3d43fcacc8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914491::DEBUG::2013-11-20 03:32:37,200::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914491::DEBUG::2013-11-20 03:32:37,201::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914491::DEBUG::2013-11-20 03:32:37,201::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d494dd46-7ebe-4744-9d3a-0d3d43fcacc8`::Granted request >Thread-15914491::DEBUG::2013-11-20 03:32:37,203::task::495::TaskManager.Task::(_debug) Task 7882378a-b52c-4c8c-9924-0afbc74d492e: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914491::DEBUG::2013-11-20 03:32:37,203::task::495::TaskManager.Task::(_debug) Task 7882378a-b52c-4c8c-9924-0afbc74d492e: ref 1 aborting False >Thread-15914491::WARNING::2013-11-20 03:32:37,205::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914491::DEBUG::2013-11-20 03:32:37,206::task::495::TaskManager.Task::(_debug) Task 7882378a-b52c-4c8c-9924-0afbc74d492e: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914491::DEBUG::2013-11-20 03:32:37,207::task::495::TaskManager.Task::(_debug) Task 7882378a-b52c-4c8c-9924-0afbc74d492e: moving from state preparing -> state finished >Thread-15914491::DEBUG::2013-11-20 03:32:37,208::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914491::DEBUG::2013-11-20 03:32:37,208::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914491::DEBUG::2013-11-20 03:32:37,209::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914491::DEBUG::2013-11-20 03:32:37,210::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914491::DEBUG::2013-11-20 03:32:37,210::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914491::DEBUG::2013-11-20 03:32:37,211::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914491::DEBUG::2013-11-20 03:32:37,211::task::495::TaskManager.Task::(_debug) Task 7882378a-b52c-4c8c-9924-0afbc74d492e: ref 0 aborting False >Thread-15914491::INFO::2013-11-20 03:32:37,212::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:32:37,314::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:32:37,833::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0516247 s, 19.8 MB/s\n'; <rc> = 0 >Thread-15914492::INFO::2013-11-20 03:32:38,008::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914492::DEBUG::2013-11-20 03:32:38,009::task::495::TaskManager.Task::(_debug) Task aa4d4e37-ebcb-4484-a3b6-a522f9b8f8b8: moving from state init -> state preparing >Thread-15914492::DEBUG::2013-11-20 03:32:38,009::task::495::TaskManager.Task::(_debug) Task aa4d4e37-ebcb-4484-a3b6-a522f9b8f8b8: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000453948974609', 'lastCheck': 1384918350.2863319, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00128388404846', 'lastCheck': 1384918355.493988, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00121712684631', 'lastCheck': 1384918357.4968879, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000419139862061', 'lastCheck': 1384918348.2810299, 'valid': True, 'code': 0}} >Thread-15914492::DEBUG::2013-11-20 03:32:38,010::task::495::TaskManager.Task::(_debug) Task aa4d4e37-ebcb-4484-a3b6-a522f9b8f8b8: moving from state preparing -> state finished >Thread-15914492::DEBUG::2013-11-20 03:32:38,011::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914492::DEBUG::2013-11-20 03:32:38,011::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914492::DEBUG::2013-11-20 03:32:38,012::task::495::TaskManager.Task::(_debug) Task aa4d4e37-ebcb-4484-a3b6-a522f9b8f8b8: ref 0 aborting False >Thread-15914492::INFO::2013-11-20 03:32:38,012::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000453948974609', 'lastCheck': 1384918350.2863319, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00128388404846', 'lastCheck': 1384918355.493988, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00121712684631', 'lastCheck': 1384918357.4968879, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000419139862061', 'lastCheck': 1384918348.2810299, 'valid': True, 'code': 0}} >Thread-15914493::DEBUG::2013-11-20 03:32:38,058::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk stats not available >Thread-15914493::DEBUG::2013-11-20 03:32:38,058::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk latency not available >Thread-15914493::DEBUG::2013-11-20 03:32:38,060::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk stats not available >Thread-15914493::DEBUG::2013-11-20 03:32:38,061::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk latency not available >Dummy-15912651::DEBUG::2013-11-20 03:32:39,843::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:32:40,281::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0523558 s, 19.6 MB/s\n'; <rc> = 0 >Thread-15914495::DEBUG::2013-11-20 03:32:41,511::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('5f587d5c-d4b4-4242-b049-ede331c7f8b4',) {} >Thread-15914495::DEBUG::2013-11-20 03:32:41,512::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk stats not available >Thread-15914495::DEBUG::2013-11-20 03:32:41,512::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk latency not available >Thread-15914495::DEBUG::2013-11-20 03:32:41,513::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5523', 'displayIp': '0', 'displayPort': u'5912', 'session': 'Unknown', 'displaySecurePort': u'5913', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet7': {'macAddr': u'00:1a:4a:a8:64:17', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet7'}}, 'vmId': '5f587d5c-d4b4-4242-b049-ede331c7f8b4', 'monitorResponse': '0', 'cpuUser': '188.92', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '56908316672', 'writeLatency': '0', 'imageID': '143cb26d-de08-4d30-89cf-aee10245b66a', 'flushLatency': '0', 'truesize': '56908316672'}}, 'boot': 'd', 'statsAge': '0.21', 'cpuIdle': '0.00', 'elapsedTime': '58', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '25201', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15914496::DEBUG::2013-11-20 03:32:41,518::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('163a3426-5ec5-46a5-b8f1-af06fc5d7e7d',) {} >Thread-15914496::DEBUG::2013-11-20 03:32:41,520::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk stats not available >Thread-15914496::DEBUG::2013-11-20 03:32:41,521::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk latency not available >Thread-15914496::DEBUG::2013-11-20 03:32:41,522::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5851', 'displayIp': '0', 'displayPort': u'5914', 'session': 'Unknown', 'displaySecurePort': u'5915', 'timeOffset': '0', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet8': {'macAddr': u'00:1a:4a:a8:64:15', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet8'}}, 'vmId': '163a3426-5ec5-46a5-b8f1-af06fc5d7e7d', 'monitorResponse': '0', 'cpuUser': '73.22', 'disks': {u'hda': {'readLatency': '0', 'apparentsize': '53821308928', 'writeLatency': '0', 'imageID': '5f930ef3-763f-4294-a62a-6d52df220f8c', 'flushLatency': '0', 'truesize': '53821308928'}}, 'boot': 'c', 'statsAge': '1.11', 'cpuIdle': '26.78', 'elapsedTime': '43', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15914497::DEBUG::2013-11-20 03:32:42,276::clientIF::54::vds::(wrapper) [192.168.100.135]::call create with ({'bridge': 'rhevm', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.2.0', 'vmId': 'e97d1c52-b688-4192-bdeb-2dfadcc79786', 'transparentHugePages': 'true', 'spiceSslCipherSuite': 'DEFAULT', 'cpuType': 'Westmere', 'smp': '4', 'macAddr': '00:1a:4a:a8:64:13', 'boot': 'c', 'custom': {}, 'vmType': 'kvm', 'memSize': 4500, 'smpCoresPerSocket': '1', 'vmName': 'sukhumi', 'spiceMonitors': '1', 'nice': '0', 'drives': [{'index': '0', 'domainID': '6b892067-ac7e-4039-a7be-5b7387525b16', 'format': 'raw', 'volumeID': '8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f', 'imageID': '5af747d8-95b1-4b7e-af53-cb77e4fcc896', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'ide'}], 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/RHEV-toolsSetup_3.0_39.iso', 'smartcardEnable': 'false', 'nicModel': 'pv', 'keyboardLayout': 'en-us', 'kvmEnable': 'true', 'timeOffset': '0', 'spiceSecureChannels': 'smain,sinputs', 'display': 'qxl'},) {} >Dummy-15912651::DEBUG::2013-11-20 03:32:42,452::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914497::INFO::2013-11-20 03:32:42,807::clientIF::523::vds::(_getNetworkIp) network None: using 0 >Thread-15914497::INFO::2013-11-20 03:32:42,809::clientIF::656::vds::(create) vmContainerLock acquired by vm e97d1c52-b688-4192-bdeb-2dfadcc79786 >Thread-15914497::DEBUG::2013-11-20 03:32:42,815::clientIF::669::vds::(create) Total desktops after creation of e97d1c52-b688-4192-bdeb-2dfadcc79786 is 9 >Thread-15914498::DEBUG::2013-11-20 03:32:42,815::vm::342::vm.Vm::(_startUnderlyingVm) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Start >Thread-15914497::DEBUG::2013-11-20 03:32:42,829::clientIF::59::vds::(wrapper) return create with {'status': {'message': 'Done', 'code': 0}, 'vmList': {'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/RHEV-toolsSetup_3.0_39.iso', 'bridge': 'rhevm', 'acpiEnable': 'true', 'emulatedMachine': 'rhel6.2.0', 'vmId': 'e97d1c52-b688-4192-bdeb-2dfadcc79786', 'pid': '0', 'drives': [{'index': '0', 'domainID': '6b892067-ac7e-4039-a7be-5b7387525b16', 'format': 'raw', 'volumeID': '8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f', 'imageID': '5af747d8-95b1-4b7e-af53-cb77e4fcc896', 'poolID': 'c4e5889a-348d-11e2-83fe-9f72369eab30', 'propagateErrors': 'off', 'if': 'ide'}], 'transparentHugePages': 'true', 'keyboardLayout': 'en-us', 'displaySecurePort': '-1', 'displayPort': '-1', 'cpuType': 'Westmere', 'custom': {}, 'clientIp': '', 'nicModel': 'pv', 'macAddr': '00:1a:4a:a8:64:13', 'kvmEnable': 'true', 'smartcardEnable': 'false', 'boot': 'c', 'smp': '4', 'vmType': 'kvm', 'spiceSslCipherSuite': 'DEFAULT', 'status': 'WaitForLaunch', 'timeOffset': '0', 'memSize': 4500, 'displayIp': '0', 'spiceSecureChannels': 'smain,sinputs', 'smpCoresPerSocket': '1', 'vmName': 'sukhumi', 'display': 'qxl', 'spiceMonitors': '1', 'nice': '0'}} >Thread-15914498::DEBUG::2013-11-20 03:32:42,836::vm::346::vm.Vm::(_startUnderlyingVm) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::_ongoingCreations acquired >Thread-15914498::INFO::2013-11-20 03:32:42,848::libvirtvm::1061::vm.Vm::(_run) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::VM wrapper has started >Thread-15914498::INFO::2013-11-20 03:32:42,865::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=5af747d8-95b1-4b7e-af53-cb77e4fcc896 volUUID=8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f) >Thread-15914498::DEBUG::2013-11-20 03:32:42,866::task::495::TaskManager.Task::(_debug) Task f377e2e2-4ed8-4891-9481-7df918a98f8a: moving from state init -> state preparing >Thread-15914498::DEBUG::2013-11-20 03:32:42,903::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ae180d1d-0e46-4f2f-aff1-5192bd351520`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1690' at 'public_prepareVolume' >Thread-15914498::DEBUG::2013-11-20 03:32:42,915::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914498::DEBUG::2013-11-20 03:32:42,928::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914498::DEBUG::2013-11-20 03:32:42,942::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ae180d1d-0e46-4f2f-aff1-5192bd351520`::Granted request >Thread-15914498::DEBUG::2013-11-20 03:32:42,961::task::495::TaskManager.Task::(_debug) Task f377e2e2-4ed8-4891-9481-7df918a98f8a: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914498::DEBUG::2013-11-20 03:32:42,963::task::495::TaskManager.Task::(_debug) Task f377e2e2-4ed8-4891-9481-7df918a98f8a: ref 1 aborting False >Thread-15914498::DEBUG::2013-11-20 03:32:42,974::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.5af747d8-95b1-4b7e-af53-cb77e4fcc896`ReqID=`b43d2d66-0a05-49bc-9593-246563eeebaf`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1695' at 'public_prepareVolume' >Thread-15914498::DEBUG::2013-11-20 03:32:42,974::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.5af747d8-95b1-4b7e-af53-cb77e4fcc896' for lock type 'exclusive' >Thread-15914498::DEBUG::2013-11-20 03:32:42,992::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15914498::DEBUG::2013-11-20 03:32:43,017::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:32:43,402::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0781838 s, 13.1 MB/s\n'; <rc> = 0 >Thread-15914498::DEBUG::2013-11-20 03:32:44,382::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:32:45,954::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914500::DEBUG::2013-11-20 03:32:46,886::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15914502::DEBUG::2013-11-20 03:32:50,858::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914502::INFO::2013-11-20 03:32:54,116::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914502::DEBUG::2013-11-20 03:32:54,194::task::495::TaskManager.Task::(_debug) Task 286296e7-6ae2-4a8a-a4bb-dafbead492a5: moving from state init -> state preparing >Thread-15914502::DEBUG::2013-11-20 03:32:54,391::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914502::DEBUG::2013-11-20 03:32:54,497::task::495::TaskManager.Task::(_debug) Task 286296e7-6ae2-4a8a-a4bb-dafbead492a5: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914498::DEBUG::2013-11-20 03:32:54,595::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15914502::DEBUG::2013-11-20 03:32:54,603::task::495::TaskManager.Task::(_debug) Task 286296e7-6ae2-4a8a-a4bb-dafbead492a5: moving from state preparing -> state finished >Thread-15914502::DEBUG::2013-11-20 03:32:54,605::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914498::DEBUG::2013-11-20 03:32:54,610::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=29 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15914502::DEBUG::2013-11-20 03:32:54,611::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914502::DEBUG::2013-11-20 03:32:54,612::task::495::TaskManager.Task::(_debug) Task 286296e7-6ae2-4a8a-a4bb-dafbead492a5: ref 0 aborting False >Thread-15914502::INFO::2013-11-20 03:32:54,779::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914503::DEBUG::2013-11-20 03:32:54,822::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('5f587d5c-d4b4-4242-b049-ede331c7f8b4',) {} >Thread-15914503::DEBUG::2013-11-20 03:32:54,824::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk stats not available >Thread-15914504::DEBUG::2013-11-20 03:32:54,829::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914503::DEBUG::2013-11-20 03:32:54,830::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk latency not available >Thread-15914504::INFO::2013-11-20 03:32:54,836::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914503::DEBUG::2013-11-20 03:32:54,836::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5523', 'displayIp': '0', 'displayPort': u'5912', 'session': 'Unknown', 'displaySecurePort': u'5913', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/SW_DVD5_Win_Svr_Std_and_DataCtr_2012_64Bit_English_Core_MLF_X18-27588.ISO', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet7': {'macAddr': u'00:1a:4a:a8:64:17', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet7'}}, 'vmId': '5f587d5c-d4b4-4242-b049-ede331c7f8b4', 'monitorResponse': '0', 'cpuUser': '192.29', 'disks': {u'vda': {'readLatency': '0', 'apparentsize': '56908316672', 'writeLatency': '0', 'imageID': '143cb26d-de08-4d30-89cf-aee10245b66a', 'flushLatency': '0', 'truesize': '56908316672'}}, 'boot': 'd', 'statsAge': '5.58', 'cpuIdle': '0.00', 'elapsedTime': '71', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '25201', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15914504::DEBUG::2013-11-20 03:32:54,838::task::495::TaskManager.Task::(_debug) Task 533b0c7b-2bf4-4843-831a-5636a6546cd2: moving from state init -> state preparing >Thread-15914504::DEBUG::2013-11-20 03:32:54,843::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3504ded7-d43f-4422-b6ef-b2fbaf52edfe`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914504::DEBUG::2013-11-20 03:32:54,843::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914504::DEBUG::2013-11-20 03:32:54,845::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914504::DEBUG::2013-11-20 03:32:54,846::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`3504ded7-d43f-4422-b6ef-b2fbaf52edfe`::Granted request >Thread-15914504::DEBUG::2013-11-20 03:32:54,852::task::495::TaskManager.Task::(_debug) Task 533b0c7b-2bf4-4843-831a-5636a6546cd2: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914504::DEBUG::2013-11-20 03:32:54,852::task::495::TaskManager.Task::(_debug) Task 533b0c7b-2bf4-4843-831a-5636a6546cd2: ref 1 aborting False >Thread-15914504::WARNING::2013-11-20 03:32:54,863::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914505::DEBUG::2013-11-20 03:32:54,867::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('163a3426-5ec5-46a5-b8f1-af06fc5d7e7d',) {} >Thread-15914504::DEBUG::2013-11-20 03:32:54,870::task::495::TaskManager.Task::(_debug) Task 533b0c7b-2bf4-4843-831a-5636a6546cd2: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914505::DEBUG::2013-11-20 03:32:54,878::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk stats not available >Thread-15914504::DEBUG::2013-11-20 03:32:54,883::task::495::TaskManager.Task::(_debug) Task 533b0c7b-2bf4-4843-831a-5636a6546cd2: moving from state preparing -> state finished >Thread-15914505::DEBUG::2013-11-20 03:32:54,884::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk latency not available >Thread-15914504::DEBUG::2013-11-20 03:32:54,886::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914505::DEBUG::2013-11-20 03:32:54,891::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5851', 'displayIp': '0', 'displayPort': u'5914', 'session': 'Unknown', 'displaySecurePort': u'5915', 'timeOffset': '0', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet8': {'macAddr': u'00:1a:4a:a8:64:15', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet8'}}, 'vmId': '163a3426-5ec5-46a5-b8f1-af06fc5d7e7d', 'monitorResponse': '0', 'cpuUser': '73.22', 'disks': {u'hda': {'readLatency': '0', 'apparentsize': '53821308928', 'writeLatency': '0', 'imageID': '5f930ef3-763f-4294-a62a-6d52df220f8c', 'flushLatency': '0', 'truesize': '53821308928'}}, 'boot': 'c', 'statsAge': '1.49', 'cpuIdle': '26.78', 'elapsedTime': '57', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15914504::DEBUG::2013-11-20 03:32:54,892::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914504::DEBUG::2013-11-20 03:32:54,899::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914504::DEBUG::2013-11-20 03:32:54,901::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914504::DEBUG::2013-11-20 03:32:54,901::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914504::DEBUG::2013-11-20 03:32:54,902::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914504::DEBUG::2013-11-20 03:32:54,903::task::495::TaskManager.Task::(_debug) Task 533b0c7b-2bf4-4843-831a-5636a6546cd2: ref 0 aborting False >Thread-15914504::INFO::2013-11-20 03:32:54,908::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:32:54,996::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 7.11684 s, 144 kB/s\n'; <rc> = 0 >Thread-15914498::DEBUG::2013-11-20 03:32:55,716::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.00870098 s, 58.8 kB/s\n'; <rc> = 0 >Thread-15914498::DEBUG::2013-11-20 03:32:55,716::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.00870098 s, 58.8 kB/s'], size: 512 >Thread-15914498::DEBUG::2013-11-20 03:32:55,718::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=29 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15914498::DEBUG::2013-11-20 03:32:56,042::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000139037 s, 3.7 MB/s\n'; <rc> = 0 >Thread-15914498::DEBUG::2013-11-20 03:32:56,042::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000139037 s, 3.7 MB/s'], size: 512 >Thread-15914498::INFO::2013-11-20 03:32:56,043::image::340::Storage.Image::(getChain) sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 imgUUID=5af747d8-95b1-4b7e-af53-cb77e4fcc896 chain=[<storage.blockVolume.BlockVolume instance at 0x7f487c0ab7e8>] >Thread-15914498::DEBUG::2013-11-20 03:32:56,047::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --autobackup n --available y 6b892067-ac7e-4039-a7be-5b7387525b16/8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f' (cwd None) >Thread-15914498::DEBUG::2013-11-20 03:32:56,800::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914498::DEBUG::2013-11-20 03:32:56,807::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15914498::DEBUG::2013-11-20 03:32:56,807::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15914498::DEBUG::2013-11-20 03:32:56,817::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f`ReqID=`de772223-8613-4c21-ba66-5a617e453a0a`::Request was made in '/usr/share/vdsm/storage/resourceFactories.py' line '149' at '__getResourceCandidatesList' >Thread-15914498::DEBUG::2013-11-20 03:32:56,818::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f' for lock type 'exclusive' >Thread-15914498::DEBUG::2013-11-20 03:32:56,822::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f' is free. Now locking as 'exclusive' (1 active user) >Thread-15914498::DEBUG::2013-11-20 03:32:56,822::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_volumeNS.8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f`ReqID=`de772223-8613-4c21-ba66-5a617e453a0a`::Granted request >Thread-15914498::DEBUG::2013-11-20 03:32:56,823::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.5af747d8-95b1-4b7e-af53-cb77e4fcc896' is free. Now locking as 'exclusive' (1 active user) >Thread-15914498::DEBUG::2013-11-20 03:32:56,824::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_imageNS.5af747d8-95b1-4b7e-af53-cb77e4fcc896`ReqID=`b43d2d66-0a05-49bc-9593-246563eeebaf`::Granted request >Thread-15914498::DEBUG::2013-11-20 03:32:56,830::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15914498::DEBUG::2013-11-20 03:32:56,845::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:32:57,006::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913658::INFO::2013-11-20 03:32:57,396::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:32:57,397::task::495::TaskManager.Task::(_debug) Task 5fbcf04f-9d83-4ec3-ae83-6c1f75a118b5: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:32:57,399::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`53d40546-66a4-406d-a8cb-bf7205e16982`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:32:57,400::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:32:57,401::resourceManager::481::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' found in shared state and queue is empty, Joining current shared lock (2 active users) >Thread-15913658::DEBUG::2013-11-20 03:32:57,401::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`53d40546-66a4-406d-a8cb-bf7205e16982`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:32:57,402::task::495::TaskManager.Task::(_debug) Task 5fbcf04f-9d83-4ec3-ae83-6c1f75a118b5: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:32:57,402::task::495::TaskManager.Task::(_debug) Task 5fbcf04f-9d83-4ec3-ae83-6c1f75a118b5: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:32:57,403::task::495::TaskManager.Task::(_debug) Task 5fbcf04f-9d83-4ec3-ae83-6c1f75a118b5: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:32:57,403::task::495::TaskManager.Task::(_debug) Task 5fbcf04f-9d83-4ec3-ae83-6c1f75a118b5: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:32:57,404::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:32:57,404::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:32:57,405::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:32:57,405::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (1 active users) >Thread-15913658::DEBUG::2013-11-20 03:32:57,406::task::495::TaskManager.Task::(_debug) Task 5fbcf04f-9d83-4ec3-ae83-6c1f75a118b5: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:32:57,406::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15914498::DEBUG::2013-11-20 03:32:57,691::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15913695::INFO::2013-11-20 03:33:00,219::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:33:00,443::task::495::TaskManager.Task::(_debug) Task 8472d58f-9926-4cbd-8277-15dbf3e0c8be: moving from state init -> state preparing >Thread-15914500::DEBUG::2013-11-20 03:33:00,950::clientIF::1219::vds::(fenceNode) rc 0 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = On >Done > err >Thread-15913695::DEBUG::2013-11-20 03:33:01,320::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`8fdc5ce3-219b-4861-a6ca-2e1f0a1fea1c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914507::DEBUG::2013-11-20 03:33:01,385::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('163a3426-5ec5-46a5-b8f1-af06fc5d7e7d',) {} >Thread-15913695::DEBUG::2013-11-20 03:33:01,391::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914507::DEBUG::2013-11-20 03:33:01,669::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk stats not available >Thread-15914507::DEBUG::2013-11-20 03:33:01,853::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk latency not available >Thread-15914508::DEBUG::2013-11-20 03:33:01,893::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15913695::DEBUG::2013-11-20 03:33:01,967::resourceManager::481::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' found in shared state and queue is empty, Joining current shared lock (2 active users) >Thread-15914507::DEBUG::2013-11-20 03:33:02,003::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '5851', 'displayIp': '0', 'displayPort': u'5914', 'session': 'Unknown', 'displaySecurePort': u'5915', 'timeOffset': '0', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet8': {'macAddr': u'00:1a:4a:a8:64:15', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet8'}}, 'vmId': '163a3426-5ec5-46a5-b8f1-af06fc5d7e7d', 'monitorResponse': '0', 'cpuUser': '365.29', 'disks': {u'hda': {'readLatency': '0', 'apparentsize': '53821308928', 'writeLatency': '0', 'imageID': '5f930ef3-763f-4294-a62a-6d52df220f8c', 'flushLatency': '0', 'truesize': '53821308928'}}, 'boot': 'c', 'statsAge': '1.92', 'cpuIdle': '0.00', 'elapsedTime': '63', 'vmType': 'kvm', 'cpuSys': '0.00', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15914498::DEBUG::2013-11-20 03:33:02,226::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15913695::DEBUG::2013-11-20 03:33:02,226::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`8fdc5ce3-219b-4861-a6ca-2e1f0a1fea1c`::Granted request >Thread-15912392::INFO::2013-11-20 03:33:02,227::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:33:02,418::task::495::TaskManager.Task::(_debug) Task 883052a6-83f9-483c-9906-61f82073285e: moving from state init -> state preparing >Thread-15914498::DEBUG::2013-11-20 03:33:02,421::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --refresh 6b892067-ac7e-4039-a7be-5b7387525b16/8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f' (cwd None) >Thread-15913695::DEBUG::2013-11-20 03:33:02,424::task::495::TaskManager.Task::(_debug) Task 8472d58f-9926-4cbd-8277-15dbf3e0c8be: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Dummy-15912651::DEBUG::2013-11-20 03:33:02,530::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 3.5718 s, 287 kB/s\n'; <rc> = 0 >Thread-15912392::DEBUG::2013-11-20 03:33:02,542::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`7565d92b-481e-4a0b-b3ee-7d67d494d39e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:33:02,543::task::495::TaskManager.Task::(_debug) Task 8472d58f-9926-4cbd-8277-15dbf3e0c8be: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:33:02,544::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:33:02,738::task::495::TaskManager.Task::(_debug) Task 8472d58f-9926-4cbd-8277-15dbf3e0c8be: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:33:02,740::task::495::TaskManager.Task::(_debug) Task 8472d58f-9926-4cbd-8277-15dbf3e0c8be: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:33:02,741::resourceManager::481::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' found in shared state and queue is empty, Joining current shared lock (3 active users) >Thread-15913695::DEBUG::2013-11-20 03:33:02,741::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:33:02,742::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`7565d92b-481e-4a0b-b3ee-7d67d494d39e`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:33:02,743::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:33:02,745::task::495::TaskManager.Task::(_debug) Task 883052a6-83f9-483c-9906-61f82073285e: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:33:02,745::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:33:02,747::task::495::TaskManager.Task::(_debug) Task 883052a6-83f9-483c-9906-61f82073285e: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:33:02,749::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (2 active users) >Thread-15912392::DEBUG::2013-11-20 03:33:02,749::task::495::TaskManager.Task::(_debug) Task 883052a6-83f9-483c-9906-61f82073285e: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:33:02,752::task::495::TaskManager.Task::(_debug) Task 883052a6-83f9-483c-9906-61f82073285e: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:33:02,753::task::495::TaskManager.Task::(_debug) Task 8472d58f-9926-4cbd-8277-15dbf3e0c8be: ref 0 aborting False >Thread-15912392::DEBUG::2013-11-20 03:33:02,754::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::INFO::2013-11-20 03:33:02,755::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15912392::DEBUG::2013-11-20 03:33:02,755::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:33:02,756::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:33:02,757::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (1 active users) >Thread-15912392::DEBUG::2013-11-20 03:33:02,758::task::495::TaskManager.Task::(_debug) Task 883052a6-83f9-483c-9906-61f82073285e: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:33:02,758::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15914508::DEBUG::2013-11-20 03:33:03,004::clientIF::1219::vds::(fenceNode) rc 0 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = On >Done > err >Thread-15914498::DEBUG::2013-11-20 03:33:04,150::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914498::DEBUG::2013-11-20 03:33:04,154::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15914498::DEBUG::2013-11-20 03:33:04,154::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15914498::INFO::2013-11-20 03:33:04,155::volume::517::Storage.Volume::(prepare) Volume: preparing volume 6b892067-ac7e-4039-a7be-5b7387525b16/8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f >Thread-15914498::DEBUG::2013-11-20 03:33:04,156::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15914498::DEBUG::2013-11-20 03:33:04,156::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:04,740::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914498::DEBUG::2013-11-20 03:33:04,955::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914509::DEBUG::2013-11-20 03:33:05,927::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914509::INFO::2013-11-20 03:33:06,221::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914509::DEBUG::2013-11-20 03:33:06,363::task::495::TaskManager.Task::(_debug) Task 77ffc288-d967-43c1-acdd-0cf34a563e78: moving from state init -> state preparing >Thread-15914509::DEBUG::2013-11-20 03:33:06,441::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914498::DEBUG::2013-11-20 03:33:06,618::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15914509::DEBUG::2013-11-20 03:33:06,620::task::495::TaskManager.Task::(_debug) Task 77ffc288-d967-43c1-acdd-0cf34a563e78: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914498::DEBUG::2013-11-20 03:33:06,625::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=29 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15914509::DEBUG::2013-11-20 03:33:06,625::task::495::TaskManager.Task::(_debug) Task 77ffc288-d967-43c1-acdd-0cf34a563e78: moving from state preparing -> state finished >Thread-15914509::DEBUG::2013-11-20 03:33:06,791::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914509::DEBUG::2013-11-20 03:33:06,793::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914509::DEBUG::2013-11-20 03:33:06,794::task::495::TaskManager.Task::(_debug) Task 77ffc288-d967-43c1-acdd-0cf34a563e78: ref 0 aborting False >Thread-15914509::INFO::2013-11-20 03:33:06,794::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914511::DEBUG::2013-11-20 03:33:06,821::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914511::INFO::2013-11-20 03:33:06,822::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914511::DEBUG::2013-11-20 03:33:06,823::task::495::TaskManager.Task::(_debug) Task 76919e73-3eaa-4e27-b2a8-2abd639c0020: moving from state init -> state preparing >Thread-15914511::DEBUG::2013-11-20 03:33:06,829::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2c514c08-b11f-4b22-b59a-cbbd1e16704d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914511::DEBUG::2013-11-20 03:33:06,830::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914511::DEBUG::2013-11-20 03:33:06,831::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914511::DEBUG::2013-11-20 03:33:06,834::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2c514c08-b11f-4b22-b59a-cbbd1e16704d`::Granted request >Thread-15914511::DEBUG::2013-11-20 03:33:06,844::task::495::TaskManager.Task::(_debug) Task 76919e73-3eaa-4e27-b2a8-2abd639c0020: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914511::DEBUG::2013-11-20 03:33:06,845::task::495::TaskManager.Task::(_debug) Task 76919e73-3eaa-4e27-b2a8-2abd639c0020: ref 1 aborting False >Thread-15914511::WARNING::2013-11-20 03:33:06,851::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914511::DEBUG::2013-11-20 03:33:06,872::task::495::TaskManager.Task::(_debug) Task 76919e73-3eaa-4e27-b2a8-2abd639c0020: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914511::DEBUG::2013-11-20 03:33:06,896::task::495::TaskManager.Task::(_debug) Task 76919e73-3eaa-4e27-b2a8-2abd639c0020: moving from state preparing -> state finished >Thread-15914511::DEBUG::2013-11-20 03:33:06,898::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914511::DEBUG::2013-11-20 03:33:06,899::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914511::DEBUG::2013-11-20 03:33:06,900::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914511::DEBUG::2013-11-20 03:33:06,901::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914511::DEBUG::2013-11-20 03:33:06,902::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914511::DEBUG::2013-11-20 03:33:06,903::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914511::DEBUG::2013-11-20 03:33:06,904::task::495::TaskManager.Task::(_debug) Task 76919e73-3eaa-4e27-b2a8-2abd639c0020: ref 0 aborting False >Thread-15914511::INFO::2013-11-20 03:33:06,905::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914498::DEBUG::2013-11-20 03:33:06,960::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000382175 s, 1.3 MB/s\n'; <rc> = 0 >Thread-15914498::DEBUG::2013-11-20 03:33:06,961::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000382175 s, 1.3 MB/s'], size: 512 >Thread-15914498::DEBUG::2013-11-20 03:33:06,962::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=29 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:07,228::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 2.05167 s, 499 kB/s\n'; <rc> = 0 >Thread-15914498::DEBUG::2013-11-20 03:33:07,286::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000135513 s, 3.8 MB/s\n'; <rc> = 0 >Thread-15914498::DEBUG::2013-11-20 03:33:07,287::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000135513 s, 3.8 MB/s'], size: 512 >Thread-15914498::DEBUG::2013-11-20 03:33:07,295::resourceManager::155::ResourceManager.Request::(__init__) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f`ReqID=`85afb3e7-c824-4ede-832b-a43c601a1c1f`::Request was made in '/usr/share/vdsm/storage/volume.py' line '533' at 'prepare' >Thread-15914498::DEBUG::2013-11-20 03:33:07,296::resourceManager::463::ResourceManager::(registerResource) Trying to register resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f' for lock type 'exclusive' >Thread-15914498::DEBUG::2013-11-20 03:33:07,297::resourceManager::505::ResourceManager::(registerResource) Resource '6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f' is free. Now locking as 'exclusive' (1 active user) >Thread-15914498::DEBUG::2013-11-20 03:33:07,298::resourceManager::192::ResourceManager.Request::(grant) ResName=`6b892067-ac7e-4039-a7be-5b7387525b16_lvmActivationNS.8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f`ReqID=`85afb3e7-c824-4ede-832b-a43c601a1c1f`::Granted request >Thread-15914498::DEBUG::2013-11-20 03:33:07,302::blockVolume::587::Storage.Misc.excCmd::(getMetadata) '/bin/dd iflag=direct skip=29 bs=512 if=/dev/6b892067-ac7e-4039-a7be-5b7387525b16/metadata count=1' (cwd None) >Thread-15914498::DEBUG::2013-11-20 03:33:07,594::blockVolume::587::Storage.Misc.excCmd::(getMetadata) SUCCESS: <err> = '1+0 records in\n1+0 records out\n512 bytes (512 B) copied, 0.000207349 s, 2.5 MB/s\n'; <rc> = 0 >Thread-15914498::DEBUG::2013-11-20 03:33:07,595::misc::306::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '512 bytes (512 B) copied, 0.000207349 s, 2.5 MB/s'], size: 512 >Thread-15914498::DEBUG::2013-11-20 03:33:07,596::task::495::TaskManager.Task::(_debug) Task f377e2e2-4ed8-4891-9481-7df918a98f8a: finished: None >Thread-15914498::DEBUG::2013-11-20 03:33:07,596::task::495::TaskManager.Task::(_debug) Task f377e2e2-4ed8-4891-9481-7df918a98f8a: moving from state preparing -> state finished >Thread-15914498::DEBUG::2013-11-20 03:33:07,597::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914498::DEBUG::2013-11-20 03:33:07,597::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914498::DEBUG::2013-11-20 03:33:07,598::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914498::DEBUG::2013-11-20 03:33:07,598::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914498::DEBUG::2013-11-20 03:33:07,598::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914498::DEBUG::2013-11-20 03:33:07,599::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914498::DEBUG::2013-11-20 03:33:07,599::task::495::TaskManager.Task::(_debug) Task f377e2e2-4ed8-4891-9481-7df918a98f8a: ref 0 aborting False >Thread-15914498::INFO::2013-11-20 03:33:07,600::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: prepareVolume, Return response: {'status': {'message': 'OK', 'code': 0}} >Thread-15914498::INFO::2013-11-20 03:33:07,600::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=5af747d8-95b1-4b7e-af53-cb77e4fcc896 volUUID=8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f) >Thread-15914498::DEBUG::2013-11-20 03:33:07,601::task::495::TaskManager.Task::(_debug) Task 9ce081f2-2124-4594-bceb-86efbe558583: moving from state init -> state preparing >Thread-15914498::DEBUG::2013-11-20 03:33:07,602::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`1fd6b1cb-d338-47e5-9d21-39f746de90bc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1666' at 'public_getVolumePath' >Thread-15914498::DEBUG::2013-11-20 03:33:07,602::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914498::DEBUG::2013-11-20 03:33:07,602::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914498::DEBUG::2013-11-20 03:33:07,603::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`1fd6b1cb-d338-47e5-9d21-39f746de90bc`::Granted request >Thread-15914498::DEBUG::2013-11-20 03:33:07,603::task::495::TaskManager.Task::(_debug) Task 9ce081f2-2124-4594-bceb-86efbe558583: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914498::DEBUG::2013-11-20 03:33:07,604::task::495::TaskManager.Task::(_debug) Task 9ce081f2-2124-4594-bceb-86efbe558583: ref 1 aborting False >Thread-15914498::DEBUG::2013-11-20 03:33:07,604::task::495::TaskManager.Task::(_debug) Task 9ce081f2-2124-4594-bceb-86efbe558583: finished: {'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/5af747d8-95b1-4b7e-af53-cb77e4fcc896/8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f'} >Thread-15914498::DEBUG::2013-11-20 03:33:07,605::task::495::TaskManager.Task::(_debug) Task 9ce081f2-2124-4594-bceb-86efbe558583: moving from state preparing -> state finished >Thread-15914498::DEBUG::2013-11-20 03:33:07,605::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914498::DEBUG::2013-11-20 03:33:07,605::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914498::DEBUG::2013-11-20 03:33:07,606::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914498::DEBUG::2013-11-20 03:33:07,606::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914498::DEBUG::2013-11-20 03:33:07,606::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914498::DEBUG::2013-11-20 03:33:07,607::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914498::DEBUG::2013-11-20 03:33:07,607::task::495::TaskManager.Task::(_debug) Task 9ce081f2-2124-4594-bceb-86efbe558583: ref 0 aborting False >Thread-15914498::INFO::2013-11-20 03:33:07,607::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumePath, Return response: {'status': {'message': 'OK', 'code': 0}, 'path': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/5af747d8-95b1-4b7e-af53-cb77e4fcc896/8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f'} >Thread-15914498::INFO::2013-11-20 03:33:07,609::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=5af747d8-95b1-4b7e-af53-cb77e4fcc896 volUUID=8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f) >Thread-15914498::DEBUG::2013-11-20 03:33:07,609::task::495::TaskManager.Task::(_debug) Task 42a2c1f4-3574-427c-a622-6b02d6d8ea08: moving from state init -> state preparing >Thread-15914498::DEBUG::2013-11-20 03:33:07,612::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`98a550c7-77b4-4fd2-9f7f-a0290471b1f8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914498::DEBUG::2013-11-20 03:33:07,612::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914498::DEBUG::2013-11-20 03:33:07,613::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914498::DEBUG::2013-11-20 03:33:07,613::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`98a550c7-77b4-4fd2-9f7f-a0290471b1f8`::Granted request >Thread-15914498::DEBUG::2013-11-20 03:33:07,614::task::495::TaskManager.Task::(_debug) Task 42a2c1f4-3574-427c-a622-6b02d6d8ea08: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914498::DEBUG::2013-11-20 03:33:07,614::task::495::TaskManager.Task::(_debug) Task 42a2c1f4-3574-427c-a622-6b02d6d8ea08: ref 1 aborting False >Thread-15914498::DEBUG::2013-11-20 03:33:07,614::task::495::TaskManager.Task::(_debug) Task 42a2c1f4-3574-427c-a622-6b02d6d8ea08: finished: {'truesize': '86033563648', 'apparentsize': '86033563648'} >Thread-15914498::DEBUG::2013-11-20 03:33:07,615::task::495::TaskManager.Task::(_debug) Task 42a2c1f4-3574-427c-a622-6b02d6d8ea08: moving from state preparing -> state finished >Thread-15914498::DEBUG::2013-11-20 03:33:07,615::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914498::DEBUG::2013-11-20 03:33:07,615::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914498::DEBUG::2013-11-20 03:33:07,618::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914498::DEBUG::2013-11-20 03:33:07,618::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914498::DEBUG::2013-11-20 03:33:07,619::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914498::DEBUG::2013-11-20 03:33:07,619::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914498::DEBUG::2013-11-20 03:33:07,619::task::495::TaskManager.Task::(_debug) Task 42a2c1f4-3574-427c-a622-6b02d6d8ea08: ref 0 aborting False >Thread-15914498::INFO::2013-11-20 03:33:07,620::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '86033563648', 'apparentsize': '86033563648'} >Thread-15914498::INFO::2013-11-20 03:33:07,620::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16) >Thread-15914498::DEBUG::2013-11-20 03:33:07,620::task::495::TaskManager.Task::(_debug) Task 478f9436-6265-4f00-97b7-17e79ec7de22: moving from state init -> state preparing >Thread-15914498::INFO::2013-11-20 03:33:07,621::blockSD::674::Storage.StorageDomain::(validate) sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 >Thread-15914498::DEBUG::2013-11-20 03:33:07,621::lvm::554::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15914498::DEBUG::2013-11-20 03:33:07,622::lvm::556::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15914498::DEBUG::2013-11-20 03:33:07,622::lvm::565::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-15914498::DEBUG::2013-11-20 03:33:07,623::lvm::577::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-15914498::DEBUG::2013-11-20 03:33:07,623::lvm::439::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-15914498::DEBUG::2013-11-20 03:33:07,628::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15914498::DEBUG::2013-11-20 03:33:08,173::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914498::DEBUG::2013-11-20 03:33:08,193::lvm::466::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-15914498::DEBUG::2013-11-20 03:33:08,194::persistentDict::204::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=[u'VERSION=2', u'LOCKPOLICY=', u'DESCRIPTION=OS', u'VGUUID=aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', u'LOGBLKSIZE=512', u'LEASERETRIES=3', u'PV0=pv:3600c0ff000152111c050605001000000,uuid:V0Qu2g-uNB7-OL9h-ogfd-i1xl-RRZk-CpgKaU,pestart:0,pecount:26791,mapoffset:0', u'LOCKRENEWALINTERVALSEC=5', u'PHYBLKSIZE=512', u'SDUUID=6b892067-ac7e-4039-a7be-5b7387525b16', u'CLASS=Data', u'TYPE=FCP', u'LEASETIMESEC=60', u'IOOPTIMEOUTSEC=10', u'POOL_UUID=c4e5889a-348d-11e2-83fe-9f72369eab30', u'POOL_DESCRIPTION=Default', u'ROLE=Master', u'MASTER_VERSION=1', u'POOL_DOMAINS=6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', u'_SHA_CKSUM=716ea7492d9134a4b13de27ac61a336ad474727c', u'POOL_SPM_ID=2', u'POOL_SPM_LVER=287'] >Thread-15914498::DEBUG::2013-11-20 03:33:08,203::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`33928e4c-e86a-418e-a4e6-866caa71f305`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1297' at 'public_getStorageDomainInfo' >Thread-15914498::DEBUG::2013-11-20 03:33:08,204::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914498::DEBUG::2013-11-20 03:33:08,205::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914498::DEBUG::2013-11-20 03:33:08,205::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`33928e4c-e86a-418e-a4e6-866caa71f305`::Granted request >Thread-15914498::DEBUG::2013-11-20 03:33:08,206::task::495::TaskManager.Task::(_debug) Task 478f9436-6265-4f00-97b7-17e79ec7de22: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914498::DEBUG::2013-11-20 03:33:08,207::task::495::TaskManager.Task::(_debug) Task 478f9436-6265-4f00-97b7-17e79ec7de22: ref 1 aborting False >Thread-15914498::WARNING::2013-11-20 03:33:08,209::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914498::DEBUG::2013-11-20 03:33:08,209::task::495::TaskManager.Task::(_debug) Task 478f9436-6265-4f00-97b7-17e79ec7de22: finished: {'info': {'uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'vguuid': 'aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', 'lver': 287, 'state': 'OK', 'version': '2', 'role': 'Master', 'master_ver': 1, 'spm_id': 2, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'OS'}} >Thread-15914498::DEBUG::2013-11-20 03:33:08,210::task::495::TaskManager.Task::(_debug) Task 478f9436-6265-4f00-97b7-17e79ec7de22: moving from state preparing -> state finished >Thread-15914498::DEBUG::2013-11-20 03:33:08,210::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914498::DEBUG::2013-11-20 03:33:08,211::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914498::DEBUG::2013-11-20 03:33:08,211::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914498::DEBUG::2013-11-20 03:33:08,212::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914498::DEBUG::2013-11-20 03:33:08,212::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914498::DEBUG::2013-11-20 03:33:08,213::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914498::DEBUG::2013-11-20 03:33:08,213::task::495::TaskManager.Task::(_debug) Task 478f9436-6265-4f00-97b7-17e79ec7de22: ref 0 aborting False >Thread-15914498::INFO::2013-11-20 03:33:08,213::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStorageDomainInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'vguuid': 'aK7xIW-K1tt-PB18-Inq2-4CkH-gEEx-jxRPYn', 'lver': 287, 'state': 'OK', 'version': '2', 'role': 'Master', 'master_ver': 1, 'spm_id': 2, 'type': 'FCP', 'class': 'Data', 'pool': [u'c4e5889a-348d-11e2-83fe-9f72369eab30'], 'name': 'OS'}} >Thread-15914498::DEBUG::2013-11-20 03:33:08,230::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/libexec/vdsm/hooks/before_vm_start/50_vhostmd' (cwd None) >Thread-15914512::INFO::2013-11-20 03:33:08,949::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914512::DEBUG::2013-11-20 03:33:08,951::task::495::TaskManager.Task::(_debug) Task 446165ea-8faa-4592-974d-e9772d2ee250: moving from state init -> state preparing >Thread-15914512::DEBUG::2013-11-20 03:33:08,952::task::495::TaskManager.Task::(_debug) Task 446165ea-8faa-4592-974d-e9772d2ee250: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00112295150757', 'lastCheck': 1384918386.62466, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0161530971527', 'lastCheck': 1384918382.4393289, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0522398948669', 'lastCheck': 1384918385.782506, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.209988832474', 'lastCheck': 1384918381.320569, 'valid': True, 'code': 0}} >Thread-15914512::DEBUG::2013-11-20 03:33:08,953::task::495::TaskManager.Task::(_debug) Task 446165ea-8faa-4592-974d-e9772d2ee250: moving from state preparing -> state finished >Thread-15914512::DEBUG::2013-11-20 03:33:08,953::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914512::DEBUG::2013-11-20 03:33:08,954::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914512::DEBUG::2013-11-20 03:33:08,954::task::495::TaskManager.Task::(_debug) Task 446165ea-8faa-4592-974d-e9772d2ee250: ref 0 aborting False >Thread-15914512::INFO::2013-11-20 03:33:08,957::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00112295150757', 'lastCheck': 1384918386.62466, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0161530971527', 'lastCheck': 1384918382.4393289, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0522398948669', 'lastCheck': 1384918385.782506, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.209988832474', 'lastCheck': 1384918381.320569, 'valid': True, 'code': 0}} >Thread-15914513::DEBUG::2013-11-20 03:33:09,150::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk stats not available >Thread-15914513::DEBUG::2013-11-20 03:33:09,151::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`163a3426-5ec5-46a5-b8f1-af06fc5d7e7d`::Disk latency not available >Thread-15914513::DEBUG::2013-11-20 03:33:09,156::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk stats not available >Thread-15914513::DEBUG::2013-11-20 03:33:09,156::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::Disk latency not available >Thread-15914498::DEBUG::2013-11-20 03:33:09,180::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914498::INFO::2013-11-20 03:33:09,181::hooks::51::root::(_runHooksDir) >Thread-15914498::DEBUG::2013-11-20 03:33:09,181::libvirtvm::1077::vm.Vm::(_run) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::<?xml version="1.0" encoding="utf-8"?> ><domain type="kvm"> > <name>sukhumi</name> > <uuid>e97d1c52-b688-4192-bdeb-2dfadcc79786</uuid> > <memory>4608000</memory> > <currentMemory>4608000</currentMemory> > <vcpu>4</vcpu> > <devices> > <disk device="disk" type="block"> > <source dev="/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/6b892067-ac7e-4039-a7be-5b7387525b16/images/5af747d8-95b1-4b7e-af53-cb77e4fcc896/8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f"/> > <target bus="ide" dev="hda"/> > <serial>7e-af53-cb77e4fcc896</serial> > <driver cache="none" error_policy="stop" io="native" name="qemu" type="raw"/> > </disk> > <disk device="cdrom" type="file"> > <source file="/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/RHEV-toolsSetup_3.0_39.iso"/> > <target bus="ide" dev="hdc"/> > </disk> > <controller index="0" ports="16" type="virtio-serial"/> > <channel type="unix"> > <target name="com.redhat.rhevm.vdsm" type="virtio"/> > <source mode="bind" path="/var/lib/libvirt/qemu/channels/sukhumi.com.redhat.rhevm.vdsm"/> > </channel> > <memballoon model="none"/> > <interface type="bridge"> > <mac address="00:1a:4a:a8:64:13"/> > <model type="virtio"/> > <source bridge="rhevm"/> > </interface> > <input bus="ps2" type="mouse"/> > <video> > <model heads="1" type="qxl" vram="65536"/> > </video> > <channel type="spicevmc"> > <target name="com.redhat.spice.0" type="virtio"/> > </channel> > <graphics autoport="yes" keymap="en-us" listen="0" passwd="*****" passwdValidTo="1970-01-01T00:00:01" port="-1" tlsPort="-1" type="spice"> > <channel mode="secure" name="main"/> > <channel mode="secure" name="inputs"/> > </graphics> > </devices> > <os> > <type arch="x86_64" machine="rhel6.2.0">hvm</type> > <boot dev="hd"/> > <smbios mode="sysinfo"/> > </os> > <sysinfo type="smbios"> > <system> > <entry name="manufacturer">Red Hat</entry> > <entry name="product">RHEV Hypervisor</entry> > <entry name="version">6.3-20121012.0.el6_3</entry> > <entry name="serial">30343536-3138-5A43-4A32-333630373442_AC:16:2D:76:EB:F0</entry> > <entry name="uuid">e97d1c52-b688-4192-bdeb-2dfadcc79786</entry> > </system> > </sysinfo> > <clock adjustment="0" offset="variable"> > <timer name="rtc" tickpolicy="catchup"/> > </clock> > <features> > <acpi/> > </features> > <cpu match="exact"> > <model>Westmere</model> > <topology cores="1" sockets="4" threads="1"/> > </cpu> ></domain> > >Dummy-15912651::DEBUG::2013-11-20 03:33:09,244::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:09,628::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0479395 s, 21.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:33:11,638::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:12,034::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0521188 s, 19.6 MB/s\n'; <rc> = 0 >Thread-15914435::INFO::2013-11-20 03:33:13,941::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=143cb26d-de08-4d30-89cf-aee10245b66a volUUID=d7028f7c-0551-4bbe-a907-6f804cc9bd54) >Thread-15914435::DEBUG::2013-11-20 03:33:13,942::task::495::TaskManager.Task::(_debug) Task 8acc6d76-103e-4825-8877-5714bdbf16b1: moving from state init -> state preparing >Thread-15914435::DEBUG::2013-11-20 03:33:13,944::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d26f61de-a3e7-4ae9-aeca-9e1bbf947e6e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914435::DEBUG::2013-11-20 03:33:13,944::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914435::DEBUG::2013-11-20 03:33:13,945::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914435::DEBUG::2013-11-20 03:33:13,945::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d26f61de-a3e7-4ae9-aeca-9e1bbf947e6e`::Granted request >Thread-15914435::DEBUG::2013-11-20 03:33:13,946::task::495::TaskManager.Task::(_debug) Task 8acc6d76-103e-4825-8877-5714bdbf16b1: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914435::DEBUG::2013-11-20 03:33:13,946::task::495::TaskManager.Task::(_debug) Task 8acc6d76-103e-4825-8877-5714bdbf16b1: ref 1 aborting False >Thread-15914435::DEBUG::2013-11-20 03:33:13,947::lvm::480::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' got the operation mutex >Thread-15914435::DEBUG::2013-11-20 03:33:13,948::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm lvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,vg_name,attr,size,seg_start_pe,devices,tags 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:14,116::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:14,670::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0473682 s, 21.6 MB/s\n'; <rc> = 0 >Thread-15914435::DEBUG::2013-11-20 03:33:14,817::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914435::DEBUG::2013-11-20 03:33:16,049::lvm::509::OperationMutex::(_reloadlvs) Operation 'lvm reload operation' released the operation mutex >Thread-15914435::DEBUG::2013-11-20 03:33:16,052::task::495::TaskManager.Task::(_debug) Task 8acc6d76-103e-4825-8877-5714bdbf16b1: finished: {'truesize': '56908316672', 'apparentsize': '56908316672'} >Thread-15914435::DEBUG::2013-11-20 03:33:16,059::task::495::TaskManager.Task::(_debug) Task 8acc6d76-103e-4825-8877-5714bdbf16b1: moving from state preparing -> state finished >Thread-15914435::DEBUG::2013-11-20 03:33:16,060::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914435::DEBUG::2013-11-20 03:33:16,060::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914435::DEBUG::2013-11-20 03:33:16,061::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914435::DEBUG::2013-11-20 03:33:16,062::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914435::DEBUG::2013-11-20 03:33:16,063::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914435::DEBUG::2013-11-20 03:33:16,063::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914435::DEBUG::2013-11-20 03:33:16,064::task::495::TaskManager.Task::(_debug) Task 8acc6d76-103e-4825-8877-5714bdbf16b1: ref 0 aborting False >Thread-15914435::INFO::2013-11-20 03:33:16,065::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '56908316672', 'apparentsize': '56908316672'} >Dummy-15912651::DEBUG::2013-11-20 03:33:16,679::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:17,071::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0511597 s, 20.0 MB/s\n'; <rc> = 0 >Thread-15914516::DEBUG::2013-11-20 03:33:17,357::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914516::INFO::2013-11-20 03:33:17,362::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914516::DEBUG::2013-11-20 03:33:17,363::task::495::TaskManager.Task::(_debug) Task bf0cb843-faa2-4c8b-af48-30dff3352034: moving from state init -> state preparing >Thread-15914516::DEBUG::2013-11-20 03:33:17,364::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914516::DEBUG::2013-11-20 03:33:17,364::task::495::TaskManager.Task::(_debug) Task bf0cb843-faa2-4c8b-af48-30dff3352034: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914516::DEBUG::2013-11-20 03:33:17,367::task::495::TaskManager.Task::(_debug) Task bf0cb843-faa2-4c8b-af48-30dff3352034: moving from state preparing -> state finished >Thread-15914516::DEBUG::2013-11-20 03:33:17,367::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914516::DEBUG::2013-11-20 03:33:17,375::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914516::DEBUG::2013-11-20 03:33:17,375::task::495::TaskManager.Task::(_debug) Task bf0cb843-faa2-4c8b-af48-30dff3352034: ref 0 aborting False >Thread-15914516::INFO::2013-11-20 03:33:17,376::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914517::DEBUG::2013-11-20 03:33:17,384::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914517::INFO::2013-11-20 03:33:17,385::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914517::DEBUG::2013-11-20 03:33:17,385::task::495::TaskManager.Task::(_debug) Task c0549829-95f5-4965-8127-ac7b714316dd: moving from state init -> state preparing >Thread-15914517::DEBUG::2013-11-20 03:33:17,389::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6f038319-6244-42d0-83d4-6f26ef110261`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914517::DEBUG::2013-11-20 03:33:17,389::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914517::DEBUG::2013-11-20 03:33:17,395::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914517::DEBUG::2013-11-20 03:33:17,397::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6f038319-6244-42d0-83d4-6f26ef110261`::Granted request >Thread-15914517::DEBUG::2013-11-20 03:33:17,401::task::495::TaskManager.Task::(_debug) Task c0549829-95f5-4965-8127-ac7b714316dd: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914517::DEBUG::2013-11-20 03:33:17,404::task::495::TaskManager.Task::(_debug) Task c0549829-95f5-4965-8127-ac7b714316dd: ref 1 aborting False >Thread-15914517::WARNING::2013-11-20 03:33:17,410::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914517::DEBUG::2013-11-20 03:33:17,421::task::495::TaskManager.Task::(_debug) Task c0549829-95f5-4965-8127-ac7b714316dd: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914517::DEBUG::2013-11-20 03:33:17,440::task::495::TaskManager.Task::(_debug) Task c0549829-95f5-4965-8127-ac7b714316dd: moving from state preparing -> state finished >Thread-15914517::DEBUG::2013-11-20 03:33:17,440::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914517::DEBUG::2013-11-20 03:33:17,446::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914517::DEBUG::2013-11-20 03:33:17,448::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914517::DEBUG::2013-11-20 03:33:17,448::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914517::DEBUG::2013-11-20 03:33:17,449::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914517::DEBUG::2013-11-20 03:33:17,449::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914517::DEBUG::2013-11-20 03:33:17,451::task::495::TaskManager.Task::(_debug) Task c0549829-95f5-4965-8127-ac7b714316dd: ref 0 aborting False >Thread-15914517::INFO::2013-11-20 03:33:17,453::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914453::INFO::2013-11-20 03:33:19,066::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=5f930ef3-763f-4294-a62a-6d52df220f8c volUUID=b9cffdbe-152e-41ce-ae5d-08db7c08454e) >Thread-15914453::DEBUG::2013-11-20 03:33:19,066::task::495::TaskManager.Task::(_debug) Task 29d4952f-c219-4890-8400-58e2ba3344f2: moving from state init -> state preparing >Thread-15914453::DEBUG::2013-11-20 03:33:19,067::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`23290589-5826-49b4-8095-82a7aaeb6315`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914453::DEBUG::2013-11-20 03:33:19,068::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914453::DEBUG::2013-11-20 03:33:19,068::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914453::DEBUG::2013-11-20 03:33:19,069::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`23290589-5826-49b4-8095-82a7aaeb6315`::Granted request >Thread-15914453::DEBUG::2013-11-20 03:33:19,069::task::495::TaskManager.Task::(_debug) Task 29d4952f-c219-4890-8400-58e2ba3344f2: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914453::DEBUG::2013-11-20 03:33:19,070::task::495::TaskManager.Task::(_debug) Task 29d4952f-c219-4890-8400-58e2ba3344f2: ref 1 aborting False >Thread-15914453::DEBUG::2013-11-20 03:33:19,070::task::495::TaskManager.Task::(_debug) Task 29d4952f-c219-4890-8400-58e2ba3344f2: finished: {'truesize': '53821308928', 'apparentsize': '53821308928'} >Thread-15914453::DEBUG::2013-11-20 03:33:19,070::task::495::TaskManager.Task::(_debug) Task 29d4952f-c219-4890-8400-58e2ba3344f2: moving from state preparing -> state finished >Thread-15914453::DEBUG::2013-11-20 03:33:19,071::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914453::DEBUG::2013-11-20 03:33:19,071::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914453::DEBUG::2013-11-20 03:33:19,072::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914453::DEBUG::2013-11-20 03:33:19,072::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914453::DEBUG::2013-11-20 03:33:19,072::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914453::DEBUG::2013-11-20 03:33:19,073::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914453::DEBUG::2013-11-20 03:33:19,073::task::495::TaskManager.Task::(_debug) Task 29d4952f-c219-4890-8400-58e2ba3344f2: ref 0 aborting False >Thread-15914453::INFO::2013-11-20 03:33:19,073::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '53821308928', 'apparentsize': '53821308928'} >Dummy-15912651::DEBUG::2013-11-20 03:33:19,080::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >libvirtEventLoop::DEBUG::2013-11-20 03:33:19,489::libvirtvm::1401::vm.Vm::(_onLibvirtLifecycleEvent) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::event Started detail 0 opaque None >Thread-15914498::DEBUG::2013-11-20 03:33:19,573::utils::307::vm.Vm::(start) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Start statistics collection >Thread-15914519::DEBUG::2013-11-20 03:33:19,575::utils::336::vm.Vm::(run) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Stats thread started >Thread-15914520::DEBUG::2013-11-20 03:33:19,586::guestIF::47::root::(_connect) Attempting connection to /var/lib/libvirt/qemu/channels/sukhumi.com.redhat.rhevm.vdsm >Thread-15914519::INFO::2013-11-20 03:33:19,596::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=5af747d8-95b1-4b7e-af53-cb77e4fcc896 volUUID=8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f) >Thread-15914520::DEBUG::2013-11-20 03:33:19,598::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /usr/share/vdsm/prepare-vmchannel /var/lib/libvirt/qemu/channels/sukhumi.com.redhat.rhevm.vdsm' (cwd None) >Thread-15914498::WARNING::2013-11-20 03:33:19,599::libvirtvm::1124::vm.Vm::(_readPauseCode) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::_readPauseCode unsupported by libvirt vm >Thread-15914519::DEBUG::2013-11-20 03:33:19,600::task::495::TaskManager.Task::(_debug) Task a6782986-0ea5-4d5a-ba00-eaa306248363: moving from state init -> state preparing >Thread-15914498::DEBUG::2013-11-20 03:33:19,755::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /usr/share/vdsm/get-vm-pid sukhumi' (cwd None) >Thread-15914519::DEBUG::2013-11-20 03:33:19,770::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`13bcf4d6-b98d-4f35-a044-b6ccbbfa6537`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914519::DEBUG::2013-11-20 03:33:19,772::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914520::DEBUG::2013-11-20 03:33:20,081::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914519::DEBUG::2013-11-20 03:33:20,082::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914519::DEBUG::2013-11-20 03:33:20,088::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`13bcf4d6-b98d-4f35-a044-b6ccbbfa6537`::Granted request >Thread-15914520::DEBUG::2013-11-20 03:33:20,090::guestIF::53::root::(_connect) connected to /var/lib/libvirt/qemu/channels/sukhumi.com.redhat.rhevm.vdsm >Thread-15914519::DEBUG::2013-11-20 03:33:20,101::task::495::TaskManager.Task::(_debug) Task a6782986-0ea5-4d5a-ba00-eaa306248363: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914519::DEBUG::2013-11-20 03:33:20,104::task::495::TaskManager.Task::(_debug) Task a6782986-0ea5-4d5a-ba00-eaa306248363: ref 1 aborting False >Thread-15914519::DEBUG::2013-11-20 03:33:20,108::task::495::TaskManager.Task::(_debug) Task a6782986-0ea5-4d5a-ba00-eaa306248363: finished: {'truesize': '86033563648', 'apparentsize': '86033563648'} >Thread-15914519::DEBUG::2013-11-20 03:33:20,114::task::495::TaskManager.Task::(_debug) Task a6782986-0ea5-4d5a-ba00-eaa306248363: moving from state preparing -> state finished >Thread-15914519::DEBUG::2013-11-20 03:33:20,116::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914519::DEBUG::2013-11-20 03:33:20,129::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914519::DEBUG::2013-11-20 03:33:20,137::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914519::DEBUG::2013-11-20 03:33:20,141::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914519::DEBUG::2013-11-20 03:33:20,142::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914519::DEBUG::2013-11-20 03:33:20,148::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914519::DEBUG::2013-11-20 03:33:20,151::task::495::TaskManager.Task::(_debug) Task a6782986-0ea5-4d5a-ba00-eaa306248363: ref 0 aborting False >Thread-15914519::INFO::2013-11-20 03:33:20,152::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '86033563648', 'apparentsize': '86033563648'} >Dummy-15912651::DEBUG::2013-11-20 03:33:20,168::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0991058 s, 10.3 MB/s\n'; <rc> = 0 >Thread-15914498::DEBUG::2013-11-20 03:33:21,506::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914498::DEBUG::2013-11-20 03:33:21,558::utils::579::Storage.Misc.excCmd::(execCmd) '/usr/bin/sudo -n /sbin/service ksmtuned retune' (cwd None) >Thread-15914498::DEBUG::2013-11-20 03:33:21,976::utils::579::Storage.Misc.excCmd::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-15914498::DEBUG::2013-11-20 03:33:21,980::vm::358::vm.Vm::(_startUnderlyingVm) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::_ongoingCreations released >Thread-15914522::DEBUG::2013-11-20 03:33:22,124::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('e97d1c52-b688-4192-bdeb-2dfadcc79786',) {} >Thread-15914522::DEBUG::2013-11-20 03:33:22,125::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::CPU stats not available >Thread-15914522::DEBUG::2013-11-20 03:33:22,125::libvirtvm::202::vm.Vm::(_getNetworkStats) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Network stats not available >Thread-15914522::DEBUG::2013-11-20 03:33:22,126::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Disk stats not available >Thread-15914522::DEBUG::2013-11-20 03:33:22,127::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Disk latency not available >Thread-15914522::DEBUG::2013-11-20 03:33:22,127::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '7039', 'displayIp': '0', 'displayPort': u'5916', 'session': 'Unknown', 'displaySecurePort': u'5917', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/RHEV-toolsSetup_3.0_39.iso', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet9': {'state': 'unknown', 'macAddr': u'00:1a:4a:a8:64:13', 'speed': '1000', 'name': u'vnet9'}}, 'vmId': 'e97d1c52-b688-4192-bdeb-2dfadcc79786', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'hda': {'readLatency': '0', 'apparentsize': '86033563648', 'writeLatency': '0', 'imageID': '5af747d8-95b1-4b7e-af53-cb77e4fcc896', 'flushLatency': '0', 'truesize': '86033563648'}}, 'boot': 'c', 'statsAge': '2.54', 'cpuIdle': '100.00', 'elapsedTime': '39', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '0', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:33:22,179::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:22,818::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.083188 s, 12.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:33:24,834::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:25,207::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0468094 s, 21.9 MB/s\n'; <rc> = 0 >Thread-15914523::INFO::2013-11-20 03:33:25,472::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914523::DEBUG::2013-11-20 03:33:25,474::task::495::TaskManager.Task::(_debug) Task 8d6f5d3b-76fa-4ca0-a7e4-3cc7f9842ef3: moving from state init -> state preparing >Thread-15914523::DEBUG::2013-11-20 03:33:25,477::task::495::TaskManager.Task::(_debug) Task 8d6f5d3b-76fa-4ca0-a7e4-3cc7f9842ef3: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000340938568115', 'lastCheck': 1384918396.843962, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0111448764801', 'lastCheck': 1384918403.0861681, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.013267993927', 'lastCheck': 1384918396.3462951, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00055193901062', 'lastCheck': 1384918401.493926, 'valid': True, 'code': 0}} >Thread-15914523::DEBUG::2013-11-20 03:33:25,478::task::495::TaskManager.Task::(_debug) Task 8d6f5d3b-76fa-4ca0-a7e4-3cc7f9842ef3: moving from state preparing -> state finished >Thread-15914523::DEBUG::2013-11-20 03:33:25,478::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914523::DEBUG::2013-11-20 03:33:25,479::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914523::DEBUG::2013-11-20 03:33:25,480::task::495::TaskManager.Task::(_debug) Task 8d6f5d3b-76fa-4ca0-a7e4-3cc7f9842ef3: ref 0 aborting False >Thread-15914523::INFO::2013-11-20 03:33:25,481::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000340938568115', 'lastCheck': 1384918396.843962, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0111448764801', 'lastCheck': 1384918403.0861681, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.013267993927', 'lastCheck': 1384918396.3462951, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00055193901062', 'lastCheck': 1384918401.493926, 'valid': True, 'code': 0}} >Thread-15914524::DEBUG::2013-11-20 03:33:25,782::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::CPU stats not available >Thread-15914524::DEBUG::2013-11-20 03:33:25,783::libvirtvm::202::vm.Vm::(_getNetworkStats) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Network stats not available >Thread-15914524::DEBUG::2013-11-20 03:33:25,783::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Disk stats not available >Thread-15914524::DEBUG::2013-11-20 03:33:25,784::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Disk latency not available >Thread-15913518::INFO::2013-11-20 03:33:26,158::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:33:26,159::task::495::TaskManager.Task::(_debug) Task fb6082c5-f2b2-4629-9f02-6fa256ff4c73: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:33:26,160::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e94b13be-956a-4fbb-b8da-4385608a2d56`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:33:26,160::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:33:26,161::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:33:26,161::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e94b13be-956a-4fbb-b8da-4385608a2d56`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:33:26,162::task::495::TaskManager.Task::(_debug) Task fb6082c5-f2b2-4629-9f02-6fa256ff4c73: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:33:26,163::task::495::TaskManager.Task::(_debug) Task fb6082c5-f2b2-4629-9f02-6fa256ff4c73: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:33:26,163::task::495::TaskManager.Task::(_debug) Task fb6082c5-f2b2-4629-9f02-6fa256ff4c73: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:33:26,164::task::495::TaskManager.Task::(_debug) Task fb6082c5-f2b2-4629-9f02-6fa256ff4c73: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:33:26,164::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:33:26,164::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:33:26,165::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:33:26,166::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:33:26,166::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:33:26,167::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:33:26,167::task::495::TaskManager.Task::(_debug) Task fb6082c5-f2b2-4629-9f02-6fa256ff4c73: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:33:26,168::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:33:26,168::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:33:26,168::task::495::TaskManager.Task::(_debug) Task 1a41cbfd-56ed-4e82-adfe-39796d768e24: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:33:26,169::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`4fae2134-1311-4c5c-9aaf-994a9c8c1055`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:33:26,170::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:33:26,170::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:33:26,170::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`4fae2134-1311-4c5c-9aaf-994a9c8c1055`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:33:26,171::task::495::TaskManager.Task::(_debug) Task 1a41cbfd-56ed-4e82-adfe-39796d768e24: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:33:26,171::task::495::TaskManager.Task::(_debug) Task 1a41cbfd-56ed-4e82-adfe-39796d768e24: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:33:26,172::task::495::TaskManager.Task::(_debug) Task 1a41cbfd-56ed-4e82-adfe-39796d768e24: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:33:26,172::task::495::TaskManager.Task::(_debug) Task 1a41cbfd-56ed-4e82-adfe-39796d768e24: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:33:26,172::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:33:26,173::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:33:26,173::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:33:26,174::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:33:26,174::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:33:26,174::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:33:26,175::task::495::TaskManager.Task::(_debug) Task 1a41cbfd-56ed-4e82-adfe-39796d768e24: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:33:26,175::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:33:27,218::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:27,717::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0486837 s, 21.0 MB/s\n'; <rc> = 0 >Thread-15914525::DEBUG::2013-11-20 03:33:27,982::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914525::INFO::2013-11-20 03:33:27,983::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914525::DEBUG::2013-11-20 03:33:27,984::task::495::TaskManager.Task::(_debug) Task 1953f0e4-f417-45e8-8291-8280f8bdb6c3: moving from state init -> state preparing >Thread-15914525::DEBUG::2013-11-20 03:33:27,985::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914525::DEBUG::2013-11-20 03:33:27,985::task::495::TaskManager.Task::(_debug) Task 1953f0e4-f417-45e8-8291-8280f8bdb6c3: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914525::DEBUG::2013-11-20 03:33:27,985::task::495::TaskManager.Task::(_debug) Task 1953f0e4-f417-45e8-8291-8280f8bdb6c3: moving from state preparing -> state finished >Thread-15914525::DEBUG::2013-11-20 03:33:27,986::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914525::DEBUG::2013-11-20 03:33:27,986::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914525::DEBUG::2013-11-20 03:33:27,986::task::495::TaskManager.Task::(_debug) Task 1953f0e4-f417-45e8-8291-8280f8bdb6c3: ref 0 aborting False >Thread-15914525::INFO::2013-11-20 03:33:27,987::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914526::DEBUG::2013-11-20 03:33:28,003::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914526::INFO::2013-11-20 03:33:28,003::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914526::DEBUG::2013-11-20 03:33:28,004::task::495::TaskManager.Task::(_debug) Task 6b20f676-b817-4f4d-8b08-37ce164fb093: moving from state init -> state preparing >Thread-15914526::DEBUG::2013-11-20 03:33:28,006::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`09b35266-5eeb-4ac8-b012-6811b8a3b13a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914526::DEBUG::2013-11-20 03:33:28,008::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914526::DEBUG::2013-11-20 03:33:28,009::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914526::DEBUG::2013-11-20 03:33:28,009::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`09b35266-5eeb-4ac8-b012-6811b8a3b13a`::Granted request >Thread-15914526::DEBUG::2013-11-20 03:33:28,130::task::495::TaskManager.Task::(_debug) Task 6b20f676-b817-4f4d-8b08-37ce164fb093: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914526::DEBUG::2013-11-20 03:33:28,130::task::495::TaskManager.Task::(_debug) Task 6b20f676-b817-4f4d-8b08-37ce164fb093: ref 1 aborting False >Thread-15914526::WARNING::2013-11-20 03:33:28,132::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914526::DEBUG::2013-11-20 03:33:28,132::task::495::TaskManager.Task::(_debug) Task 6b20f676-b817-4f4d-8b08-37ce164fb093: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914526::DEBUG::2013-11-20 03:33:28,133::task::495::TaskManager.Task::(_debug) Task 6b20f676-b817-4f4d-8b08-37ce164fb093: moving from state preparing -> state finished >Thread-15914526::DEBUG::2013-11-20 03:33:28,133::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914526::DEBUG::2013-11-20 03:33:28,133::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914526::DEBUG::2013-11-20 03:33:28,134::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914526::DEBUG::2013-11-20 03:33:28,134::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914526::DEBUG::2013-11-20 03:33:28,135::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914526::DEBUG::2013-11-20 03:33:28,135::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914526::DEBUG::2013-11-20 03:33:28,136::task::495::TaskManager.Task::(_debug) Task 6b20f676-b817-4f4d-8b08-37ce164fb093: ref 0 aborting False >Thread-15914526::INFO::2013-11-20 03:33:28,136::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914528::DEBUG::2013-11-20 03:33:29,256::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('e97d1c52-b688-4192-bdeb-2dfadcc79786',) {} >Thread-15914528::DEBUG::2013-11-20 03:33:29,257::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::CPU stats not available >Thread-15914528::DEBUG::2013-11-20 03:33:29,257::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Disk stats not available >Thread-15914528::DEBUG::2013-11-20 03:33:29,257::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Disk latency not available >Thread-15914528::DEBUG::2013-11-20 03:33:29,258::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '7039', 'displayIp': '0', 'displayPort': u'5916', 'session': 'Unknown', 'displaySecurePort': u'5917', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/RHEV-toolsSetup_3.0_39.iso', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet9': {'macAddr': u'00:1a:4a:a8:64:13', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet9'}}, 'vmId': 'e97d1c52-b688-4192-bdeb-2dfadcc79786', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'hda': {'readLatency': '0', 'apparentsize': '86033563648', 'writeLatency': '0', 'imageID': '5af747d8-95b1-4b7e-af53-cb77e4fcc896', 'flushLatency': '0', 'truesize': '86033563648'}}, 'boot': 'c', 'statsAge': '0.84', 'cpuIdle': '100.00', 'elapsedTime': '46', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '0', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15913546::INFO::2013-11-20 03:33:29,485::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:33:29,486::task::495::TaskManager.Task::(_debug) Task 7cc01f7e-3d82-4af4-b1ea-0c0b6f70062b: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:33:29,487::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`cac5c5ab-71c0-485c-81d5-13f1920dc3f3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:33:29,488::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:33:29,488::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:33:29,489::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`cac5c5ab-71c0-485c-81d5-13f1920dc3f3`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:33:29,489::task::495::TaskManager.Task::(_debug) Task 7cc01f7e-3d82-4af4-b1ea-0c0b6f70062b: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:33:29,490::task::495::TaskManager.Task::(_debug) Task 7cc01f7e-3d82-4af4-b1ea-0c0b6f70062b: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:33:29,491::task::495::TaskManager.Task::(_debug) Task 7cc01f7e-3d82-4af4-b1ea-0c0b6f70062b: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:33:29,491::task::495::TaskManager.Task::(_debug) Task 7cc01f7e-3d82-4af4-b1ea-0c0b6f70062b: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:33:29,492::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:33:29,492::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:33:29,493::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:33:29,493::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:33:29,493::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:33:29,494::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:33:29,494::task::495::TaskManager.Task::(_debug) Task 7cc01f7e-3d82-4af4-b1ea-0c0b6f70062b: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:33:29,495::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:33:29,495::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:33:29,496::task::495::TaskManager.Task::(_debug) Task 7716432c-19ca-4116-990d-1c8bc53a3f06: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:33:29,496::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`f8d3505d-a3dd-4a27-9a9f-6bc342537783`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:33:29,497::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:33:29,497::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:33:29,498::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`f8d3505d-a3dd-4a27-9a9f-6bc342537783`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:33:29,498::task::495::TaskManager.Task::(_debug) Task 7716432c-19ca-4116-990d-1c8bc53a3f06: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:33:29,499::task::495::TaskManager.Task::(_debug) Task 7716432c-19ca-4116-990d-1c8bc53a3f06: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:33:29,500::task::495::TaskManager.Task::(_debug) Task 7716432c-19ca-4116-990d-1c8bc53a3f06: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:33:29,500::task::495::TaskManager.Task::(_debug) Task 7716432c-19ca-4116-990d-1c8bc53a3f06: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:33:29,501::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:33:29,501::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:33:29,502::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:33:29,502::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:33:29,502::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:33:29,503::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:33:29,503::task::495::TaskManager.Task::(_debug) Task 7716432c-19ca-4116-990d-1c8bc53a3f06: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:33:29,504::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:33:29,504::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:33:29,505::task::495::TaskManager.Task::(_debug) Task 888526e3-ba4e-4b60-bfe2-0f4c8f398d14: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:33:29,506::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`2d01c298-ea4b-42c9-bee1-45b4907c3b35`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:33:29,506::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:33:29,507::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:33:29,507::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`2d01c298-ea4b-42c9-bee1-45b4907c3b35`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:33:29,508::task::495::TaskManager.Task::(_debug) Task 888526e3-ba4e-4b60-bfe2-0f4c8f398d14: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:33:29,508::task::495::TaskManager.Task::(_debug) Task 888526e3-ba4e-4b60-bfe2-0f4c8f398d14: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:33:29,509::task::495::TaskManager.Task::(_debug) Task 888526e3-ba4e-4b60-bfe2-0f4c8f398d14: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:33:29,509::task::495::TaskManager.Task::(_debug) Task 888526e3-ba4e-4b60-bfe2-0f4c8f398d14: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:33:29,509::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:33:29,510::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:33:29,510::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:33:29,511::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:33:29,511::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:33:29,511::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:33:29,512::task::495::TaskManager.Task::(_debug) Task 888526e3-ba4e-4b60-bfe2-0f4c8f398d14: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:33:29,512::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:33:29,513::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:33:29,513::task::495::TaskManager.Task::(_debug) Task 7b99fbbd-c089-4f07-a008-7b110db04b51: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:33:29,514::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`f3bd4daa-de51-42fe-9f30-c4ad1ff62c8b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:33:29,514::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:33:29,515::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:33:29,515::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`f3bd4daa-de51-42fe-9f30-c4ad1ff62c8b`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:33:29,516::task::495::TaskManager.Task::(_debug) Task 7b99fbbd-c089-4f07-a008-7b110db04b51: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:33:29,516::task::495::TaskManager.Task::(_debug) Task 7b99fbbd-c089-4f07-a008-7b110db04b51: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:33:29,517::task::495::TaskManager.Task::(_debug) Task 7b99fbbd-c089-4f07-a008-7b110db04b51: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:33:29,517::task::495::TaskManager.Task::(_debug) Task 7b99fbbd-c089-4f07-a008-7b110db04b51: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:33:29,518::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:33:29,518::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:33:29,518::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:33:29,519::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:33:29,519::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:33:29,520::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:33:29,520::task::495::TaskManager.Task::(_debug) Task 7b99fbbd-c089-4f07-a008-7b110db04b51: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:33:29,521::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:33:29,727::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:30,388::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0490419 s, 20.9 MB/s\n'; <rc> = 0 >Thread-15914530::DEBUG::2013-11-20 03:33:32,330::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('e97d1c52-b688-4192-bdeb-2dfadcc79786',) {} >Thread-15914530::DEBUG::2013-11-20 03:33:32,331::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::CPU stats not available >Thread-15914530::DEBUG::2013-11-20 03:33:32,331::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Disk stats not available >Thread-15914530::DEBUG::2013-11-20 03:33:32,332::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Disk latency not available >Thread-15914530::DEBUG::2013-11-20 03:33:32,333::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '7039', 'displayIp': '0', 'displayPort': u'5916', 'session': 'Unknown', 'displaySecurePort': u'5917', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/RHEV-toolsSetup_3.0_39.iso', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet9': {'macAddr': u'00:1a:4a:a8:64:13', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet9'}}, 'vmId': 'e97d1c52-b688-4192-bdeb-2dfadcc79786', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'hda': {'readLatency': '0', 'apparentsize': '86033563648', 'writeLatency': '0', 'imageID': '5af747d8-95b1-4b7e-af53-cb77e4fcc896', 'flushLatency': '0', 'truesize': '86033563648'}}, 'boot': 'c', 'statsAge': '1.92', 'cpuIdle': '100.00', 'elapsedTime': '49', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '0', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:33:32,399::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:32,819::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0466718 s, 21.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:33:34,830::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:35,321::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0530039 s, 19.3 MB/s\n'; <rc> = 0 >Thread-15914532::DEBUG::2013-11-20 03:33:35,369::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('e97d1c52-b688-4192-bdeb-2dfadcc79786',) {} >Thread-15914532::DEBUG::2013-11-20 03:33:35,370::libvirtvm::171::vm.Vm::(_getCpuStats) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::CPU stats not available >Thread-15914532::DEBUG::2013-11-20 03:33:35,371::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Disk stats not available >Thread-15914532::DEBUG::2013-11-20 03:33:35,371::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Disk latency not available >Thread-15914532::DEBUG::2013-11-20 03:33:35,372::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '7039', 'displayIp': '0', 'displayPort': u'5916', 'session': 'Unknown', 'displaySecurePort': u'5917', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/RHEV-toolsSetup_3.0_39.iso', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet9': {'macAddr': u'00:1a:4a:a8:64:13', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet9'}}, 'vmId': 'e97d1c52-b688-4192-bdeb-2dfadcc79786', 'monitorResponse': '0', 'cpuUser': '0.00', 'disks': {u'hda': {'readLatency': '0', 'apparentsize': '86033563648', 'writeLatency': '0', 'imageID': '5af747d8-95b1-4b7e-af53-cb77e4fcc896', 'flushLatency': '0', 'truesize': '86033563648'}}, 'boot': 'c', 'statsAge': '0.81', 'cpuIdle': '100.00', 'elapsedTime': '52', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '0', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:33:37,331::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:37,767::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.047309 s, 21.6 MB/s\n'; <rc> = 0 >Thread-15914534::DEBUG::2013-11-20 03:33:38,396::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('e97d1c52-b688-4192-bdeb-2dfadcc79786',) {} >Thread-15914534::DEBUG::2013-11-20 03:33:38,398::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Disk stats not available >Thread-15914534::DEBUG::2013-11-20 03:33:38,398::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Disk latency not available >Thread-15914534::DEBUG::2013-11-20 03:33:38,399::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Powering up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '7039', 'displayIp': '0', 'displayPort': u'5916', 'session': 'Unknown', 'displaySecurePort': u'5917', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/RHEV-toolsSetup_3.0_39.iso', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet9': {'macAddr': u'00:1a:4a:a8:64:13', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet9'}}, 'vmId': 'e97d1c52-b688-4192-bdeb-2dfadcc79786', 'monitorResponse': '0', 'cpuUser': '51.69', 'disks': {u'hda': {'readLatency': '0', 'apparentsize': '86033563648', 'writeLatency': '0', 'imageID': '5af747d8-95b1-4b7e-af53-cb77e4fcc896', 'flushLatency': '0', 'truesize': '86033563648'}}, 'boot': 'c', 'statsAge': '0.84', 'cpuIdle': '48.31', 'elapsedTime': '55', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '0', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Thread-15914535::DEBUG::2013-11-20 03:33:38,407::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914535::INFO::2013-11-20 03:33:38,408::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914535::DEBUG::2013-11-20 03:33:38,409::task::495::TaskManager.Task::(_debug) Task 72db870f-1fd8-4993-8839-390dfd396dc6: moving from state init -> state preparing >Thread-15914535::DEBUG::2013-11-20 03:33:38,410::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914535::DEBUG::2013-11-20 03:33:38,410::task::495::TaskManager.Task::(_debug) Task 72db870f-1fd8-4993-8839-390dfd396dc6: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914535::DEBUG::2013-11-20 03:33:38,411::task::495::TaskManager.Task::(_debug) Task 72db870f-1fd8-4993-8839-390dfd396dc6: moving from state preparing -> state finished >Thread-15914535::DEBUG::2013-11-20 03:33:38,412::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914535::DEBUG::2013-11-20 03:33:38,412::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914535::DEBUG::2013-11-20 03:33:38,413::task::495::TaskManager.Task::(_debug) Task 72db870f-1fd8-4993-8839-390dfd396dc6: ref 0 aborting False >Thread-15914535::INFO::2013-11-20 03:33:38,413::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914536::DEBUG::2013-11-20 03:33:38,427::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914536::INFO::2013-11-20 03:33:38,428::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914536::DEBUG::2013-11-20 03:33:38,429::task::495::TaskManager.Task::(_debug) Task a78d133a-0846-43db-91e1-bfac8cc33032: moving from state init -> state preparing >Thread-15914536::DEBUG::2013-11-20 03:33:38,436::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ee9d4292-16ae-48fc-a3d0-c1a17f61bd1e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914536::DEBUG::2013-11-20 03:33:38,437::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914536::DEBUG::2013-11-20 03:33:38,438::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914536::DEBUG::2013-11-20 03:33:38,438::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`ee9d4292-16ae-48fc-a3d0-c1a17f61bd1e`::Granted request >Thread-15914536::DEBUG::2013-11-20 03:33:38,440::task::495::TaskManager.Task::(_debug) Task a78d133a-0846-43db-91e1-bfac8cc33032: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914536::DEBUG::2013-11-20 03:33:38,440::task::495::TaskManager.Task::(_debug) Task a78d133a-0846-43db-91e1-bfac8cc33032: ref 1 aborting False >Thread-15914536::WARNING::2013-11-20 03:33:38,443::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914536::DEBUG::2013-11-20 03:33:38,444::task::495::TaskManager.Task::(_debug) Task a78d133a-0846-43db-91e1-bfac8cc33032: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914536::DEBUG::2013-11-20 03:33:38,445::task::495::TaskManager.Task::(_debug) Task a78d133a-0846-43db-91e1-bfac8cc33032: moving from state preparing -> state finished >Thread-15914536::DEBUG::2013-11-20 03:33:38,445::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914536::DEBUG::2013-11-20 03:33:38,446::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914536::DEBUG::2013-11-20 03:33:38,447::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914536::DEBUG::2013-11-20 03:33:38,447::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914536::DEBUG::2013-11-20 03:33:38,448::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914536::DEBUG::2013-11-20 03:33:38,449::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914536::DEBUG::2013-11-20 03:33:38,449::task::495::TaskManager.Task::(_debug) Task a78d133a-0846-43db-91e1-bfac8cc33032: ref 0 aborting False >Thread-15914536::INFO::2013-11-20 03:33:38,450::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:33:39,777::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:40,664::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0908262 s, 11.3 MB/s\n'; <rc> = 0 >Thread-15914537::INFO::2013-11-20 03:33:41,441::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914537::DEBUG::2013-11-20 03:33:41,442::task::495::TaskManager.Task::(_debug) Task ad593370-5876-4500-9e4c-2b7010cf1b90: moving from state init -> state preparing >Thread-15914537::DEBUG::2013-11-20 03:33:41,443::task::495::TaskManager.Task::(_debug) Task ad593370-5876-4500-9e4c-2b7010cf1b90: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000235795974731', 'lastCheck': 1384918417.1302299, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000959873199463', 'lastCheck': 1384918413.1315451, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000960111618042', 'lastCheck': 1384918416.5685389, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000353097915649', 'lastCheck': 1384918411.5051551, 'valid': True, 'code': 0}} >Thread-15914537::DEBUG::2013-11-20 03:33:41,447::task::495::TaskManager.Task::(_debug) Task ad593370-5876-4500-9e4c-2b7010cf1b90: moving from state preparing -> state finished >Thread-15914537::DEBUG::2013-11-20 03:33:41,448::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914537::DEBUG::2013-11-20 03:33:41,448::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914537::DEBUG::2013-11-20 03:33:41,449::task::495::TaskManager.Task::(_debug) Task ad593370-5876-4500-9e4c-2b7010cf1b90: ref 0 aborting False >Thread-15914537::INFO::2013-11-20 03:33:41,449::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000235795974731', 'lastCheck': 1384918417.1302299, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000959873199463', 'lastCheck': 1384918413.1315451, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000960111618042', 'lastCheck': 1384918416.5685389, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000353097915649', 'lastCheck': 1384918411.5051551, 'valid': True, 'code': 0}} >Thread-15914538::DEBUG::2013-11-20 03:33:41,557::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Disk stats not available >Thread-15914538::DEBUG::2013-11-20 03:33:41,559::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Disk latency not available >Dummy-15912651::DEBUG::2013-11-20 03:33:42,677::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:43,713::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.117225 s, 8.7 MB/s\n'; <rc> = 0 >Thread-15913639::INFO::2013-11-20 03:33:44,569::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:33:44,588::task::495::TaskManager.Task::(_debug) Task 1eb91a55-8dfe-450b-87a5-c8ac33861ea6: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:33:44,589::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c4aa3ac7-4f19-48f4-83c3-6fd0d23a6fbd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:33:44,590::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:33:44,591::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:33:44,592::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c4aa3ac7-4f19-48f4-83c3-6fd0d23a6fbd`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:33:44,593::task::495::TaskManager.Task::(_debug) Task 1eb91a55-8dfe-450b-87a5-c8ac33861ea6: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:33:44,594::task::495::TaskManager.Task::(_debug) Task 1eb91a55-8dfe-450b-87a5-c8ac33861ea6: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:33:44,595::task::495::TaskManager.Task::(_debug) Task 1eb91a55-8dfe-450b-87a5-c8ac33861ea6: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:33:44,595::task::495::TaskManager.Task::(_debug) Task 1eb91a55-8dfe-450b-87a5-c8ac33861ea6: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:33:44,596::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:33:44,596::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:33:44,597::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:33:44,597::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:33:44,598::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:33:44,598::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:33:44,599::task::495::TaskManager.Task::(_debug) Task 1eb91a55-8dfe-450b-87a5-c8ac33861ea6: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:33:44,604::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15914540::DEBUG::2013-11-20 03:33:45,121::clientIF::54::vds::(wrapper) [192.168.100.135]::call getVmStats with ('e97d1c52-b688-4192-bdeb-2dfadcc79786',) {} >Thread-15914540::DEBUG::2013-11-20 03:33:45,122::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Disk stats not available >Thread-15914540::DEBUG::2013-11-20 03:33:45,122::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Disk latency not available >Thread-15914540::DEBUG::2013-11-20 03:33:45,123::clientIF::59::vds::(wrapper) return getVmStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{'status': 'Up', 'username': 'Unknown', 'memUsage': '0', 'acpiEnable': 'true', 'pid': '7039', 'displayIp': '0', 'displayPort': u'5916', 'session': 'Unknown', 'displaySecurePort': u'5917', 'cdrom': '/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111/RHEV-toolsSetup_3.0_39.iso', 'pauseCode': 'NOERR', 'clientIp': '', 'kvmEnable': 'true', 'network': {u'vnet9': {'macAddr': u'00:1a:4a:a8:64:13', 'rxDropped': '0', 'rxErrors': '0', 'txDropped': '0', 'txRate': '0.0', 'rxRate': '0.0', 'txErrors': '0', 'state': 'unknown', 'speed': '1000', 'name': u'vnet9'}}, 'vmId': 'e97d1c52-b688-4192-bdeb-2dfadcc79786', 'monitorResponse': '0', 'cpuUser': '51.69', 'disks': {u'hda': {'readLatency': '0', 'apparentsize': '86033563648', 'writeLatency': '0', 'imageID': '5af747d8-95b1-4b7e-af53-cb77e4fcc896', 'flushLatency': '0', 'truesize': '86033563648'}}, 'boot': 'c', 'statsAge': '0.54', 'cpuIdle': '48.31', 'elapsedTime': '62', 'vmType': 'kvm', 'cpuSys': '0.00', 'timeOffset': '0', 'appsList': [], 'guestIPs': '', 'displayType': 'qxl', 'nice': ''}]} >Dummy-15912651::DEBUG::2013-11-20 03:33:45,723::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:46,082::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0477413 s, 21.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:33:48,092::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:48,548::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0571942 s, 17.9 MB/s\n'; <rc> = 0 >Thread-15914542::DEBUG::2013-11-20 03:33:48,716::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914542::INFO::2013-11-20 03:33:48,717::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914542::DEBUG::2013-11-20 03:33:48,718::task::495::TaskManager.Task::(_debug) Task 9ca5293f-ff85-4844-8ac3-5e7d827d2a4b: moving from state init -> state preparing >Thread-15914542::DEBUG::2013-11-20 03:33:48,719::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914542::DEBUG::2013-11-20 03:33:48,720::task::495::TaskManager.Task::(_debug) Task 9ca5293f-ff85-4844-8ac3-5e7d827d2a4b: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914542::DEBUG::2013-11-20 03:33:48,721::task::495::TaskManager.Task::(_debug) Task 9ca5293f-ff85-4844-8ac3-5e7d827d2a4b: moving from state preparing -> state finished >Thread-15914542::DEBUG::2013-11-20 03:33:48,721::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914542::DEBUG::2013-11-20 03:33:48,722::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914542::DEBUG::2013-11-20 03:33:48,723::task::495::TaskManager.Task::(_debug) Task 9ca5293f-ff85-4844-8ac3-5e7d827d2a4b: ref 0 aborting False >Thread-15914542::INFO::2013-11-20 03:33:48,723::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914543::DEBUG::2013-11-20 03:33:48,740::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914543::INFO::2013-11-20 03:33:48,741::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914543::DEBUG::2013-11-20 03:33:48,742::task::495::TaskManager.Task::(_debug) Task 1671377d-be56-4380-823b-519a3479e090: moving from state init -> state preparing >Thread-15914543::DEBUG::2013-11-20 03:33:48,746::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5a134d03-f077-4f7e-9bc2-973c45c1e7f7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914543::DEBUG::2013-11-20 03:33:48,747::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914543::DEBUG::2013-11-20 03:33:48,748::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914543::DEBUG::2013-11-20 03:33:48,749::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`5a134d03-f077-4f7e-9bc2-973c45c1e7f7`::Granted request >Thread-15914543::DEBUG::2013-11-20 03:33:48,753::task::495::TaskManager.Task::(_debug) Task 1671377d-be56-4380-823b-519a3479e090: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914543::DEBUG::2013-11-20 03:33:48,754::task::495::TaskManager.Task::(_debug) Task 1671377d-be56-4380-823b-519a3479e090: ref 1 aborting False >Thread-15914543::WARNING::2013-11-20 03:33:48,758::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914543::DEBUG::2013-11-20 03:33:48,762::task::495::TaskManager.Task::(_debug) Task 1671377d-be56-4380-823b-519a3479e090: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914543::DEBUG::2013-11-20 03:33:48,763::task::495::TaskManager.Task::(_debug) Task 1671377d-be56-4380-823b-519a3479e090: moving from state preparing -> state finished >Thread-15914543::DEBUG::2013-11-20 03:33:48,763::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914543::DEBUG::2013-11-20 03:33:48,764::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914543::DEBUG::2013-11-20 03:33:48,765::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914543::DEBUG::2013-11-20 03:33:48,766::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914543::DEBUG::2013-11-20 03:33:48,767::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914543::DEBUG::2013-11-20 03:33:48,768::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914543::DEBUG::2013-11-20 03:33:48,769::task::495::TaskManager.Task::(_debug) Task 1671377d-be56-4380-823b-519a3479e090: ref 0 aborting False >Thread-15914543::INFO::2013-11-20 03:33:48,770::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:33:50,559::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:50,962::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0515776 s, 19.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:33:52,972::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:53,416::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0490247 s, 20.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:33:55,431::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:55,865::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0472158 s, 21.7 MB/s\n'; <rc> = 0 >Thread-15914546::INFO::2013-11-20 03:33:57,452::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914546::DEBUG::2013-11-20 03:33:57,453::task::495::TaskManager.Task::(_debug) Task 4a71ddfb-cb93-4878-a24f-c7d85c117342: moving from state init -> state preparing >Thread-15914546::DEBUG::2013-11-20 03:33:57,454::task::495::TaskManager.Task::(_debug) Task 4a71ddfb-cb93-4878-a24f-c7d85c117342: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000285148620605', 'lastCheck': 1384918437.2874401, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00111794471741', 'lastCheck': 1384918433.2869589, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000910997390747', 'lastCheck': 1384918436.7378831, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000389099121094', 'lastCheck': 1384918431.7309451, 'valid': True, 'code': 0}} >Thread-15914546::DEBUG::2013-11-20 03:33:57,454::task::495::TaskManager.Task::(_debug) Task 4a71ddfb-cb93-4878-a24f-c7d85c117342: moving from state preparing -> state finished >Thread-15914546::DEBUG::2013-11-20 03:33:57,455::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914546::DEBUG::2013-11-20 03:33:57,455::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914546::DEBUG::2013-11-20 03:33:57,456::task::495::TaskManager.Task::(_debug) Task 4a71ddfb-cb93-4878-a24f-c7d85c117342: ref 0 aborting False >Thread-15914546::INFO::2013-11-20 03:33:57,456::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000285148620605', 'lastCheck': 1384918437.2874401, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00111794471741', 'lastCheck': 1384918433.2869589, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000910997390747', 'lastCheck': 1384918436.7378831, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000389099121094', 'lastCheck': 1384918431.7309451, 'valid': True, 'code': 0}} >Thread-15914547::DEBUG::2013-11-20 03:33:57,522::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Disk stats not available >Thread-15914547::DEBUG::2013-11-20 03:33:57,523::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Disk latency not available >Dummy-15912651::DEBUG::2013-11-20 03:33:57,875::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:33:58,387::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.061715 s, 16.6 MB/s\n'; <rc> = 0 >Thread-15914548::DEBUG::2013-11-20 03:33:59,328::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914548::INFO::2013-11-20 03:33:59,329::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914548::DEBUG::2013-11-20 03:33:59,330::task::495::TaskManager.Task::(_debug) Task 88569765-0000-410d-b85c-73921519231e: moving from state init -> state preparing >Thread-15914548::DEBUG::2013-11-20 03:33:59,331::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914548::DEBUG::2013-11-20 03:33:59,331::task::495::TaskManager.Task::(_debug) Task 88569765-0000-410d-b85c-73921519231e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914548::DEBUG::2013-11-20 03:33:59,332::task::495::TaskManager.Task::(_debug) Task 88569765-0000-410d-b85c-73921519231e: moving from state preparing -> state finished >Thread-15914548::DEBUG::2013-11-20 03:33:59,332::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914548::DEBUG::2013-11-20 03:33:59,333::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914548::DEBUG::2013-11-20 03:33:59,333::task::495::TaskManager.Task::(_debug) Task 88569765-0000-410d-b85c-73921519231e: ref 0 aborting False >Thread-15914548::INFO::2013-11-20 03:33:59,334::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914549::DEBUG::2013-11-20 03:33:59,340::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914549::INFO::2013-11-20 03:33:59,341::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914549::DEBUG::2013-11-20 03:33:59,341::task::495::TaskManager.Task::(_debug) Task f91a39f3-35f3-4b91-9089-5316d1f61ec6: moving from state init -> state preparing >Thread-15914549::DEBUG::2013-11-20 03:33:59,342::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`19dbde40-010f-42ce-9e7d-a938da84e20b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914549::DEBUG::2013-11-20 03:33:59,342::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914549::DEBUG::2013-11-20 03:33:59,343::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914549::DEBUG::2013-11-20 03:33:59,343::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`19dbde40-010f-42ce-9e7d-a938da84e20b`::Granted request >Thread-15914549::DEBUG::2013-11-20 03:33:59,344::task::495::TaskManager.Task::(_debug) Task f91a39f3-35f3-4b91-9089-5316d1f61ec6: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914549::DEBUG::2013-11-20 03:33:59,348::task::495::TaskManager.Task::(_debug) Task f91a39f3-35f3-4b91-9089-5316d1f61ec6: ref 1 aborting False >Thread-15914549::WARNING::2013-11-20 03:33:59,350::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914549::DEBUG::2013-11-20 03:33:59,350::task::495::TaskManager.Task::(_debug) Task f91a39f3-35f3-4b91-9089-5316d1f61ec6: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914549::DEBUG::2013-11-20 03:33:59,351::task::495::TaskManager.Task::(_debug) Task f91a39f3-35f3-4b91-9089-5316d1f61ec6: moving from state preparing -> state finished >Thread-15914549::DEBUG::2013-11-20 03:33:59,351::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914549::DEBUG::2013-11-20 03:33:59,352::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914549::DEBUG::2013-11-20 03:33:59,352::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914549::DEBUG::2013-11-20 03:33:59,352::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914549::DEBUG::2013-11-20 03:33:59,353::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914549::DEBUG::2013-11-20 03:33:59,353::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914549::DEBUG::2013-11-20 03:33:59,353::task::495::TaskManager.Task::(_debug) Task f91a39f3-35f3-4b91-9089-5316d1f61ec6: ref 0 aborting False >Thread-15914549::INFO::2013-11-20 03:33:59,354::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15913658::INFO::2013-11-20 03:33:59,608::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:33:59,609::task::495::TaskManager.Task::(_debug) Task 78ed25f3-f2eb-4b37-8627-31d657a7faa9: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:33:59,610::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`59f5f36c-c313-4863-be6c-f809883150a9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:33:59,610::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:33:59,611::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:33:59,611::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`59f5f36c-c313-4863-be6c-f809883150a9`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:33:59,612::task::495::TaskManager.Task::(_debug) Task 78ed25f3-f2eb-4b37-8627-31d657a7faa9: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:33:59,612::task::495::TaskManager.Task::(_debug) Task 78ed25f3-f2eb-4b37-8627-31d657a7faa9: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:33:59,613::task::495::TaskManager.Task::(_debug) Task 78ed25f3-f2eb-4b37-8627-31d657a7faa9: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:33:59,613::task::495::TaskManager.Task::(_debug) Task 78ed25f3-f2eb-4b37-8627-31d657a7faa9: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:33:59,614::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:33:59,614::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:33:59,615::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:33:59,615::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:33:59,616::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:33:59,616::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:33:59,617::task::495::TaskManager.Task::(_debug) Task 78ed25f3-f2eb-4b37-8627-31d657a7faa9: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:33:59,618::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:34:00,397::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:34:01,851::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0504666 s, 20.3 MB/s\n'; <rc> = 0 >Thread-15914436::DEBUG::2013-11-20 03:34:03,372::guestIF::97::vm.Vm::(_handleMessage) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::u'None' >Dummy-15912651::DEBUG::2013-11-20 03:34:03,861::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:34:04,310::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0559272 s, 18.3 MB/s\n'; <rc> = 0 >Thread-15912392::INFO::2013-11-20 03:34:04,736::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:34:04,737::task::495::TaskManager.Task::(_debug) Task 8a13a4e5-8786-4905-9497-a655304146a8: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:34:04,739::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e71562e9-64ab-41bc-9634-2db524069139`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:34:04,740::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:34:04,740::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:34:04,741::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e71562e9-64ab-41bc-9634-2db524069139`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:34:04,742::task::495::TaskManager.Task::(_debug) Task 8a13a4e5-8786-4905-9497-a655304146a8: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:34:04,742::task::495::TaskManager.Task::(_debug) Task 8a13a4e5-8786-4905-9497-a655304146a8: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:34:04,743::task::495::TaskManager.Task::(_debug) Task 8a13a4e5-8786-4905-9497-a655304146a8: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:34:04,743::task::495::TaskManager.Task::(_debug) Task 8a13a4e5-8786-4905-9497-a655304146a8: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:34:04,743::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:34:04,744::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:34:04,744::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:34:04,745::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:34:04,745::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:34:04,745::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:34:04,746::task::495::TaskManager.Task::(_debug) Task 8a13a4e5-8786-4905-9497-a655304146a8: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:34:04,746::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:34:06,321::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913695::INFO::2013-11-20 03:34:06,499::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:34:06,501::task::495::TaskManager.Task::(_debug) Task 71ad15f7-895e-406e-a20f-fe1257b2f516: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:34:06,503::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f35a4b0b-9931-494e-ac8a-e7ca5d4d6138`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:34:06,504::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:34:06,505::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:34:06,505::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f35a4b0b-9931-494e-ac8a-e7ca5d4d6138`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:34:06,506::task::495::TaskManager.Task::(_debug) Task 71ad15f7-895e-406e-a20f-fe1257b2f516: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:34:06,506::task::495::TaskManager.Task::(_debug) Task 71ad15f7-895e-406e-a20f-fe1257b2f516: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:34:06,507::task::495::TaskManager.Task::(_debug) Task 71ad15f7-895e-406e-a20f-fe1257b2f516: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:34:06,507::task::495::TaskManager.Task::(_debug) Task 71ad15f7-895e-406e-a20f-fe1257b2f516: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:34:06,507::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:34:06,508::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:34:06,508::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:34:06,509::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:34:06,509::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:34:06,509::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:34:06,510::task::495::TaskManager.Task::(_debug) Task 71ad15f7-895e-406e-a20f-fe1257b2f516: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:34:06,510::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:34:06,758::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0522821 s, 19.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:34:08,769::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:34:09,194::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0476029 s, 21.5 MB/s\n'; <rc> = 0 >Thread-15914553::DEBUG::2013-11-20 03:34:10,081::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914553::INFO::2013-11-20 03:34:10,082::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914553::DEBUG::2013-11-20 03:34:10,083::task::495::TaskManager.Task::(_debug) Task 4d471e2d-96e6-4c4e-b76f-c1089317b351: moving from state init -> state preparing >Thread-15914553::DEBUG::2013-11-20 03:34:10,084::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914553::DEBUG::2013-11-20 03:34:10,084::task::495::TaskManager.Task::(_debug) Task 4d471e2d-96e6-4c4e-b76f-c1089317b351: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914553::DEBUG::2013-11-20 03:34:10,084::task::495::TaskManager.Task::(_debug) Task 4d471e2d-96e6-4c4e-b76f-c1089317b351: moving from state preparing -> state finished >Thread-15914553::DEBUG::2013-11-20 03:34:10,085::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914553::DEBUG::2013-11-20 03:34:10,085::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914553::DEBUG::2013-11-20 03:34:10,086::task::495::TaskManager.Task::(_debug) Task 4d471e2d-96e6-4c4e-b76f-c1089317b351: ref 0 aborting False >Thread-15914553::INFO::2013-11-20 03:34:10,086::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914554::DEBUG::2013-11-20 03:34:10,099::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914554::INFO::2013-11-20 03:34:10,099::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914554::DEBUG::2013-11-20 03:34:10,100::task::495::TaskManager.Task::(_debug) Task 1885de10-53b1-4f48-bcaf-dfd53df8de21: moving from state init -> state preparing >Thread-15914554::DEBUG::2013-11-20 03:34:10,111::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`040ebcbe-c28f-4a11-8d50-628ddedd05da`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914554::DEBUG::2013-11-20 03:34:10,127::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914554::DEBUG::2013-11-20 03:34:10,135::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914554::DEBUG::2013-11-20 03:34:10,136::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`040ebcbe-c28f-4a11-8d50-628ddedd05da`::Granted request >Thread-15914554::DEBUG::2013-11-20 03:34:10,136::task::495::TaskManager.Task::(_debug) Task 1885de10-53b1-4f48-bcaf-dfd53df8de21: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914554::DEBUG::2013-11-20 03:34:10,137::task::495::TaskManager.Task::(_debug) Task 1885de10-53b1-4f48-bcaf-dfd53df8de21: ref 1 aborting False >Thread-15914554::WARNING::2013-11-20 03:34:10,138::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914554::DEBUG::2013-11-20 03:34:10,139::task::495::TaskManager.Task::(_debug) Task 1885de10-53b1-4f48-bcaf-dfd53df8de21: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914554::DEBUG::2013-11-20 03:34:10,139::task::495::TaskManager.Task::(_debug) Task 1885de10-53b1-4f48-bcaf-dfd53df8de21: moving from state preparing -> state finished >Thread-15914554::DEBUG::2013-11-20 03:34:10,140::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914554::DEBUG::2013-11-20 03:34:10,140::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914554::DEBUG::2013-11-20 03:34:10,141::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914554::DEBUG::2013-11-20 03:34:10,141::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914554::DEBUG::2013-11-20 03:34:10,142::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914554::DEBUG::2013-11-20 03:34:10,142::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914554::DEBUG::2013-11-20 03:34:10,142::task::495::TaskManager.Task::(_debug) Task 1885de10-53b1-4f48-bcaf-dfd53df8de21: ref 0 aborting False >Thread-15914554::INFO::2013-11-20 03:34:10,143::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:34:11,204::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:34:11,609::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0502048 s, 20.4 MB/s\n'; <rc> = 0 >Thread-15914556::INFO::2013-11-20 03:34:13,402::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914556::DEBUG::2013-11-20 03:34:13,402::task::495::TaskManager.Task::(_debug) Task 8a99443b-c742-4096-8794-0096a606b531: moving from state init -> state preparing >Thread-15914556::DEBUG::2013-11-20 03:34:13,403::task::495::TaskManager.Task::(_debug) Task 8a99443b-c742-4096-8794-0096a606b531: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000276803970337', 'lastCheck': 1384918447.2985301, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147891044617', 'lastCheck': 1384918453.3734889, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00186204910278', 'lastCheck': 1384918446.75231, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000308990478516', 'lastCheck': 1384918451.750294, 'valid': True, 'code': 0}} >Thread-15914556::DEBUG::2013-11-20 03:34:13,403::task::495::TaskManager.Task::(_debug) Task 8a99443b-c742-4096-8794-0096a606b531: moving from state preparing -> state finished >Thread-15914556::DEBUG::2013-11-20 03:34:13,404::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914556::DEBUG::2013-11-20 03:34:13,404::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914556::DEBUG::2013-11-20 03:34:13,404::task::495::TaskManager.Task::(_debug) Task 8a99443b-c742-4096-8794-0096a606b531: ref 0 aborting False >Thread-15914556::INFO::2013-11-20 03:34:13,405::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000276803970337', 'lastCheck': 1384918447.2985301, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00147891044617', 'lastCheck': 1384918453.3734889, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00186204910278', 'lastCheck': 1384918446.75231, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000308990478516', 'lastCheck': 1384918451.750294, 'valid': True, 'code': 0}} >Thread-15914557::DEBUG::2013-11-20 03:34:13,466::libvirtvm::222::vm.Vm::(_getDiskStats) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Disk stats not available >Thread-15914557::DEBUG::2013-11-20 03:34:13,467::libvirtvm::251::vm.Vm::(_getDiskLatency) vmId=`e97d1c52-b688-4192-bdeb-2dfadcc79786`::Disk latency not available >Dummy-15912651::DEBUG::2013-11-20 03:34:13,620::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:34:14,015::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0493171 s, 20.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:34:16,026::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:34:16,470::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0550574 s, 18.6 MB/s\n'; <rc> = 0 >Thread-15914435::INFO::2013-11-20 03:34:16,538::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=143cb26d-de08-4d30-89cf-aee10245b66a volUUID=d7028f7c-0551-4bbe-a907-6f804cc9bd54) >Thread-15914435::DEBUG::2013-11-20 03:34:16,539::task::495::TaskManager.Task::(_debug) Task 55d1a819-ec4c-470d-bb28-8cbfb31dbbd9: moving from state init -> state preparing >Thread-15914435::DEBUG::2013-11-20 03:34:16,540::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f34cee40-71f1-43e0-9b20-2b1e9a2176bd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914435::DEBUG::2013-11-20 03:34:16,540::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914435::DEBUG::2013-11-20 03:34:16,541::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914435::DEBUG::2013-11-20 03:34:16,541::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f34cee40-71f1-43e0-9b20-2b1e9a2176bd`::Granted request >Thread-15914435::DEBUG::2013-11-20 03:34:16,542::task::495::TaskManager.Task::(_debug) Task 55d1a819-ec4c-470d-bb28-8cbfb31dbbd9: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914435::DEBUG::2013-11-20 03:34:16,543::task::495::TaskManager.Task::(_debug) Task 55d1a819-ec4c-470d-bb28-8cbfb31dbbd9: ref 1 aborting False >Thread-15914435::DEBUG::2013-11-20 03:34:16,543::task::495::TaskManager.Task::(_debug) Task 55d1a819-ec4c-470d-bb28-8cbfb31dbbd9: finished: {'truesize': '56908316672', 'apparentsize': '56908316672'} >Thread-15914435::DEBUG::2013-11-20 03:34:16,544::task::495::TaskManager.Task::(_debug) Task 55d1a819-ec4c-470d-bb28-8cbfb31dbbd9: moving from state preparing -> state finished >Thread-15914435::DEBUG::2013-11-20 03:34:16,544::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914435::DEBUG::2013-11-20 03:34:16,544::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914435::DEBUG::2013-11-20 03:34:16,545::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914435::DEBUG::2013-11-20 03:34:16,545::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914435::DEBUG::2013-11-20 03:34:16,546::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914435::DEBUG::2013-11-20 03:34:16,546::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914435::DEBUG::2013-11-20 03:34:16,547::task::495::TaskManager.Task::(_debug) Task 55d1a819-ec4c-470d-bb28-8cbfb31dbbd9: ref 0 aborting False >Thread-15914435::INFO::2013-11-20 03:34:16,547::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '56908316672', 'apparentsize': '56908316672'} >Dummy-15912651::DEBUG::2013-11-20 03:34:18,480::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:34:19,140::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.252569 s, 4.1 MB/s\n'; <rc> = 0 >Thread-15914453::INFO::2013-11-20 03:34:19,784::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=5f930ef3-763f-4294-a62a-6d52df220f8c volUUID=b9cffdbe-152e-41ce-ae5d-08db7c08454e) >Thread-15914453::DEBUG::2013-11-20 03:34:19,788::task::495::TaskManager.Task::(_debug) Task 80e453fb-2e41-425e-8543-f3155981aef7: moving from state init -> state preparing >Thread-15914453::DEBUG::2013-11-20 03:34:19,791::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`1cb4aa7b-f8e5-45a1-95e0-8f63e5ebc01e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914453::DEBUG::2013-11-20 03:34:19,793::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914453::DEBUG::2013-11-20 03:34:19,800::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914453::DEBUG::2013-11-20 03:34:19,801::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`1cb4aa7b-f8e5-45a1-95e0-8f63e5ebc01e`::Granted request >Thread-15914453::DEBUG::2013-11-20 03:34:19,803::task::495::TaskManager.Task::(_debug) Task 80e453fb-2e41-425e-8543-f3155981aef7: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914453::DEBUG::2013-11-20 03:34:19,803::task::495::TaskManager.Task::(_debug) Task 80e453fb-2e41-425e-8543-f3155981aef7: ref 1 aborting False >Thread-15914453::DEBUG::2013-11-20 03:34:19,804::task::495::TaskManager.Task::(_debug) Task 80e453fb-2e41-425e-8543-f3155981aef7: finished: {'truesize': '53821308928', 'apparentsize': '53821308928'} >Thread-15914453::DEBUG::2013-11-20 03:34:19,804::task::495::TaskManager.Task::(_debug) Task 80e453fb-2e41-425e-8543-f3155981aef7: moving from state preparing -> state finished >Thread-15914453::DEBUG::2013-11-20 03:34:19,805::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914453::DEBUG::2013-11-20 03:34:19,805::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914453::DEBUG::2013-11-20 03:34:19,806::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914453::DEBUG::2013-11-20 03:34:19,806::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914453::DEBUG::2013-11-20 03:34:19,807::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914453::DEBUG::2013-11-20 03:34:19,807::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914453::DEBUG::2013-11-20 03:34:19,808::task::495::TaskManager.Task::(_debug) Task 80e453fb-2e41-425e-8543-f3155981aef7: ref 0 aborting False >Thread-15914453::INFO::2013-11-20 03:34:19,808::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '53821308928', 'apparentsize': '53821308928'} >Thread-15914560::DEBUG::2013-11-20 03:34:20,838::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914560::INFO::2013-11-20 03:34:20,839::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914560::DEBUG::2013-11-20 03:34:20,840::task::495::TaskManager.Task::(_debug) Task 5baae15d-2c85-4f25-a63f-c8e4ad296ed6: moving from state init -> state preparing >Thread-15914560::DEBUG::2013-11-20 03:34:20,840::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914560::DEBUG::2013-11-20 03:34:20,841::task::495::TaskManager.Task::(_debug) Task 5baae15d-2c85-4f25-a63f-c8e4ad296ed6: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914560::DEBUG::2013-11-20 03:34:20,841::task::495::TaskManager.Task::(_debug) Task 5baae15d-2c85-4f25-a63f-c8e4ad296ed6: moving from state preparing -> state finished >Thread-15914560::DEBUG::2013-11-20 03:34:20,842::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914560::DEBUG::2013-11-20 03:34:20,842::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914560::DEBUG::2013-11-20 03:34:20,842::task::495::TaskManager.Task::(_debug) Task 5baae15d-2c85-4f25-a63f-c8e4ad296ed6: ref 0 aborting False >Thread-15914560::INFO::2013-11-20 03:34:20,843::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914561::DEBUG::2013-11-20 03:34:20,852::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914561::INFO::2013-11-20 03:34:20,853::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914561::DEBUG::2013-11-20 03:34:20,853::task::495::TaskManager.Task::(_debug) Task d4377859-f1b3-4946-b8e8-e91a96f06efd: moving from state init -> state preparing >Thread-15914561::DEBUG::2013-11-20 03:34:20,859::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a8eb0bbe-7639-4a7c-bf99-fe1daff69525`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914561::DEBUG::2013-11-20 03:34:20,860::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914561::DEBUG::2013-11-20 03:34:20,861::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914561::DEBUG::2013-11-20 03:34:20,861::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a8eb0bbe-7639-4a7c-bf99-fe1daff69525`::Granted request >Thread-15914561::DEBUG::2013-11-20 03:34:20,862::task::495::TaskManager.Task::(_debug) Task d4377859-f1b3-4946-b8e8-e91a96f06efd: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914561::DEBUG::2013-11-20 03:34:20,862::task::495::TaskManager.Task::(_debug) Task d4377859-f1b3-4946-b8e8-e91a96f06efd: ref 1 aborting False >Thread-15914561::WARNING::2013-11-20 03:34:20,864::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914561::DEBUG::2013-11-20 03:34:20,865::task::495::TaskManager.Task::(_debug) Task d4377859-f1b3-4946-b8e8-e91a96f06efd: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914561::DEBUG::2013-11-20 03:34:20,866::task::495::TaskManager.Task::(_debug) Task d4377859-f1b3-4946-b8e8-e91a96f06efd: moving from state preparing -> state finished >Thread-15914561::DEBUG::2013-11-20 03:34:20,866::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914561::DEBUG::2013-11-20 03:34:20,867::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914561::DEBUG::2013-11-20 03:34:20,867::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914561::DEBUG::2013-11-20 03:34:20,868::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914561::DEBUG::2013-11-20 03:34:20,868::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914561::DEBUG::2013-11-20 03:34:20,869::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914561::DEBUG::2013-11-20 03:34:20,869::task::495::TaskManager.Task::(_debug) Task d4377859-f1b3-4946-b8e8-e91a96f06efd: ref 0 aborting False >Thread-15914561::INFO::2013-11-20 03:34:20,870::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:34:21,151::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913547::DEBUG::2013-11-20 03:34:21,324::guestIF::97::vm.Vm::(_handleMessage) vmId=`6574d88c-00ac-46f4-99ce-a2726932bfda`::u'None' >Dummy-15912651::DEBUG::2013-11-20 03:34:21,576::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0481104 s, 21.3 MB/s\n'; <rc> = 0 >Thread-15914519::INFO::2013-11-20 03:34:23,032::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=5af747d8-95b1-4b7e-af53-cb77e4fcc896 volUUID=8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f) >Thread-15914519::DEBUG::2013-11-20 03:34:23,043::task::495::TaskManager.Task::(_debug) Task 2c1a6768-0fd1-46be-92ff-d35a85e503a0: moving from state init -> state preparing >Thread-15914519::DEBUG::2013-11-20 03:34:23,053::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`843ce11a-0ea2-4824-aa6a-9ed11765d9d5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914519::DEBUG::2013-11-20 03:34:23,054::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914519::DEBUG::2013-11-20 03:34:23,055::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914519::DEBUG::2013-11-20 03:34:23,056::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`843ce11a-0ea2-4824-aa6a-9ed11765d9d5`::Granted request >Thread-15914519::DEBUG::2013-11-20 03:34:23,066::task::495::TaskManager.Task::(_debug) Task 2c1a6768-0fd1-46be-92ff-d35a85e503a0: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914519::DEBUG::2013-11-20 03:34:23,071::task::495::TaskManager.Task::(_debug) Task 2c1a6768-0fd1-46be-92ff-d35a85e503a0: ref 1 aborting False >Thread-15914519::DEBUG::2013-11-20 03:34:23,071::task::495::TaskManager.Task::(_debug) Task 2c1a6768-0fd1-46be-92ff-d35a85e503a0: finished: {'truesize': '86033563648', 'apparentsize': '86033563648'} >Thread-15914519::DEBUG::2013-11-20 03:34:23,072::task::495::TaskManager.Task::(_debug) Task 2c1a6768-0fd1-46be-92ff-d35a85e503a0: moving from state preparing -> state finished >Thread-15914519::DEBUG::2013-11-20 03:34:23,073::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914519::DEBUG::2013-11-20 03:34:23,074::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914519::DEBUG::2013-11-20 03:34:23,075::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914519::DEBUG::2013-11-20 03:34:23,075::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914519::DEBUG::2013-11-20 03:34:23,076::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914519::DEBUG::2013-11-20 03:34:23,077::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914519::DEBUG::2013-11-20 03:34:23,077::task::495::TaskManager.Task::(_debug) Task 2c1a6768-0fd1-46be-92ff-d35a85e503a0: ref 0 aborting False >Thread-15914519::INFO::2013-11-20 03:34:23,078::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '86033563648', 'apparentsize': '86033563648'} >Dummy-15912651::DEBUG::2013-11-20 03:34:23,585::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:34:24,064::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0525048 s, 19.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:34:26,075::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:34:26,483::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0477096 s, 21.5 MB/s\n'; <rc> = 0 >Thread-15913518::INFO::2013-11-20 03:34:27,104::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:34:27,104::task::495::TaskManager.Task::(_debug) Task d04463b7-1a32-4145-ba3f-253f1be0afd1: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:34:27,110::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0f87fe20-0fba-4c56-ba45-13d1024377e6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:34:27,111::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:34:27,111::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:34:27,112::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0f87fe20-0fba-4c56-ba45-13d1024377e6`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:34:27,112::task::495::TaskManager.Task::(_debug) Task d04463b7-1a32-4145-ba3f-253f1be0afd1: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:34:27,113::task::495::TaskManager.Task::(_debug) Task d04463b7-1a32-4145-ba3f-253f1be0afd1: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:34:27,113::task::495::TaskManager.Task::(_debug) Task d04463b7-1a32-4145-ba3f-253f1be0afd1: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:34:27,113::task::495::TaskManager.Task::(_debug) Task d04463b7-1a32-4145-ba3f-253f1be0afd1: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:34:27,114::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:34:27,114::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:34:27,115::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:34:27,115::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:34:27,115::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:34:27,116::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:34:27,116::task::495::TaskManager.Task::(_debug) Task d04463b7-1a32-4145-ba3f-253f1be0afd1: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:34:27,116::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:34:27,117::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:34:27,117::task::495::TaskManager.Task::(_debug) Task 89e148b0-1ab5-4b5f-9e9c-b5e78ef1f658: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:34:27,118::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`3d26ae06-df56-4cc1-8759-dc1ec3c2029e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:34:27,118::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:34:27,118::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:34:27,119::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`3d26ae06-df56-4cc1-8759-dc1ec3c2029e`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:34:27,119::task::495::TaskManager.Task::(_debug) Task 89e148b0-1ab5-4b5f-9e9c-b5e78ef1f658: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:34:27,120::task::495::TaskManager.Task::(_debug) Task 89e148b0-1ab5-4b5f-9e9c-b5e78ef1f658: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:34:27,120::task::495::TaskManager.Task::(_debug) Task 89e148b0-1ab5-4b5f-9e9c-b5e78ef1f658: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:34:27,120::task::495::TaskManager.Task::(_debug) Task 89e148b0-1ab5-4b5f-9e9c-b5e78ef1f658: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:34:27,121::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:34:27,121::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:34:27,121::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:34:27,122::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:34:27,122::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:34:27,122::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:34:27,123::task::495::TaskManager.Task::(_debug) Task 89e148b0-1ab5-4b5f-9e9c-b5e78ef1f658: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:34:27,123::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:34:28,493::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914564::INFO::2013-11-20 03:34:29,310::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914564::DEBUG::2013-11-20 03:34:29,311::task::495::TaskManager.Task::(_debug) Task 16febbc3-d896-40fd-a952-c1e39325c811: moving from state init -> state preparing >Thread-15914564::DEBUG::2013-11-20 03:34:29,312::task::495::TaskManager.Task::(_debug) Task 16febbc3-d896-40fd-a952-c1e39325c811: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000256061553955', 'lastCheck': 1384918467.3858111, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0106360912323', 'lastCheck': 1384918463.3950641, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000881910324097', 'lastCheck': 1384918467.0397849, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000660181045532', 'lastCheck': 1384918462.0317309, 'valid': True, 'code': 0}} >Thread-15914564::DEBUG::2013-11-20 03:34:29,313::task::495::TaskManager.Task::(_debug) Task 16febbc3-d896-40fd-a952-c1e39325c811: moving from state preparing -> state finished >Thread-15914564::DEBUG::2013-11-20 03:34:29,314::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914564::DEBUG::2013-11-20 03:34:29,314::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914564::DEBUG::2013-11-20 03:34:29,315::task::495::TaskManager.Task::(_debug) Task 16febbc3-d896-40fd-a952-c1e39325c811: ref 0 aborting False >Thread-15914564::INFO::2013-11-20 03:34:29,315::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000256061553955', 'lastCheck': 1384918467.3858111, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0106360912323', 'lastCheck': 1384918463.3950641, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000881910324097', 'lastCheck': 1384918467.0397849, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000660181045532', 'lastCheck': 1384918462.0317309, 'valid': True, 'code': 0}} >Thread-15913546::INFO::2013-11-20 03:34:29,804::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:34:29,805::task::495::TaskManager.Task::(_debug) Task 8b53dd1a-40e0-48b5-b8be-06eea0f58540: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:34:29,806::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fa215797-3550-496f-b0d3-691ff4216507`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:34:29,807::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:34:29,808::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:34:29,808::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`fa215797-3550-496f-b0d3-691ff4216507`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:34:29,809::task::495::TaskManager.Task::(_debug) Task 8b53dd1a-40e0-48b5-b8be-06eea0f58540: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:34:29,809::task::495::TaskManager.Task::(_debug) Task 8b53dd1a-40e0-48b5-b8be-06eea0f58540: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:34:29,810::task::495::TaskManager.Task::(_debug) Task 8b53dd1a-40e0-48b5-b8be-06eea0f58540: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:34:29,810::task::495::TaskManager.Task::(_debug) Task 8b53dd1a-40e0-48b5-b8be-06eea0f58540: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:34:29,811::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:34:29,811::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:34:29,812::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:34:29,812::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:34:29,813::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:34:29,813::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:34:29,814::task::495::TaskManager.Task::(_debug) Task 8b53dd1a-40e0-48b5-b8be-06eea0f58540: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:34:29,814::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:34:29,815::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:34:29,815::task::495::TaskManager.Task::(_debug) Task 9913c84a-a44e-4640-b5c8-d5eb7c0ebe37: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:34:29,816::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`404895b7-ac02-43c3-b914-9675c58770f5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:34:29,816::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:34:29,817::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:34:29,817::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`404895b7-ac02-43c3-b914-9675c58770f5`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:34:29,818::task::495::TaskManager.Task::(_debug) Task 9913c84a-a44e-4640-b5c8-d5eb7c0ebe37: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:34:29,818::task::495::TaskManager.Task::(_debug) Task 9913c84a-a44e-4640-b5c8-d5eb7c0ebe37: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:34:29,819::task::495::TaskManager.Task::(_debug) Task 9913c84a-a44e-4640-b5c8-d5eb7c0ebe37: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:34:29,819::task::495::TaskManager.Task::(_debug) Task 9913c84a-a44e-4640-b5c8-d5eb7c0ebe37: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:34:29,820::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:34:29,820::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:34:29,821::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:34:29,821::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:34:29,821::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:34:29,822::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:34:29,822::task::495::TaskManager.Task::(_debug) Task 9913c84a-a44e-4640-b5c8-d5eb7c0ebe37: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:34:29,823::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:34:29,823::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:34:29,824::task::495::TaskManager.Task::(_debug) Task 7eaeca28-0767-47dd-a86c-a13f5b764d12: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:34:29,824::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`d7cd440e-3539-4849-b534-fd8b348beab5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:34:29,825::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:34:29,825::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:34:29,826::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`d7cd440e-3539-4849-b534-fd8b348beab5`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:34:29,826::task::495::TaskManager.Task::(_debug) Task 7eaeca28-0767-47dd-a86c-a13f5b764d12: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:34:29,827::task::495::TaskManager.Task::(_debug) Task 7eaeca28-0767-47dd-a86c-a13f5b764d12: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:34:29,827::task::495::TaskManager.Task::(_debug) Task 7eaeca28-0767-47dd-a86c-a13f5b764d12: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:34:29,843::task::495::TaskManager.Task::(_debug) Task 7eaeca28-0767-47dd-a86c-a13f5b764d12: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:34:29,844::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:34:29,844::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:34:29,845::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:34:29,845::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:34:29,846::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:34:29,846::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:34:29,847::task::495::TaskManager.Task::(_debug) Task 7eaeca28-0767-47dd-a86c-a13f5b764d12: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:34:29,847::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:34:29,848::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:34:29,848::task::495::TaskManager.Task::(_debug) Task 68b3aab4-b5d1-4d97-b125-124a6e2ebdfc: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:34:29,849::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`ccc4a9b8-bdf9-482b-b119-528608e6a1a3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:34:29,849::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:34:29,850::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:34:29,850::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`ccc4a9b8-bdf9-482b-b119-528608e6a1a3`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:34:29,851::task::495::TaskManager.Task::(_debug) Task 68b3aab4-b5d1-4d97-b125-124a6e2ebdfc: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:34:29,851::task::495::TaskManager.Task::(_debug) Task 68b3aab4-b5d1-4d97-b125-124a6e2ebdfc: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:34:29,852::task::495::TaskManager.Task::(_debug) Task 68b3aab4-b5d1-4d97-b125-124a6e2ebdfc: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:34:29,852::task::495::TaskManager.Task::(_debug) Task 68b3aab4-b5d1-4d97-b125-124a6e2ebdfc: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:34:29,853::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:34:29,853::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:34:29,853::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:34:29,854::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:34:29,854::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:34:29,855::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:34:29,855::task::495::TaskManager.Task::(_debug) Task 68b3aab4-b5d1-4d97-b125-124a6e2ebdfc: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:34:29,855::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15914566::DEBUG::2013-11-20 03:34:31,094::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914566::INFO::2013-11-20 03:34:31,095::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914566::DEBUG::2013-11-20 03:34:31,095::task::495::TaskManager.Task::(_debug) Task 80019b38-0d7e-490b-bba6-c16827362141: moving from state init -> state preparing >Thread-15914566::DEBUG::2013-11-20 03:34:31,096::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914566::DEBUG::2013-11-20 03:34:31,096::task::495::TaskManager.Task::(_debug) Task 80019b38-0d7e-490b-bba6-c16827362141: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914566::DEBUG::2013-11-20 03:34:31,097::task::495::TaskManager.Task::(_debug) Task 80019b38-0d7e-490b-bba6-c16827362141: moving from state preparing -> state finished >Thread-15914566::DEBUG::2013-11-20 03:34:31,097::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914566::DEBUG::2013-11-20 03:34:31,098::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914566::DEBUG::2013-11-20 03:34:31,098::task::495::TaskManager.Task::(_debug) Task 80019b38-0d7e-490b-bba6-c16827362141: ref 0 aborting False >Thread-15914566::INFO::2013-11-20 03:34:31,099::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914567::DEBUG::2013-11-20 03:34:31,113::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914567::INFO::2013-11-20 03:34:31,114::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914567::DEBUG::2013-11-20 03:34:31,115::task::495::TaskManager.Task::(_debug) Task 7c7f5673-40da-47c1-aeca-011d0f611e1f: moving from state init -> state preparing >Thread-15914567::DEBUG::2013-11-20 03:34:31,116::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`48272940-6dba-4bd5-915a-2d9c860b0c0a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914567::DEBUG::2013-11-20 03:34:31,116::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914567::DEBUG::2013-11-20 03:34:31,117::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914567::DEBUG::2013-11-20 03:34:31,117::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`48272940-6dba-4bd5-915a-2d9c860b0c0a`::Granted request >Thread-15914567::DEBUG::2013-11-20 03:34:31,118::task::495::TaskManager.Task::(_debug) Task 7c7f5673-40da-47c1-aeca-011d0f611e1f: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914567::DEBUG::2013-11-20 03:34:31,119::task::495::TaskManager.Task::(_debug) Task 7c7f5673-40da-47c1-aeca-011d0f611e1f: ref 1 aborting False >Thread-15914567::WARNING::2013-11-20 03:34:31,121::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914567::DEBUG::2013-11-20 03:34:31,122::task::495::TaskManager.Task::(_debug) Task 7c7f5673-40da-47c1-aeca-011d0f611e1f: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914567::DEBUG::2013-11-20 03:34:31,123::task::495::TaskManager.Task::(_debug) Task 7c7f5673-40da-47c1-aeca-011d0f611e1f: moving from state preparing -> state finished >Thread-15914567::DEBUG::2013-11-20 03:34:31,124::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914567::DEBUG::2013-11-20 03:34:31,124::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914567::DEBUG::2013-11-20 03:34:31,125::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914567::DEBUG::2013-11-20 03:34:31,126::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914567::DEBUG::2013-11-20 03:34:31,126::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914567::DEBUG::2013-11-20 03:34:31,127::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914567::DEBUG::2013-11-20 03:34:31,127::task::495::TaskManager.Task::(_debug) Task 7c7f5673-40da-47c1-aeca-011d0f611e1f: ref 0 aborting False >Thread-15914567::INFO::2013-11-20 03:34:31,128::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:34:31,314::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0568641 s, 18.0 MB/s\n'; <rc> = 0 >Thread-42840::DEBUG::2013-11-20 03:34:32,045::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 772472cb-2429-466d-96cd-00b3301ef78a' (cwd None) >Thread-42840::DEBUG::2013-11-20 03:34:32,637::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:34:33,325::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:34:33,730::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.066942 s, 15.3 MB/s\n'; <rc> = 0 >libvirtEventLoop::DEBUG::2013-11-20 03:34:34,728::vm::525::vm.Vm::(_rtcUpdate) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::new rtc offset -1 >libvirtEventLoop::DEBUG::2013-11-20 03:34:34,729::vm::525::vm.Vm::(_rtcUpdate) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::new rtc offset -1 >libvirtEventLoop::DEBUG::2013-11-20 03:34:34,730::vm::525::vm.Vm::(_rtcUpdate) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::new rtc offset 3599 >libvirtEventLoop::DEBUG::2013-11-20 03:34:34,730::vm::525::vm.Vm::(_rtcUpdate) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::new rtc offset 3599 >libvirtEventLoop::DEBUG::2013-11-20 03:34:34,731::vm::525::vm.Vm::(_rtcUpdate) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::new rtc offset 3599 >libvirtEventLoop::DEBUG::2013-11-20 03:34:34,732::vm::525::vm.Vm::(_rtcUpdate) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::new rtc offset 3599 >libvirtEventLoop::DEBUG::2013-11-20 03:34:34,732::vm::525::vm.Vm::(_rtcUpdate) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::new rtc offset 3599 >libvirtEventLoop::DEBUG::2013-11-20 03:34:34,736::vm::525::vm.Vm::(_rtcUpdate) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::new rtc offset 3599 >libvirtEventLoop::DEBUG::2013-11-20 03:34:34,737::vm::525::vm.Vm::(_rtcUpdate) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::new rtc offset 3599 >libvirtEventLoop::DEBUG::2013-11-20 03:34:34,738::vm::525::vm.Vm::(_rtcUpdate) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::new rtc offset 3599 >libvirtEventLoop::DEBUG::2013-11-20 03:34:34,739::vm::525::vm.Vm::(_rtcUpdate) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::new rtc offset 3599 >libvirtEventLoop::DEBUG::2013-11-20 03:34:34,739::vm::525::vm.Vm::(_rtcUpdate) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::new rtc offset 3599 >libvirtEventLoop::DEBUG::2013-11-20 03:34:34,740::vm::525::vm.Vm::(_rtcUpdate) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::new rtc offset 3599 >libvirtEventLoop::DEBUG::2013-11-20 03:34:34,741::vm::525::vm.Vm::(_rtcUpdate) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::new rtc offset 3599 >Dummy-15912651::DEBUG::2013-11-20 03:34:35,740::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:34:36,234::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0436533 s, 23.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:34:38,243::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:34:38,625::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0509532 s, 20.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:34:40,635::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914571::DEBUG::2013-11-20 03:34:41,603::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914571::INFO::2013-11-20 03:34:41,605::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914571::DEBUG::2013-11-20 03:34:41,606::task::495::TaskManager.Task::(_debug) Task c0fdd9f3-a07d-454c-8ec0-7bf6b97e63ec: moving from state init -> state preparing >Thread-15914571::DEBUG::2013-11-20 03:34:41,607::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914571::DEBUG::2013-11-20 03:34:41,607::task::495::TaskManager.Task::(_debug) Task c0fdd9f3-a07d-454c-8ec0-7bf6b97e63ec: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914571::DEBUG::2013-11-20 03:34:41,608::task::495::TaskManager.Task::(_debug) Task c0fdd9f3-a07d-454c-8ec0-7bf6b97e63ec: moving from state preparing -> state finished >Thread-15914571::DEBUG::2013-11-20 03:34:41,608::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914571::DEBUG::2013-11-20 03:34:41,609::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914571::DEBUG::2013-11-20 03:34:41,609::task::495::TaskManager.Task::(_debug) Task c0fdd9f3-a07d-454c-8ec0-7bf6b97e63ec: ref 0 aborting False >Thread-15914571::INFO::2013-11-20 03:34:41,609::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914572::DEBUG::2013-11-20 03:34:41,624::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914572::INFO::2013-11-20 03:34:41,625::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914572::DEBUG::2013-11-20 03:34:41,625::task::495::TaskManager.Task::(_debug) Task 92331752-9e28-4693-b91d-013829ff2ee7: moving from state init -> state preparing >Thread-15914572::DEBUG::2013-11-20 03:34:41,627::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`071e312d-3c78-4136-8ad4-9f3a91246f71`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914572::DEBUG::2013-11-20 03:34:41,627::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914572::DEBUG::2013-11-20 03:34:41,628::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914572::DEBUG::2013-11-20 03:34:41,628::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`071e312d-3c78-4136-8ad4-9f3a91246f71`::Granted request >Thread-15914572::DEBUG::2013-11-20 03:34:41,629::task::495::TaskManager.Task::(_debug) Task 92331752-9e28-4693-b91d-013829ff2ee7: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914572::DEBUG::2013-11-20 03:34:41,630::task::495::TaskManager.Task::(_debug) Task 92331752-9e28-4693-b91d-013829ff2ee7: ref 1 aborting False >Thread-15914572::WARNING::2013-11-20 03:34:41,632::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914572::DEBUG::2013-11-20 03:34:41,633::task::495::TaskManager.Task::(_debug) Task 92331752-9e28-4693-b91d-013829ff2ee7: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914572::DEBUG::2013-11-20 03:34:41,634::task::495::TaskManager.Task::(_debug) Task 92331752-9e28-4693-b91d-013829ff2ee7: moving from state preparing -> state finished >Thread-15914572::DEBUG::2013-11-20 03:34:41,634::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914572::DEBUG::2013-11-20 03:34:41,635::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914572::DEBUG::2013-11-20 03:34:41,638::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914572::DEBUG::2013-11-20 03:34:41,638::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914572::DEBUG::2013-11-20 03:34:41,639::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914572::DEBUG::2013-11-20 03:34:41,639::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914572::DEBUG::2013-11-20 03:34:41,640::task::495::TaskManager.Task::(_debug) Task 92331752-9e28-4693-b91d-013829ff2ee7: ref 0 aborting False >Thread-15914572::INFO::2013-11-20 03:34:41,641::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:34:42,246::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0490081 s, 20.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:34:44,258::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913639::INFO::2013-11-20 03:34:45,056::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:34:45,058::task::495::TaskManager.Task::(_debug) Task 186dca3c-9d07-49cc-8bff-66ee8da00d07: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:34:45,060::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`726773b1-a620-4c42-846b-9f03ea54a936`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:34:45,060::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:34:45,061::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:34:45,062::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`726773b1-a620-4c42-846b-9f03ea54a936`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:34:45,062::task::495::TaskManager.Task::(_debug) Task 186dca3c-9d07-49cc-8bff-66ee8da00d07: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:34:45,063::task::495::TaskManager.Task::(_debug) Task 186dca3c-9d07-49cc-8bff-66ee8da00d07: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:34:45,064::task::495::TaskManager.Task::(_debug) Task 186dca3c-9d07-49cc-8bff-66ee8da00d07: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:34:45,064::task::495::TaskManager.Task::(_debug) Task 186dca3c-9d07-49cc-8bff-66ee8da00d07: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:34:45,064::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:34:45,065::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:34:45,065::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:34:45,066::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:34:45,066::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:34:45,066::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:34:45,067::task::495::TaskManager.Task::(_debug) Task 186dca3c-9d07-49cc-8bff-66ee8da00d07: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:34:45,067::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15914574::INFO::2013-11-20 03:34:45,614::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914574::DEBUG::2013-11-20 03:34:45,615::task::495::TaskManager.Task::(_debug) Task ac88b49f-7dad-4a73-b432-d0b081fdf3a3: moving from state init -> state preparing >Thread-15914574::DEBUG::2013-11-20 03:34:45,616::task::495::TaskManager.Task::(_debug) Task ac88b49f-7dad-4a73-b432-d0b081fdf3a3: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000227928161621', 'lastCheck': 1384918477.480813, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00110983848572', 'lastCheck': 1384918483.493912, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000996112823486', 'lastCheck': 1384918477.0526969, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00031590461731', 'lastCheck': 1384918482.8180659, 'valid': True, 'code': 0}} >Thread-15914574::DEBUG::2013-11-20 03:34:45,617::task::495::TaskManager.Task::(_debug) Task ac88b49f-7dad-4a73-b432-d0b081fdf3a3: moving from state preparing -> state finished >Thread-15914574::DEBUG::2013-11-20 03:34:45,618::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914574::DEBUG::2013-11-20 03:34:45,618::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914574::DEBUG::2013-11-20 03:34:45,619::task::495::TaskManager.Task::(_debug) Task ac88b49f-7dad-4a73-b432-d0b081fdf3a3: ref 0 aborting False >Thread-15914574::INFO::2013-11-20 03:34:45,620::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000227928161621', 'lastCheck': 1384918477.480813, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00110983848572', 'lastCheck': 1384918483.493912, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000996112823486', 'lastCheck': 1384918477.0526969, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00031590461731', 'lastCheck': 1384918482.8180659, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:34:45,677::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0626231 s, 16.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:34:47,695::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:34:48,124::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0485093 s, 21.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:34:50,134::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:34:50,597::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0482564 s, 21.2 MB/s\n'; <rc> = 0 >Thread-15914578::DEBUG::2013-11-20 03:34:52,243::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914578::INFO::2013-11-20 03:34:52,245::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914578::DEBUG::2013-11-20 03:34:52,246::task::495::TaskManager.Task::(_debug) Task fbdff6fc-5526-4c10-b88f-33475eb021ae: moving from state init -> state preparing >Thread-15914578::DEBUG::2013-11-20 03:34:52,248::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914578::DEBUG::2013-11-20 03:34:52,249::task::495::TaskManager.Task::(_debug) Task fbdff6fc-5526-4c10-b88f-33475eb021ae: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914578::DEBUG::2013-11-20 03:34:52,249::task::495::TaskManager.Task::(_debug) Task fbdff6fc-5526-4c10-b88f-33475eb021ae: moving from state preparing -> state finished >Thread-15914578::DEBUG::2013-11-20 03:34:52,250::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914578::DEBUG::2013-11-20 03:34:52,250::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914578::DEBUG::2013-11-20 03:34:52,251::task::495::TaskManager.Task::(_debug) Task fbdff6fc-5526-4c10-b88f-33475eb021ae: ref 0 aborting False >Thread-15914578::INFO::2013-11-20 03:34:52,251::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914579::DEBUG::2013-11-20 03:34:52,268::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914579::INFO::2013-11-20 03:34:52,269::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914579::DEBUG::2013-11-20 03:34:52,269::task::495::TaskManager.Task::(_debug) Task 335a05e6-b633-46d2-babc-2fc336b4ed3f: moving from state init -> state preparing >Thread-15914579::DEBUG::2013-11-20 03:34:52,270::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e19a7ea0-1bc9-46e8-a915-edb7eb2f8c79`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914579::DEBUG::2013-11-20 03:34:52,271::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914579::DEBUG::2013-11-20 03:34:52,271::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914579::DEBUG::2013-11-20 03:34:52,277::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`e19a7ea0-1bc9-46e8-a915-edb7eb2f8c79`::Granted request >Thread-15914579::DEBUG::2013-11-20 03:34:52,279::task::495::TaskManager.Task::(_debug) Task 335a05e6-b633-46d2-babc-2fc336b4ed3f: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914579::DEBUG::2013-11-20 03:34:52,279::task::495::TaskManager.Task::(_debug) Task 335a05e6-b633-46d2-babc-2fc336b4ed3f: ref 1 aborting False >Thread-15914579::WARNING::2013-11-20 03:34:52,281::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914579::DEBUG::2013-11-20 03:34:52,281::task::495::TaskManager.Task::(_debug) Task 335a05e6-b633-46d2-babc-2fc336b4ed3f: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914579::DEBUG::2013-11-20 03:34:52,282::task::495::TaskManager.Task::(_debug) Task 335a05e6-b633-46d2-babc-2fc336b4ed3f: moving from state preparing -> state finished >Thread-15914579::DEBUG::2013-11-20 03:34:52,282::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914579::DEBUG::2013-11-20 03:34:52,283::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914579::DEBUG::2013-11-20 03:34:52,283::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914579::DEBUG::2013-11-20 03:34:52,284::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914579::DEBUG::2013-11-20 03:34:52,284::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914579::DEBUG::2013-11-20 03:34:52,284::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914579::DEBUG::2013-11-20 03:34:52,285::task::495::TaskManager.Task::(_debug) Task 335a05e6-b633-46d2-babc-2fc336b4ed3f: ref 0 aborting False >Thread-15914579::INFO::2013-11-20 03:34:52,285::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:34:52,610::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:34:53,000::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0534901 s, 19.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:34:55,010::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:34:55,481::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0458152 s, 22.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:34:57,491::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:34:57,903::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0485026 s, 21.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:34:59,912::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913658::INFO::2013-11-20 03:35:00,091::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:35:00,092::task::495::TaskManager.Task::(_debug) Task 67a94347-7254-4f4b-ba7b-96bebe64037e: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:35:00,094::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0ea9ae83-9571-4bc3-85e3-abf9ddc65e21`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:35:00,095::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:35:00,096::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:35:00,096::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0ea9ae83-9571-4bc3-85e3-abf9ddc65e21`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:35:00,097::task::495::TaskManager.Task::(_debug) Task 67a94347-7254-4f4b-ba7b-96bebe64037e: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:35:00,098::task::495::TaskManager.Task::(_debug) Task 67a94347-7254-4f4b-ba7b-96bebe64037e: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:35:00,098::task::495::TaskManager.Task::(_debug) Task 67a94347-7254-4f4b-ba7b-96bebe64037e: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:35:00,099::task::495::TaskManager.Task::(_debug) Task 67a94347-7254-4f4b-ba7b-96bebe64037e: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:35:00,099::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:35:00,100::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:35:00,100::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:35:00,101::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:35:00,101::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:35:00,102::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:35:00,102::task::495::TaskManager.Task::(_debug) Task 67a94347-7254-4f4b-ba7b-96bebe64037e: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:35:00,103::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:35:00,964::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0535012 s, 19.1 MB/s\n'; <rc> = 0 >Thread-15914582::INFO::2013-11-20 03:35:01,360::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914582::DEBUG::2013-11-20 03:35:01,361::task::495::TaskManager.Task::(_debug) Task 8466d027-362b-4134-9e16-6f6f9224608d: moving from state init -> state preparing >Thread-15914582::DEBUG::2013-11-20 03:35:01,362::task::495::TaskManager.Task::(_debug) Task 8466d027-362b-4134-9e16-6f6f9224608d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000375986099243', 'lastCheck': 1384918497.6561351, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00091814994812', 'lastCheck': 1384918493.5062599, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0110697746277', 'lastCheck': 1384918497.2010829, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000396013259888', 'lastCheck': 1384918492.882699, 'valid': True, 'code': 0}} >Thread-15914582::DEBUG::2013-11-20 03:35:01,363::task::495::TaskManager.Task::(_debug) Task 8466d027-362b-4134-9e16-6f6f9224608d: moving from state preparing -> state finished >Thread-15914582::DEBUG::2013-11-20 03:35:01,363::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914582::DEBUG::2013-11-20 03:35:01,363::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914582::DEBUG::2013-11-20 03:35:01,364::task::495::TaskManager.Task::(_debug) Task 8466d027-362b-4134-9e16-6f6f9224608d: ref 0 aborting False >Thread-15914582::INFO::2013-11-20 03:35:01,364::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000375986099243', 'lastCheck': 1384918497.6561351, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00091814994812', 'lastCheck': 1384918493.5062599, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.0110697746277', 'lastCheck': 1384918497.2010829, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000396013259888', 'lastCheck': 1384918492.882699, 'valid': True, 'code': 0}} >Thread-15914584::DEBUG::2013-11-20 03:35:02,571::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914584::INFO::2013-11-20 03:35:02,572::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914584::DEBUG::2013-11-20 03:35:02,573::task::495::TaskManager.Task::(_debug) Task 0f6f608b-304b-423b-8e09-c9384bd615ca: moving from state init -> state preparing >Thread-15914584::DEBUG::2013-11-20 03:35:02,574::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914584::DEBUG::2013-11-20 03:35:02,574::task::495::TaskManager.Task::(_debug) Task 0f6f608b-304b-423b-8e09-c9384bd615ca: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914584::DEBUG::2013-11-20 03:35:02,574::task::495::TaskManager.Task::(_debug) Task 0f6f608b-304b-423b-8e09-c9384bd615ca: moving from state preparing -> state finished >Thread-15914584::DEBUG::2013-11-20 03:35:02,575::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914584::DEBUG::2013-11-20 03:35:02,575::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914584::DEBUG::2013-11-20 03:35:02,576::task::495::TaskManager.Task::(_debug) Task 0f6f608b-304b-423b-8e09-c9384bd615ca: ref 0 aborting False >Thread-15914584::INFO::2013-11-20 03:35:02,576::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914585::DEBUG::2013-11-20 03:35:02,587::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914585::INFO::2013-11-20 03:35:02,588::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914585::DEBUG::2013-11-20 03:35:02,588::task::495::TaskManager.Task::(_debug) Task 183b0390-7fdd-4510-bf6a-d232599ed96d: moving from state init -> state preparing >Thread-15914585::DEBUG::2013-11-20 03:35:02,589::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b3f30a76-89de-469b-b244-9f98bff3892d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914585::DEBUG::2013-11-20 03:35:02,590::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914585::DEBUG::2013-11-20 03:35:02,590::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914585::DEBUG::2013-11-20 03:35:02,590::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`b3f30a76-89de-469b-b244-9f98bff3892d`::Granted request >Thread-15914585::DEBUG::2013-11-20 03:35:02,591::task::495::TaskManager.Task::(_debug) Task 183b0390-7fdd-4510-bf6a-d232599ed96d: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914585::DEBUG::2013-11-20 03:35:02,591::task::495::TaskManager.Task::(_debug) Task 183b0390-7fdd-4510-bf6a-d232599ed96d: ref 1 aborting False >Thread-15914585::WARNING::2013-11-20 03:35:02,593::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914585::DEBUG::2013-11-20 03:35:02,593::task::495::TaskManager.Task::(_debug) Task 183b0390-7fdd-4510-bf6a-d232599ed96d: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914585::DEBUG::2013-11-20 03:35:02,594::task::495::TaskManager.Task::(_debug) Task 183b0390-7fdd-4510-bf6a-d232599ed96d: moving from state preparing -> state finished >Thread-15914585::DEBUG::2013-11-20 03:35:02,594::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914585::DEBUG::2013-11-20 03:35:02,595::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914585::DEBUG::2013-11-20 03:35:02,595::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914585::DEBUG::2013-11-20 03:35:02,596::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914585::DEBUG::2013-11-20 03:35:02,596::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914585::DEBUG::2013-11-20 03:35:02,596::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914585::DEBUG::2013-11-20 03:35:02,597::task::495::TaskManager.Task::(_debug) Task 183b0390-7fdd-4510-bf6a-d232599ed96d: ref 0 aborting False >Thread-15914585::INFO::2013-11-20 03:35:02,597::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:35:02,975::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:35:03,383::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0506343 s, 20.2 MB/s\n'; <rc> = 0 >Thread-15912392::INFO::2013-11-20 03:35:05,223::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:35:05,224::task::495::TaskManager.Task::(_debug) Task 82c01ac7-9e5c-4e10-b620-8a324b111ddb: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:35:05,225::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`99937e20-1c79-4ed2-b124-1a206faaebc0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:35:05,225::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:35:05,226::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:35:05,226::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`99937e20-1c79-4ed2-b124-1a206faaebc0`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:35:05,227::task::495::TaskManager.Task::(_debug) Task 82c01ac7-9e5c-4e10-b620-8a324b111ddb: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:35:05,227::task::495::TaskManager.Task::(_debug) Task 82c01ac7-9e5c-4e10-b620-8a324b111ddb: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:35:05,228::task::495::TaskManager.Task::(_debug) Task 82c01ac7-9e5c-4e10-b620-8a324b111ddb: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:35:05,229::task::495::TaskManager.Task::(_debug) Task 82c01ac7-9e5c-4e10-b620-8a324b111ddb: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:35:05,229::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:35:05,230::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:35:05,231::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:35:05,231::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:35:05,232::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:35:05,232::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:35:05,232::task::495::TaskManager.Task::(_debug) Task 82c01ac7-9e5c-4e10-b620-8a324b111ddb: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:35:05,233::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:35:05,393::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:35:05,821::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0549609 s, 18.6 MB/s\n'; <rc> = 0 >Thread-15913695::INFO::2013-11-20 03:35:07,340::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:35:07,340::task::495::TaskManager.Task::(_debug) Task c8b4202f-52ba-4dfe-a3ef-73f38075595d: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:35:07,341::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0b2e28c0-48b3-4d7a-a6ea-115120671525`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:35:07,342::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:35:07,342::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:35:07,343::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0b2e28c0-48b3-4d7a-a6ea-115120671525`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:35:07,343::task::495::TaskManager.Task::(_debug) Task c8b4202f-52ba-4dfe-a3ef-73f38075595d: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:35:07,344::task::495::TaskManager.Task::(_debug) Task c8b4202f-52ba-4dfe-a3ef-73f38075595d: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:35:07,345::task::495::TaskManager.Task::(_debug) Task c8b4202f-52ba-4dfe-a3ef-73f38075595d: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:35:07,345::task::495::TaskManager.Task::(_debug) Task c8b4202f-52ba-4dfe-a3ef-73f38075595d: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:35:07,345::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:35:07,346::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:35:07,346::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:35:07,347::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:35:07,347::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:35:07,347::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:35:07,348::task::495::TaskManager.Task::(_debug) Task c8b4202f-52ba-4dfe-a3ef-73f38075595d: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:35:07,348::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:35:07,832::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:35:08,275::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0530487 s, 19.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:35:10,289::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:35:10,679::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0532089 s, 19.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:35:12,690::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914589::DEBUG::2013-11-20 03:35:13,119::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914589::INFO::2013-11-20 03:35:13,121::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914589::DEBUG::2013-11-20 03:35:13,121::task::495::TaskManager.Task::(_debug) Task 8277621f-e694-4035-8275-8824e4be9bb7: moving from state init -> state preparing >Thread-15914589::DEBUG::2013-11-20 03:35:13,122::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914589::DEBUG::2013-11-20 03:35:13,123::task::495::TaskManager.Task::(_debug) Task 8277621f-e694-4035-8275-8824e4be9bb7: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914589::DEBUG::2013-11-20 03:35:13,123::task::495::TaskManager.Task::(_debug) Task 8277621f-e694-4035-8275-8824e4be9bb7: moving from state preparing -> state finished >Thread-15914589::DEBUG::2013-11-20 03:35:13,124::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914589::DEBUG::2013-11-20 03:35:13,124::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914589::DEBUG::2013-11-20 03:35:13,125::task::495::TaskManager.Task::(_debug) Task 8277621f-e694-4035-8275-8824e4be9bb7: ref 0 aborting False >Thread-15914589::INFO::2013-11-20 03:35:13,125::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914590::DEBUG::2013-11-20 03:35:13,134::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914590::INFO::2013-11-20 03:35:13,135::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914590::DEBUG::2013-11-20 03:35:13,135::task::495::TaskManager.Task::(_debug) Task 9c5b23b1-b83b-4ea6-b4ed-cd5a7bb180a6: moving from state init -> state preparing >Thread-15914590::DEBUG::2013-11-20 03:35:13,136::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1e0276d1-ab40-4363-8a57-a433010e6497`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914590::DEBUG::2013-11-20 03:35:13,137::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914590::DEBUG::2013-11-20 03:35:13,137::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914590::DEBUG::2013-11-20 03:35:13,138::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1e0276d1-ab40-4363-8a57-a433010e6497`::Granted request >Thread-15914590::DEBUG::2013-11-20 03:35:13,141::task::495::TaskManager.Task::(_debug) Task 9c5b23b1-b83b-4ea6-b4ed-cd5a7bb180a6: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914590::DEBUG::2013-11-20 03:35:13,147::task::495::TaskManager.Task::(_debug) Task 9c5b23b1-b83b-4ea6-b4ed-cd5a7bb180a6: ref 1 aborting False >Thread-15914590::WARNING::2013-11-20 03:35:13,149::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914590::DEBUG::2013-11-20 03:35:13,153::task::495::TaskManager.Task::(_debug) Task 9c5b23b1-b83b-4ea6-b4ed-cd5a7bb180a6: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914590::DEBUG::2013-11-20 03:35:13,154::task::495::TaskManager.Task::(_debug) Task 9c5b23b1-b83b-4ea6-b4ed-cd5a7bb180a6: moving from state preparing -> state finished >Thread-15914590::DEBUG::2013-11-20 03:35:13,155::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914590::DEBUG::2013-11-20 03:35:13,155::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914590::DEBUG::2013-11-20 03:35:13,155::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914590::DEBUG::2013-11-20 03:35:13,156::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914590::DEBUG::2013-11-20 03:35:13,156::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914590::DEBUG::2013-11-20 03:35:13,157::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914590::DEBUG::2013-11-20 03:35:13,157::task::495::TaskManager.Task::(_debug) Task 9c5b23b1-b83b-4ea6-b4ed-cd5a7bb180a6: ref 0 aborting False >Thread-15914590::INFO::2013-11-20 03:35:13,158::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:35:13,585::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0549145 s, 18.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:35:15,596::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:35:15,984::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0480675 s, 21.3 MB/s\n'; <rc> = 0 >Thread-15914592::INFO::2013-11-20 03:35:17,099::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914592::DEBUG::2013-11-20 03:35:17,100::task::495::TaskManager.Task::(_debug) Task 7423b35d-3370-4cbc-a9b3-035ef0dd4f99: moving from state init -> state preparing >Thread-15914592::DEBUG::2013-11-20 03:35:17,101::task::495::TaskManager.Task::(_debug) Task 7423b35d-3370-4cbc-a9b3-035ef0dd4f99: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000241041183472', 'lastCheck': 1384918507.6680329, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144720077515', 'lastCheck': 1384918513.695401, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00119590759277', 'lastCheck': 1384918507.214628, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000439882278442', 'lastCheck': 1384918513.0087731, 'valid': True, 'code': 0}} >Thread-15914592::DEBUG::2013-11-20 03:35:17,102::task::495::TaskManager.Task::(_debug) Task 7423b35d-3370-4cbc-a9b3-035ef0dd4f99: moving from state preparing -> state finished >Thread-15914592::DEBUG::2013-11-20 03:35:17,102::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914592::DEBUG::2013-11-20 03:35:17,103::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914592::DEBUG::2013-11-20 03:35:17,103::task::495::TaskManager.Task::(_debug) Task 7423b35d-3370-4cbc-a9b3-035ef0dd4f99: ref 0 aborting False >Thread-15914592::INFO::2013-11-20 03:35:17,104::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000241041183472', 'lastCheck': 1384918507.6680329, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00144720077515', 'lastCheck': 1384918513.695401, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00119590759277', 'lastCheck': 1384918507.214628, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000439882278442', 'lastCheck': 1384918513.0087731, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:35:17,994::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914435::INFO::2013-11-20 03:35:18,216::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=143cb26d-de08-4d30-89cf-aee10245b66a volUUID=d7028f7c-0551-4bbe-a907-6f804cc9bd54) >Thread-15914435::DEBUG::2013-11-20 03:35:18,217::task::495::TaskManager.Task::(_debug) Task ac155344-461c-4b36-ad5b-026a4d9c8a65: moving from state init -> state preparing >Thread-15914435::DEBUG::2013-11-20 03:35:18,220::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a2bf5066-4211-4d9d-a624-6f3c2e855a27`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914435::DEBUG::2013-11-20 03:35:18,221::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914435::DEBUG::2013-11-20 03:35:18,222::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914435::DEBUG::2013-11-20 03:35:18,222::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`a2bf5066-4211-4d9d-a624-6f3c2e855a27`::Granted request >Thread-15914435::DEBUG::2013-11-20 03:35:18,224::task::495::TaskManager.Task::(_debug) Task ac155344-461c-4b36-ad5b-026a4d9c8a65: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914435::DEBUG::2013-11-20 03:35:18,224::task::495::TaskManager.Task::(_debug) Task ac155344-461c-4b36-ad5b-026a4d9c8a65: ref 1 aborting False >Thread-15914435::DEBUG::2013-11-20 03:35:18,225::task::495::TaskManager.Task::(_debug) Task ac155344-461c-4b36-ad5b-026a4d9c8a65: finished: {'truesize': '56908316672', 'apparentsize': '56908316672'} >Thread-15914435::DEBUG::2013-11-20 03:35:18,226::task::495::TaskManager.Task::(_debug) Task ac155344-461c-4b36-ad5b-026a4d9c8a65: moving from state preparing -> state finished >Thread-15914435::DEBUG::2013-11-20 03:35:18,226::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914435::DEBUG::2013-11-20 03:35:18,227::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914435::DEBUG::2013-11-20 03:35:18,227::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914435::DEBUG::2013-11-20 03:35:18,228::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914435::DEBUG::2013-11-20 03:35:18,228::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914435::DEBUG::2013-11-20 03:35:18,229::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914435::DEBUG::2013-11-20 03:35:18,229::task::495::TaskManager.Task::(_debug) Task ac155344-461c-4b36-ad5b-026a4d9c8a65: ref 0 aborting False >Thread-15914435::INFO::2013-11-20 03:35:18,230::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '56908316672', 'apparentsize': '56908316672'} >Dummy-15912651::DEBUG::2013-11-20 03:35:18,407::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0522994 s, 19.6 MB/s\n'; <rc> = 0 >Thread-15914453::INFO::2013-11-20 03:35:20,163::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=5f930ef3-763f-4294-a62a-6d52df220f8c volUUID=b9cffdbe-152e-41ce-ae5d-08db7c08454e) >Thread-15914453::DEBUG::2013-11-20 03:35:20,165::task::495::TaskManager.Task::(_debug) Task bb653145-2c2d-4991-ad80-7f336ad46308: moving from state init -> state preparing >Thread-15914453::DEBUG::2013-11-20 03:35:20,166::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`5f6634a7-9dc6-444a-b080-7709279e876e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914453::DEBUG::2013-11-20 03:35:20,167::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914453::DEBUG::2013-11-20 03:35:20,167::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914453::DEBUG::2013-11-20 03:35:20,168::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`5f6634a7-9dc6-444a-b080-7709279e876e`::Granted request >Thread-15914453::DEBUG::2013-11-20 03:35:20,168::task::495::TaskManager.Task::(_debug) Task bb653145-2c2d-4991-ad80-7f336ad46308: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914453::DEBUG::2013-11-20 03:35:20,169::task::495::TaskManager.Task::(_debug) Task bb653145-2c2d-4991-ad80-7f336ad46308: ref 1 aborting False >Thread-15914453::DEBUG::2013-11-20 03:35:20,169::task::495::TaskManager.Task::(_debug) Task bb653145-2c2d-4991-ad80-7f336ad46308: finished: {'truesize': '53821308928', 'apparentsize': '53821308928'} >Thread-15914453::DEBUG::2013-11-20 03:35:20,170::task::495::TaskManager.Task::(_debug) Task bb653145-2c2d-4991-ad80-7f336ad46308: moving from state preparing -> state finished >Thread-15914453::DEBUG::2013-11-20 03:35:20,170::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914453::DEBUG::2013-11-20 03:35:20,171::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914453::DEBUG::2013-11-20 03:35:20,171::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914453::DEBUG::2013-11-20 03:35:20,172::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914453::DEBUG::2013-11-20 03:35:20,172::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914453::DEBUG::2013-11-20 03:35:20,172::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914453::DEBUG::2013-11-20 03:35:20,173::task::495::TaskManager.Task::(_debug) Task bb653145-2c2d-4991-ad80-7f336ad46308: ref 0 aborting False >Thread-15914453::INFO::2013-11-20 03:35:20,173::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '53821308928', 'apparentsize': '53821308928'} >Dummy-15912651::DEBUG::2013-11-20 03:35:20,417::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:35:20,824::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0515338 s, 19.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:35:22,837::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914519::INFO::2013-11-20 03:35:23,305::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=5af747d8-95b1-4b7e-af53-cb77e4fcc896 volUUID=8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f) >Thread-15914519::DEBUG::2013-11-20 03:35:23,306::task::495::TaskManager.Task::(_debug) Task 73a62ad0-6b01-4642-ba00-0f278eadc612: moving from state init -> state preparing >Thread-15914519::DEBUG::2013-11-20 03:35:23,315::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`88ad181f-5643-49ab-b1d8-1dbe11a72f20`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914519::DEBUG::2013-11-20 03:35:23,316::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914519::DEBUG::2013-11-20 03:35:23,317::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914519::DEBUG::2013-11-20 03:35:23,318::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`88ad181f-5643-49ab-b1d8-1dbe11a72f20`::Granted request >Thread-15914519::DEBUG::2013-11-20 03:35:23,320::task::495::TaskManager.Task::(_debug) Task 73a62ad0-6b01-4642-ba00-0f278eadc612: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914519::DEBUG::2013-11-20 03:35:23,320::task::495::TaskManager.Task::(_debug) Task 73a62ad0-6b01-4642-ba00-0f278eadc612: ref 1 aborting False >Thread-15914519::DEBUG::2013-11-20 03:35:23,321::task::495::TaskManager.Task::(_debug) Task 73a62ad0-6b01-4642-ba00-0f278eadc612: finished: {'truesize': '86033563648', 'apparentsize': '86033563648'} >Thread-15914519::DEBUG::2013-11-20 03:35:23,322::task::495::TaskManager.Task::(_debug) Task 73a62ad0-6b01-4642-ba00-0f278eadc612: moving from state preparing -> state finished >Thread-15914519::DEBUG::2013-11-20 03:35:23,322::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914519::DEBUG::2013-11-20 03:35:23,323::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914519::DEBUG::2013-11-20 03:35:23,324::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914519::DEBUG::2013-11-20 03:35:23,325::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914519::DEBUG::2013-11-20 03:35:23,325::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914519::DEBUG::2013-11-20 03:35:23,326::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914519::DEBUG::2013-11-20 03:35:23,327::task::495::TaskManager.Task::(_debug) Task 73a62ad0-6b01-4642-ba00-0f278eadc612: ref 0 aborting False >Thread-15914519::INFO::2013-11-20 03:35:23,327::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '86033563648', 'apparentsize': '86033563648'} >Thread-15914595::DEBUG::2013-11-20 03:35:23,482::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914595::INFO::2013-11-20 03:35:23,483::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914595::DEBUG::2013-11-20 03:35:23,483::task::495::TaskManager.Task::(_debug) Task c37c5bf0-1af0-43e1-a0fb-4e27a32124e2: moving from state init -> state preparing >Thread-15914595::DEBUG::2013-11-20 03:35:23,484::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914595::DEBUG::2013-11-20 03:35:23,484::task::495::TaskManager.Task::(_debug) Task c37c5bf0-1af0-43e1-a0fb-4e27a32124e2: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914595::DEBUG::2013-11-20 03:35:23,485::task::495::TaskManager.Task::(_debug) Task c37c5bf0-1af0-43e1-a0fb-4e27a32124e2: moving from state preparing -> state finished >Thread-15914595::DEBUG::2013-11-20 03:35:23,485::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914595::DEBUG::2013-11-20 03:35:23,486::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914595::DEBUG::2013-11-20 03:35:23,486::task::495::TaskManager.Task::(_debug) Task c37c5bf0-1af0-43e1-a0fb-4e27a32124e2: ref 0 aborting False >Thread-15914595::INFO::2013-11-20 03:35:23,486::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914596::DEBUG::2013-11-20 03:35:23,500::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914596::INFO::2013-11-20 03:35:23,500::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914596::DEBUG::2013-11-20 03:35:23,501::task::495::TaskManager.Task::(_debug) Task 6c8b94b4-584e-47fe-b9dd-167e9deccec3: moving from state init -> state preparing >Thread-15914596::DEBUG::2013-11-20 03:35:23,502::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`df8783f1-43c2-4542-ab49-cedcb652e6a7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914596::DEBUG::2013-11-20 03:35:23,502::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914596::DEBUG::2013-11-20 03:35:23,503::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914596::DEBUG::2013-11-20 03:35:23,503::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`df8783f1-43c2-4542-ab49-cedcb652e6a7`::Granted request >Thread-15914596::DEBUG::2013-11-20 03:35:23,504::task::495::TaskManager.Task::(_debug) Task 6c8b94b4-584e-47fe-b9dd-167e9deccec3: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914596::DEBUG::2013-11-20 03:35:23,504::task::495::TaskManager.Task::(_debug) Task 6c8b94b4-584e-47fe-b9dd-167e9deccec3: ref 1 aborting False >Thread-15914596::WARNING::2013-11-20 03:35:23,506::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914596::DEBUG::2013-11-20 03:35:23,507::task::495::TaskManager.Task::(_debug) Task 6c8b94b4-584e-47fe-b9dd-167e9deccec3: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914596::DEBUG::2013-11-20 03:35:23,508::task::495::TaskManager.Task::(_debug) Task 6c8b94b4-584e-47fe-b9dd-167e9deccec3: moving from state preparing -> state finished >Thread-15914596::DEBUG::2013-11-20 03:35:23,508::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914596::DEBUG::2013-11-20 03:35:23,509::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914596::DEBUG::2013-11-20 03:35:23,509::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914596::DEBUG::2013-11-20 03:35:23,510::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914596::DEBUG::2013-11-20 03:35:23,510::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914596::DEBUG::2013-11-20 03:35:23,510::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914596::DEBUG::2013-11-20 03:35:23,511::task::495::TaskManager.Task::(_debug) Task 6c8b94b4-584e-47fe-b9dd-167e9deccec3: ref 0 aborting False >Thread-15914596::INFO::2013-11-20 03:35:23,511::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:35:25,999::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0535362 s, 19.1 MB/s\n'; <rc> = 0 >Thread-15913518::INFO::2013-11-20 03:35:27,865::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:35:27,866::task::495::TaskManager.Task::(_debug) Task e49658d7-8359-4817-96d1-73ffe4aaef65: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:35:27,867::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`214b908b-4458-441a-bbe9-d2be097e4892`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:35:27,867::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:35:27,868::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:35:27,868::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`214b908b-4458-441a-bbe9-d2be097e4892`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:35:27,869::task::495::TaskManager.Task::(_debug) Task e49658d7-8359-4817-96d1-73ffe4aaef65: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:35:27,869::task::495::TaskManager.Task::(_debug) Task e49658d7-8359-4817-96d1-73ffe4aaef65: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:35:27,869::task::495::TaskManager.Task::(_debug) Task e49658d7-8359-4817-96d1-73ffe4aaef65: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:35:27,870::task::495::TaskManager.Task::(_debug) Task e49658d7-8359-4817-96d1-73ffe4aaef65: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:35:27,870::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:35:27,870::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:35:27,871::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:35:27,871::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:35:27,872::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:35:27,872::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:35:27,872::task::495::TaskManager.Task::(_debug) Task e49658d7-8359-4817-96d1-73ffe4aaef65: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:35:27,873::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:35:27,873::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:35:27,873::task::495::TaskManager.Task::(_debug) Task b11ae253-1f5c-44f5-bff3-6b540febafbe: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:35:27,874::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`fa5bb5eb-9fad-4dd3-9801-d7b2833b21b2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:35:27,875::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:35:27,875::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:35:27,875::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`fa5bb5eb-9fad-4dd3-9801-d7b2833b21b2`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:35:27,876::task::495::TaskManager.Task::(_debug) Task b11ae253-1f5c-44f5-bff3-6b540febafbe: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:35:27,876::task::495::TaskManager.Task::(_debug) Task b11ae253-1f5c-44f5-bff3-6b540febafbe: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:35:27,877::task::495::TaskManager.Task::(_debug) Task b11ae253-1f5c-44f5-bff3-6b540febafbe: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:35:27,877::task::495::TaskManager.Task::(_debug) Task b11ae253-1f5c-44f5-bff3-6b540febafbe: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:35:27,877::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:35:27,878::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:35:27,878::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:35:27,879::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:35:27,879::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:35:27,879::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:35:27,880::task::495::TaskManager.Task::(_debug) Task b11ae253-1f5c-44f5-bff3-6b540febafbe: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:35:27,880::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:35:28,018::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:35:28,520::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0564533 s, 18.1 MB/s\n'; <rc> = 0 >Thread-15913546::INFO::2013-11-20 03:35:30,194::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:35:30,205::task::495::TaskManager.Task::(_debug) Task 73cbbfe6-e056-4125-a248-0209d53277b7: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:35:30,207::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`d80932e2-1ac5-435f-8ae0-d796112283f3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:35:30,208::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:35:30,209::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:35:30,209::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`d80932e2-1ac5-435f-8ae0-d796112283f3`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:35:30,210::task::495::TaskManager.Task::(_debug) Task 73cbbfe6-e056-4125-a248-0209d53277b7: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:35:30,210::task::495::TaskManager.Task::(_debug) Task 73cbbfe6-e056-4125-a248-0209d53277b7: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:35:30,211::task::495::TaskManager.Task::(_debug) Task 73cbbfe6-e056-4125-a248-0209d53277b7: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:35:30,211::task::495::TaskManager.Task::(_debug) Task 73cbbfe6-e056-4125-a248-0209d53277b7: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:35:30,212::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:35:30,212::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:35:30,213::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:35:30,213::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:35:30,214::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:35:30,214::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:35:30,214::task::495::TaskManager.Task::(_debug) Task 73cbbfe6-e056-4125-a248-0209d53277b7: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:35:30,215::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:35:30,215::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:35:30,216::task::495::TaskManager.Task::(_debug) Task f3ed7ca4-8270-42a3-a06b-09119f29a87e: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:35:30,217::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`665c40a2-0025-49de-9453-4e2011b0ab37`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:35:30,217::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:35:30,218::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:35:30,218::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`665c40a2-0025-49de-9453-4e2011b0ab37`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:35:30,219::task::495::TaskManager.Task::(_debug) Task f3ed7ca4-8270-42a3-a06b-09119f29a87e: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:35:30,219::task::495::TaskManager.Task::(_debug) Task f3ed7ca4-8270-42a3-a06b-09119f29a87e: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:35:30,219::task::495::TaskManager.Task::(_debug) Task f3ed7ca4-8270-42a3-a06b-09119f29a87e: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:35:30,220::task::495::TaskManager.Task::(_debug) Task f3ed7ca4-8270-42a3-a06b-09119f29a87e: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:35:30,220::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:35:30,221::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:35:30,221::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:35:30,222::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:35:30,222::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:35:30,222::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:35:30,223::task::495::TaskManager.Task::(_debug) Task f3ed7ca4-8270-42a3-a06b-09119f29a87e: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:35:30,223::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:35:30,224::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:35:30,224::task::495::TaskManager.Task::(_debug) Task 6d7b8049-260a-4ce8-b170-bb5625988f75: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:35:30,225::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`b6b6e807-8124-48c5-b88c-62fc067a2b29`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:35:30,225::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:35:30,226::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:35:30,226::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`b6b6e807-8124-48c5-b88c-62fc067a2b29`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:35:30,227::task::495::TaskManager.Task::(_debug) Task 6d7b8049-260a-4ce8-b170-bb5625988f75: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:35:30,227::task::495::TaskManager.Task::(_debug) Task 6d7b8049-260a-4ce8-b170-bb5625988f75: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:35:30,228::task::495::TaskManager.Task::(_debug) Task 6d7b8049-260a-4ce8-b170-bb5625988f75: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:35:30,228::task::495::TaskManager.Task::(_debug) Task 6d7b8049-260a-4ce8-b170-bb5625988f75: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:35:30,228::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:35:30,229::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:35:30,229::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:35:30,230::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:35:30,230::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:35:30,230::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:35:30,231::task::495::TaskManager.Task::(_debug) Task 6d7b8049-260a-4ce8-b170-bb5625988f75: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:35:30,231::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:35:30,232::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:35:30,232::task::495::TaskManager.Task::(_debug) Task fafe2867-e71e-42b2-9e9f-ff37a18cd54f: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:35:30,233::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`663fd657-9b6b-4546-a9f5-cfaa37e43368`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:35:30,233::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:35:30,234::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:35:30,234::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`663fd657-9b6b-4546-a9f5-cfaa37e43368`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:35:30,235::task::495::TaskManager.Task::(_debug) Task fafe2867-e71e-42b2-9e9f-ff37a18cd54f: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:35:30,235::task::495::TaskManager.Task::(_debug) Task fafe2867-e71e-42b2-9e9f-ff37a18cd54f: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:35:30,236::task::495::TaskManager.Task::(_debug) Task fafe2867-e71e-42b2-9e9f-ff37a18cd54f: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:35:30,236::task::495::TaskManager.Task::(_debug) Task fafe2867-e71e-42b2-9e9f-ff37a18cd54f: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:35:30,237::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:35:30,237::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:35:30,238::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:35:30,238::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:35:30,238::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:35:30,239::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:35:30,239::task::495::TaskManager.Task::(_debug) Task fafe2867-e71e-42b2-9e9f-ff37a18cd54f: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:35:30,240::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:35:30,530::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:35:30,912::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0478249 s, 21.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:35:32,923::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914600::INFO::2013-11-20 03:35:32,926::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914600::DEBUG::2013-11-20 03:35:33,077::task::495::TaskManager.Task::(_debug) Task ce73e285-1146-480c-9a73-e5fee1a339d0: moving from state init -> state preparing >Thread-15914600::DEBUG::2013-11-20 03:35:33,079::task::495::TaskManager.Task::(_debug) Task ce73e285-1146-480c-9a73-e5fee1a339d0: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000240087509155', 'lastCheck': 1384918527.773176, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0011260509491', 'lastCheck': 1384918523.7671471, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000925779342651', 'lastCheck': 1384918527.238373, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00043797492981', 'lastCheck': 1384918523.1650529, 'valid': True, 'code': 0}} >Thread-15914600::DEBUG::2013-11-20 03:35:33,080::task::495::TaskManager.Task::(_debug) Task ce73e285-1146-480c-9a73-e5fee1a339d0: moving from state preparing -> state finished >Thread-15914600::DEBUG::2013-11-20 03:35:33,080::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914600::DEBUG::2013-11-20 03:35:33,080::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914600::DEBUG::2013-11-20 03:35:33,081::task::495::TaskManager.Task::(_debug) Task ce73e285-1146-480c-9a73-e5fee1a339d0: ref 0 aborting False >Thread-15914600::INFO::2013-11-20 03:35:33,082::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000240087509155', 'lastCheck': 1384918527.773176, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0011260509491', 'lastCheck': 1384918523.7671471, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000925779342651', 'lastCheck': 1384918527.238373, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00043797492981', 'lastCheck': 1384918523.1650529, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:35:33,325::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0530863 s, 19.3 MB/s\n'; <rc> = 0 >Thread-15914602::DEBUG::2013-11-20 03:35:33,929::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914602::INFO::2013-11-20 03:35:33,930::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914602::DEBUG::2013-11-20 03:35:33,930::task::495::TaskManager.Task::(_debug) Task edd2d863-9576-4f80-84d4-e7b5d37a8eef: moving from state init -> state preparing >Thread-15914602::DEBUG::2013-11-20 03:35:33,931::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914602::DEBUG::2013-11-20 03:35:33,931::task::495::TaskManager.Task::(_debug) Task edd2d863-9576-4f80-84d4-e7b5d37a8eef: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914602::DEBUG::2013-11-20 03:35:33,932::task::495::TaskManager.Task::(_debug) Task edd2d863-9576-4f80-84d4-e7b5d37a8eef: moving from state preparing -> state finished >Thread-15914602::DEBUG::2013-11-20 03:35:33,932::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914602::DEBUG::2013-11-20 03:35:33,933::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914602::DEBUG::2013-11-20 03:35:33,933::task::495::TaskManager.Task::(_debug) Task edd2d863-9576-4f80-84d4-e7b5d37a8eef: ref 0 aborting False >Thread-15914602::INFO::2013-11-20 03:35:33,933::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914603::DEBUG::2013-11-20 03:35:33,938::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914603::INFO::2013-11-20 03:35:33,938::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914603::DEBUG::2013-11-20 03:35:33,939::task::495::TaskManager.Task::(_debug) Task d1c8380f-9fad-42dd-ab94-d8faf3c54d16: moving from state init -> state preparing >Thread-15914603::DEBUG::2013-11-20 03:35:33,940::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2e5e0451-7e16-49aa-ac4a-d7964d8f1c38`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914603::DEBUG::2013-11-20 03:35:33,940::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914603::DEBUG::2013-11-20 03:35:33,941::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914603::DEBUG::2013-11-20 03:35:33,941::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2e5e0451-7e16-49aa-ac4a-d7964d8f1c38`::Granted request >Thread-15914603::DEBUG::2013-11-20 03:35:33,944::task::495::TaskManager.Task::(_debug) Task d1c8380f-9fad-42dd-ab94-d8faf3c54d16: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914603::DEBUG::2013-11-20 03:35:33,950::task::495::TaskManager.Task::(_debug) Task d1c8380f-9fad-42dd-ab94-d8faf3c54d16: ref 1 aborting False >Thread-15914603::WARNING::2013-11-20 03:35:33,951::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914603::DEBUG::2013-11-20 03:35:33,952::task::495::TaskManager.Task::(_debug) Task d1c8380f-9fad-42dd-ab94-d8faf3c54d16: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914603::DEBUG::2013-11-20 03:35:33,952::task::495::TaskManager.Task::(_debug) Task d1c8380f-9fad-42dd-ab94-d8faf3c54d16: moving from state preparing -> state finished >Thread-15914603::DEBUG::2013-11-20 03:35:33,952::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914603::DEBUG::2013-11-20 03:35:33,953::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914603::DEBUG::2013-11-20 03:35:33,953::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914603::DEBUG::2013-11-20 03:35:33,954::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914603::DEBUG::2013-11-20 03:35:33,954::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914603::DEBUG::2013-11-20 03:35:33,954::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914603::DEBUG::2013-11-20 03:35:33,955::task::495::TaskManager.Task::(_debug) Task d1c8380f-9fad-42dd-ab94-d8faf3c54d16: ref 0 aborting False >Thread-15914603::INFO::2013-11-20 03:35:33,955::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:35:35,335::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:35:36,803::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0572287 s, 17.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:35:38,813::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:35:39,211::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0546583 s, 18.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:35:41,221::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:35:42,098::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0442831 s, 23.1 MB/s\n'; <rc> = 0 >Thread-15914436::DEBUG::2013-11-20 03:35:43,388::guestIF::97::vm.Vm::(_handleMessage) vmId=`5f587d5c-d4b4-4242-b049-ede331c7f8b4`::u'BRD-KLEE\\administrator' >Dummy-15912651::DEBUG::2013-11-20 03:35:44,108::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914607::DEBUG::2013-11-20 03:35:44,323::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914607::INFO::2013-11-20 03:35:44,326::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914607::DEBUG::2013-11-20 03:35:44,326::task::495::TaskManager.Task::(_debug) Task 43bfd8e3-e341-4cd8-9462-cb569b10840e: moving from state init -> state preparing >Thread-15914607::DEBUG::2013-11-20 03:35:44,327::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914607::DEBUG::2013-11-20 03:35:44,327::task::495::TaskManager.Task::(_debug) Task 43bfd8e3-e341-4cd8-9462-cb569b10840e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914607::DEBUG::2013-11-20 03:35:44,328::task::495::TaskManager.Task::(_debug) Task 43bfd8e3-e341-4cd8-9462-cb569b10840e: moving from state preparing -> state finished >Thread-15914607::DEBUG::2013-11-20 03:35:44,329::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914607::DEBUG::2013-11-20 03:35:44,329::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914607::DEBUG::2013-11-20 03:35:44,330::task::495::TaskManager.Task::(_debug) Task 43bfd8e3-e341-4cd8-9462-cb569b10840e: ref 0 aborting False >Thread-15914607::INFO::2013-11-20 03:35:44,330::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914608::DEBUG::2013-11-20 03:35:44,343::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914608::INFO::2013-11-20 03:35:44,344::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914608::DEBUG::2013-11-20 03:35:44,345::task::495::TaskManager.Task::(_debug) Task d28e61bf-104f-4820-8826-1563803d48fc: moving from state init -> state preparing >Thread-15914608::DEBUG::2013-11-20 03:35:44,351::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2be672fa-539c-485d-8913-6900833bb4c1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914608::DEBUG::2013-11-20 03:35:44,353::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914608::DEBUG::2013-11-20 03:35:44,353::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914608::DEBUG::2013-11-20 03:35:44,354::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2be672fa-539c-485d-8913-6900833bb4c1`::Granted request >Thread-15914608::DEBUG::2013-11-20 03:35:44,355::task::495::TaskManager.Task::(_debug) Task d28e61bf-104f-4820-8826-1563803d48fc: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914608::DEBUG::2013-11-20 03:35:44,355::task::495::TaskManager.Task::(_debug) Task d28e61bf-104f-4820-8826-1563803d48fc: ref 1 aborting False >Thread-15914608::WARNING::2013-11-20 03:35:44,357::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914608::DEBUG::2013-11-20 03:35:44,357::task::495::TaskManager.Task::(_debug) Task d28e61bf-104f-4820-8826-1563803d48fc: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914608::DEBUG::2013-11-20 03:35:44,358::task::495::TaskManager.Task::(_debug) Task d28e61bf-104f-4820-8826-1563803d48fc: moving from state preparing -> state finished >Thread-15914608::DEBUG::2013-11-20 03:35:44,358::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914608::DEBUG::2013-11-20 03:35:44,359::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914608::DEBUG::2013-11-20 03:35:44,362::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914608::DEBUG::2013-11-20 03:35:44,363::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914608::DEBUG::2013-11-20 03:35:44,363::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914608::DEBUG::2013-11-20 03:35:44,364::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914608::DEBUG::2013-11-20 03:35:44,364::task::495::TaskManager.Task::(_debug) Task d28e61bf-104f-4820-8826-1563803d48fc: ref 0 aborting False >Thread-15914608::INFO::2013-11-20 03:35:44,365::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:35:44,493::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0447846 s, 22.9 MB/s\n'; <rc> = 0 >Thread-15913639::INFO::2013-11-20 03:35:45,438::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:35:45,439::task::495::TaskManager.Task::(_debug) Task 24e628fd-3f42-4aa6-b751-515eed177c49: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:35:45,441::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0c30eb12-2242-467b-bb96-74d263beb10b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:35:45,441::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:35:45,442::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:35:45,442::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`0c30eb12-2242-467b-bb96-74d263beb10b`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:35:45,443::task::495::TaskManager.Task::(_debug) Task 24e628fd-3f42-4aa6-b751-515eed177c49: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:35:45,444::task::495::TaskManager.Task::(_debug) Task 24e628fd-3f42-4aa6-b751-515eed177c49: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:35:45,444::task::495::TaskManager.Task::(_debug) Task 24e628fd-3f42-4aa6-b751-515eed177c49: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:35:45,445::task::495::TaskManager.Task::(_debug) Task 24e628fd-3f42-4aa6-b751-515eed177c49: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:35:45,445::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:35:45,446::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:35:45,446::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:35:45,447::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:35:45,447::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:35:45,448::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:35:45,448::task::495::TaskManager.Task::(_debug) Task 24e628fd-3f42-4aa6-b751-515eed177c49: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:35:45,449::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:35:46,503::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:35:46,920::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0784612 s, 13.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:35:48,930::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914610::INFO::2013-11-20 03:35:49,097::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914610::DEBUG::2013-11-20 03:35:49,098::task::495::TaskManager.Task::(_debug) Task ed1b5161-b520-4947-afc8-9c8844862c42: moving from state init -> state preparing >Thread-15914610::DEBUG::2013-11-20 03:35:49,099::task::495::TaskManager.Task::(_debug) Task ed1b5161-b520-4947-afc8-9c8844862c42: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000236988067627', 'lastCheck': 1384918547.7964621, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0397200584412', 'lastCheck': 1384918543.8350899, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000925064086914', 'lastCheck': 1384918547.4431219, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00066089630127', 'lastCheck': 1384918543.2095771, 'valid': True, 'code': 0}} >Thread-15914610::DEBUG::2013-11-20 03:35:49,099::task::495::TaskManager.Task::(_debug) Task ed1b5161-b520-4947-afc8-9c8844862c42: moving from state preparing -> state finished >Thread-15914610::DEBUG::2013-11-20 03:35:49,100::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914610::DEBUG::2013-11-20 03:35:49,100::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914610::DEBUG::2013-11-20 03:35:49,101::task::495::TaskManager.Task::(_debug) Task ed1b5161-b520-4947-afc8-9c8844862c42: ref 0 aborting False >Thread-15914610::INFO::2013-11-20 03:35:49,101::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000236988067627', 'lastCheck': 1384918547.7964621, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0397200584412', 'lastCheck': 1384918543.8350899, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000925064086914', 'lastCheck': 1384918547.4431219, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00066089630127', 'lastCheck': 1384918543.2095771, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:35:49,339::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0576739 s, 17.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:35:51,349::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:35:51,780::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0495278 s, 20.7 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:35:53,792::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:35:54,210::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.056467 s, 18.1 MB/s\n'; <rc> = 0 >Thread-15914613::DEBUG::2013-11-20 03:35:54,770::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914613::INFO::2013-11-20 03:35:54,771::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914613::DEBUG::2013-11-20 03:35:54,772::task::495::TaskManager.Task::(_debug) Task 1621fd4e-72ae-40a4-9d9a-033135e76a51: moving from state init -> state preparing >Thread-15914613::DEBUG::2013-11-20 03:35:54,772::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914613::DEBUG::2013-11-20 03:35:54,773::task::495::TaskManager.Task::(_debug) Task 1621fd4e-72ae-40a4-9d9a-033135e76a51: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914613::DEBUG::2013-11-20 03:35:54,773::task::495::TaskManager.Task::(_debug) Task 1621fd4e-72ae-40a4-9d9a-033135e76a51: moving from state preparing -> state finished >Thread-15914613::DEBUG::2013-11-20 03:35:54,774::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914613::DEBUG::2013-11-20 03:35:54,774::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914613::DEBUG::2013-11-20 03:35:54,774::task::495::TaskManager.Task::(_debug) Task 1621fd4e-72ae-40a4-9d9a-033135e76a51: ref 0 aborting False >Thread-15914613::INFO::2013-11-20 03:35:54,775::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914614::DEBUG::2013-11-20 03:35:54,782::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914614::INFO::2013-11-20 03:35:54,783::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914614::DEBUG::2013-11-20 03:35:54,783::task::495::TaskManager.Task::(_debug) Task fb3c4cdb-7795-41e4-b983-a26afda4a4af: moving from state init -> state preparing >Thread-15914614::DEBUG::2013-11-20 03:35:54,796::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a2d9710b-de7c-47b4-a17f-4f3ce37f8a1d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914614::DEBUG::2013-11-20 03:35:54,797::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914614::DEBUG::2013-11-20 03:35:54,797::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914614::DEBUG::2013-11-20 03:35:54,798::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a2d9710b-de7c-47b4-a17f-4f3ce37f8a1d`::Granted request >Thread-15914614::DEBUG::2013-11-20 03:35:54,799::task::495::TaskManager.Task::(_debug) Task fb3c4cdb-7795-41e4-b983-a26afda4a4af: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914614::DEBUG::2013-11-20 03:35:54,799::task::495::TaskManager.Task::(_debug) Task fb3c4cdb-7795-41e4-b983-a26afda4a4af: ref 1 aborting False >Thread-15914614::WARNING::2013-11-20 03:35:54,801::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914614::DEBUG::2013-11-20 03:35:54,801::task::495::TaskManager.Task::(_debug) Task fb3c4cdb-7795-41e4-b983-a26afda4a4af: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914614::DEBUG::2013-11-20 03:35:54,802::task::495::TaskManager.Task::(_debug) Task fb3c4cdb-7795-41e4-b983-a26afda4a4af: moving from state preparing -> state finished >Thread-15914614::DEBUG::2013-11-20 03:35:54,803::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914614::DEBUG::2013-11-20 03:35:54,803::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914614::DEBUG::2013-11-20 03:35:54,804::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914614::DEBUG::2013-11-20 03:35:54,804::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914614::DEBUG::2013-11-20 03:35:54,805::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914614::DEBUG::2013-11-20 03:35:54,805::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914614::DEBUG::2013-11-20 03:35:54,806::task::495::TaskManager.Task::(_debug) Task fb3c4cdb-7795-41e4-b983-a26afda4a4af: ref 0 aborting False >Thread-15914614::INFO::2013-11-20 03:35:54,806::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:35:56,221::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:35:56,639::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0590469 s, 17.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:35:58,647::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:35:59,068::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0587303 s, 17.4 MB/s\n'; <rc> = 0 >Thread-15913658::INFO::2013-11-20 03:36:00,390::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:36:00,401::task::495::TaskManager.Task::(_debug) Task dc361f7f-540e-45b5-bbab-3be91d7bdec9: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:36:00,403::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f67d8db9-0b81-48a9-9c30-11f8fdbb9f02`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:36:00,404::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:36:00,404::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:36:00,405::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f67d8db9-0b81-48a9-9c30-11f8fdbb9f02`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:36:00,406::task::495::TaskManager.Task::(_debug) Task dc361f7f-540e-45b5-bbab-3be91d7bdec9: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:36:00,406::task::495::TaskManager.Task::(_debug) Task dc361f7f-540e-45b5-bbab-3be91d7bdec9: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:36:00,407::task::495::TaskManager.Task::(_debug) Task dc361f7f-540e-45b5-bbab-3be91d7bdec9: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:36:00,407::task::495::TaskManager.Task::(_debug) Task dc361f7f-540e-45b5-bbab-3be91d7bdec9: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:36:00,408::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:36:00,408::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:36:00,409::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:36:00,410::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:36:00,410::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:36:00,411::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:36:00,411::task::495::TaskManager.Task::(_debug) Task dc361f7f-540e-45b5-bbab-3be91d7bdec9: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:36:00,412::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:36:01,081::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:36:02,481::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0567035 s, 18.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:36:04,494::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:36:04,921::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0705668 s, 14.5 MB/s\n'; <rc> = 0 >Thread-15914618::INFO::2013-11-20 03:36:05,266::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914618::DEBUG::2013-11-20 03:36:05,267::task::495::TaskManager.Task::(_debug) Task f6db3d7a-350d-467f-9738-d3893cb51b25: moving from state init -> state preparing >Thread-15914618::DEBUG::2013-11-20 03:36:05,267::task::495::TaskManager.Task::(_debug) Task f6db3d7a-350d-467f-9738-d3893cb51b25: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000257968902588', 'lastCheck': 1384918557.9730289, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000876188278198', 'lastCheck': 1384918563.9844229, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000818014144897', 'lastCheck': 1384918557.533258, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000280141830444', 'lastCheck': 1384918563.3964851, 'valid': True, 'code': 0}} >Thread-15914618::DEBUG::2013-11-20 03:36:05,268::task::495::TaskManager.Task::(_debug) Task f6db3d7a-350d-467f-9738-d3893cb51b25: moving from state preparing -> state finished >Thread-15914618::DEBUG::2013-11-20 03:36:05,269::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914619::DEBUG::2013-11-20 03:36:05,282::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914618::DEBUG::2013-11-20 03:36:05,282::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914619::INFO::2013-11-20 03:36:05,283::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914618::DEBUG::2013-11-20 03:36:05,283::task::495::TaskManager.Task::(_debug) Task f6db3d7a-350d-467f-9738-d3893cb51b25: ref 0 aborting False >Thread-15914619::DEBUG::2013-11-20 03:36:05,284::task::495::TaskManager.Task::(_debug) Task 411eae88-7129-4bda-ac47-3e94d4163cbf: moving from state init -> state preparing >Thread-15914618::INFO::2013-11-20 03:36:05,284::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000257968902588', 'lastCheck': 1384918557.9730289, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000876188278198', 'lastCheck': 1384918563.9844229, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000818014144897', 'lastCheck': 1384918557.533258, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000280141830444', 'lastCheck': 1384918563.3964851, 'valid': True, 'code': 0}} >Thread-15914619::DEBUG::2013-11-20 03:36:05,285::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914619::DEBUG::2013-11-20 03:36:05,286::task::495::TaskManager.Task::(_debug) Task 411eae88-7129-4bda-ac47-3e94d4163cbf: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914619::DEBUG::2013-11-20 03:36:05,287::task::495::TaskManager.Task::(_debug) Task 411eae88-7129-4bda-ac47-3e94d4163cbf: moving from state preparing -> state finished >Thread-15914619::DEBUG::2013-11-20 03:36:05,287::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914619::DEBUG::2013-11-20 03:36:05,288::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914619::DEBUG::2013-11-20 03:36:05,311::task::495::TaskManager.Task::(_debug) Task 411eae88-7129-4bda-ac47-3e94d4163cbf: ref 0 aborting False >Thread-15914619::INFO::2013-11-20 03:36:05,312::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914620::DEBUG::2013-11-20 03:36:05,341::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914620::INFO::2013-11-20 03:36:05,342::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914620::DEBUG::2013-11-20 03:36:05,343::task::495::TaskManager.Task::(_debug) Task 56d2f663-6385-41cd-96d8-d05dbd45fcfc: moving from state init -> state preparing >Thread-15914620::DEBUG::2013-11-20 03:36:05,346::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6d76d377-c499-4848-b4c4-514cd5b8c7ef`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914620::DEBUG::2013-11-20 03:36:05,347::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914620::DEBUG::2013-11-20 03:36:05,347::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914620::DEBUG::2013-11-20 03:36:05,348::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6d76d377-c499-4848-b4c4-514cd5b8c7ef`::Granted request >Thread-15914620::DEBUG::2013-11-20 03:36:05,349::task::495::TaskManager.Task::(_debug) Task 56d2f663-6385-41cd-96d8-d05dbd45fcfc: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914620::DEBUG::2013-11-20 03:36:05,349::task::495::TaskManager.Task::(_debug) Task 56d2f663-6385-41cd-96d8-d05dbd45fcfc: ref 1 aborting False >Thread-15914620::WARNING::2013-11-20 03:36:05,351::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914620::DEBUG::2013-11-20 03:36:05,352::task::495::TaskManager.Task::(_debug) Task 56d2f663-6385-41cd-96d8-d05dbd45fcfc: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914620::DEBUG::2013-11-20 03:36:05,353::task::495::TaskManager.Task::(_debug) Task 56d2f663-6385-41cd-96d8-d05dbd45fcfc: moving from state preparing -> state finished >Thread-15914620::DEBUG::2013-11-20 03:36:05,354::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914620::DEBUG::2013-11-20 03:36:05,354::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914620::DEBUG::2013-11-20 03:36:05,355::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914620::DEBUG::2013-11-20 03:36:05,355::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914620::DEBUG::2013-11-20 03:36:05,355::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914620::DEBUG::2013-11-20 03:36:05,356::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914620::DEBUG::2013-11-20 03:36:05,356::task::495::TaskManager.Task::(_debug) Task 56d2f663-6385-41cd-96d8-d05dbd45fcfc: ref 0 aborting False >Thread-15914620::INFO::2013-11-20 03:36:05,361::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15912392::INFO::2013-11-20 03:36:05,533::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:36:05,534::task::495::TaskManager.Task::(_debug) Task 4691648a-e5cc-4e57-a6f8-10c0b84fba84: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:36:05,535::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d2593d85-0a0b-46dd-a1f4-d9b1ba3b2934`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:36:05,536::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:36:05,537::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:36:05,537::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d2593d85-0a0b-46dd-a1f4-d9b1ba3b2934`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:36:05,538::task::495::TaskManager.Task::(_debug) Task 4691648a-e5cc-4e57-a6f8-10c0b84fba84: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:36:05,538::task::495::TaskManager.Task::(_debug) Task 4691648a-e5cc-4e57-a6f8-10c0b84fba84: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:36:05,539::task::495::TaskManager.Task::(_debug) Task 4691648a-e5cc-4e57-a6f8-10c0b84fba84: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:36:05,539::task::495::TaskManager.Task::(_debug) Task 4691648a-e5cc-4e57-a6f8-10c0b84fba84: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:36:05,540::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:36:05,540::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:36:05,541::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:36:05,541::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:36:05,542::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:36:05,542::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:36:05,542::task::495::TaskManager.Task::(_debug) Task 4691648a-e5cc-4e57-a6f8-10c0b84fba84: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:36:05,543::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:36:06,932::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:36:07,356::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0766409 s, 13.4 MB/s\n'; <rc> = 0 >Thread-15913695::INFO::2013-11-20 03:36:08,211::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:36:08,212::task::495::TaskManager.Task::(_debug) Task c203883f-869d-44ca-990b-ef08cbf41732: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:36:08,213::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e80bf180-8c10-4617-b31d-56dcaf7120b1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:36:08,214::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:36:08,214::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:36:08,214::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e80bf180-8c10-4617-b31d-56dcaf7120b1`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:36:08,215::task::495::TaskManager.Task::(_debug) Task c203883f-869d-44ca-990b-ef08cbf41732: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:36:08,216::task::495::TaskManager.Task::(_debug) Task c203883f-869d-44ca-990b-ef08cbf41732: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:36:08,216::task::495::TaskManager.Task::(_debug) Task c203883f-869d-44ca-990b-ef08cbf41732: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:36:08,216::task::495::TaskManager.Task::(_debug) Task c203883f-869d-44ca-990b-ef08cbf41732: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:36:08,217::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:36:08,217::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:36:08,218::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:36:08,218::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:36:08,218::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:36:08,219::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:36:08,219::task::495::TaskManager.Task::(_debug) Task c203883f-869d-44ca-990b-ef08cbf41732: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:36:08,220::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Dummy-15912651::DEBUG::2013-11-20 03:36:09,367::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:36:09,807::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0529161 s, 19.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:36:11,819::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:36:12,259::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0541716 s, 18.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:36:14,269::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:36:14,689::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0492822 s, 20.8 MB/s\n'; <rc> = 0 >Thread-15914625::DEBUG::2013-11-20 03:36:15,910::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914625::INFO::2013-11-20 03:36:15,911::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914625::DEBUG::2013-11-20 03:36:15,912::task::495::TaskManager.Task::(_debug) Task 69b22a07-85ce-4220-8a75-0592fbf8dbd4: moving from state init -> state preparing >Thread-15914625::DEBUG::2013-11-20 03:36:15,913::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914625::DEBUG::2013-11-20 03:36:15,913::task::495::TaskManager.Task::(_debug) Task 69b22a07-85ce-4220-8a75-0592fbf8dbd4: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914625::DEBUG::2013-11-20 03:36:15,914::task::495::TaskManager.Task::(_debug) Task 69b22a07-85ce-4220-8a75-0592fbf8dbd4: moving from state preparing -> state finished >Thread-15914625::DEBUG::2013-11-20 03:36:15,914::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914625::DEBUG::2013-11-20 03:36:15,915::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914625::DEBUG::2013-11-20 03:36:15,915::task::495::TaskManager.Task::(_debug) Task 69b22a07-85ce-4220-8a75-0592fbf8dbd4: ref 0 aborting False >Thread-15914625::INFO::2013-11-20 03:36:15,916::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914626::DEBUG::2013-11-20 03:36:15,925::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914626::INFO::2013-11-20 03:36:15,926::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914626::DEBUG::2013-11-20 03:36:15,926::task::495::TaskManager.Task::(_debug) Task 6bf08d54-6536-4e40-a2bf-47674d5d0522: moving from state init -> state preparing >Thread-15914626::DEBUG::2013-11-20 03:36:15,927::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6e22e976-fa5e-4e23-a9d4-4953976433d4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914626::DEBUG::2013-11-20 03:36:15,928::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914626::DEBUG::2013-11-20 03:36:15,928::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914626::DEBUG::2013-11-20 03:36:15,929::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`6e22e976-fa5e-4e23-a9d4-4953976433d4`::Granted request >Thread-15914626::DEBUG::2013-11-20 03:36:15,929::task::495::TaskManager.Task::(_debug) Task 6bf08d54-6536-4e40-a2bf-47674d5d0522: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914626::DEBUG::2013-11-20 03:36:15,934::task::495::TaskManager.Task::(_debug) Task 6bf08d54-6536-4e40-a2bf-47674d5d0522: ref 1 aborting False >Thread-15914626::WARNING::2013-11-20 03:36:15,935::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914626::DEBUG::2013-11-20 03:36:15,936::task::495::TaskManager.Task::(_debug) Task 6bf08d54-6536-4e40-a2bf-47674d5d0522: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914626::DEBUG::2013-11-20 03:36:15,937::task::495::TaskManager.Task::(_debug) Task 6bf08d54-6536-4e40-a2bf-47674d5d0522: moving from state preparing -> state finished >Thread-15914626::DEBUG::2013-11-20 03:36:15,938::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914626::DEBUG::2013-11-20 03:36:15,938::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914626::DEBUG::2013-11-20 03:36:15,939::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914626::DEBUG::2013-11-20 03:36:15,939::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914626::DEBUG::2013-11-20 03:36:15,939::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914626::DEBUG::2013-11-20 03:36:15,940::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914626::DEBUG::2013-11-20 03:36:15,940::task::495::TaskManager.Task::(_debug) Task 6bf08d54-6536-4e40-a2bf-47674d5d0522: ref 0 aborting False >Thread-15914626::INFO::2013-11-20 03:36:15,941::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:36:16,698::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:36:17,090::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0466652 s, 21.9 MB/s\n'; <rc> = 0 >Thread-15914435::INFO::2013-11-20 03:36:18,490::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=143cb26d-de08-4d30-89cf-aee10245b66a volUUID=d7028f7c-0551-4bbe-a907-6f804cc9bd54) >Thread-15914435::DEBUG::2013-11-20 03:36:18,491::task::495::TaskManager.Task::(_debug) Task 4c5e6ba8-c896-4a4a-9ce0-ba35b30e2751: moving from state init -> state preparing >Thread-15914435::DEBUG::2013-11-20 03:36:18,492::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`dd3deb08-02c6-4ee6-b00d-9be377e153ef`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914435::DEBUG::2013-11-20 03:36:18,492::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914435::DEBUG::2013-11-20 03:36:18,493::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914435::DEBUG::2013-11-20 03:36:18,493::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`dd3deb08-02c6-4ee6-b00d-9be377e153ef`::Granted request >Thread-15914435::DEBUG::2013-11-20 03:36:18,494::task::495::TaskManager.Task::(_debug) Task 4c5e6ba8-c896-4a4a-9ce0-ba35b30e2751: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914435::DEBUG::2013-11-20 03:36:18,494::task::495::TaskManager.Task::(_debug) Task 4c5e6ba8-c896-4a4a-9ce0-ba35b30e2751: ref 1 aborting False >Thread-15914435::DEBUG::2013-11-20 03:36:18,495::task::495::TaskManager.Task::(_debug) Task 4c5e6ba8-c896-4a4a-9ce0-ba35b30e2751: finished: {'truesize': '56908316672', 'apparentsize': '56908316672'} >Thread-15914435::DEBUG::2013-11-20 03:36:18,495::task::495::TaskManager.Task::(_debug) Task 4c5e6ba8-c896-4a4a-9ce0-ba35b30e2751: moving from state preparing -> state finished >Thread-15914435::DEBUG::2013-11-20 03:36:18,496::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914435::DEBUG::2013-11-20 03:36:18,496::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914435::DEBUG::2013-11-20 03:36:18,497::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914435::DEBUG::2013-11-20 03:36:18,497::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914435::DEBUG::2013-11-20 03:36:18,499::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914435::DEBUG::2013-11-20 03:36:18,500::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914435::DEBUG::2013-11-20 03:36:18,500::task::495::TaskManager.Task::(_debug) Task 4c5e6ba8-c896-4a4a-9ce0-ba35b30e2751: ref 0 aborting False >Thread-15914435::INFO::2013-11-20 03:36:18,501::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '56908316672', 'apparentsize': '56908316672'} >Dummy-15912651::DEBUG::2013-11-20 03:36:19,100::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:36:19,757::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0499419 s, 20.5 MB/s\n'; <rc> = 0 >Thread-15914453::INFO::2013-11-20 03:36:20,466::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=5f930ef3-763f-4294-a62a-6d52df220f8c volUUID=b9cffdbe-152e-41ce-ae5d-08db7c08454e) >Thread-15914453::DEBUG::2013-11-20 03:36:20,467::task::495::TaskManager.Task::(_debug) Task 395a7bbb-f416-4769-adaa-b82283bf56ce: moving from state init -> state preparing >Thread-15914453::DEBUG::2013-11-20 03:36:20,484::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d465cb05-f80d-4ee5-8e07-457443bd41d9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914453::DEBUG::2013-11-20 03:36:20,485::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914453::DEBUG::2013-11-20 03:36:20,486::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914453::DEBUG::2013-11-20 03:36:20,487::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`d465cb05-f80d-4ee5-8e07-457443bd41d9`::Granted request >Thread-15914453::DEBUG::2013-11-20 03:36:20,488::task::495::TaskManager.Task::(_debug) Task 395a7bbb-f416-4769-adaa-b82283bf56ce: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914453::DEBUG::2013-11-20 03:36:20,488::task::495::TaskManager.Task::(_debug) Task 395a7bbb-f416-4769-adaa-b82283bf56ce: ref 1 aborting False >Thread-15914453::DEBUG::2013-11-20 03:36:20,489::task::495::TaskManager.Task::(_debug) Task 395a7bbb-f416-4769-adaa-b82283bf56ce: finished: {'truesize': '53821308928', 'apparentsize': '53821308928'} >Thread-15914453::DEBUG::2013-11-20 03:36:20,489::task::495::TaskManager.Task::(_debug) Task 395a7bbb-f416-4769-adaa-b82283bf56ce: moving from state preparing -> state finished >Thread-15914453::DEBUG::2013-11-20 03:36:20,490::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914453::DEBUG::2013-11-20 03:36:20,490::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914453::DEBUG::2013-11-20 03:36:20,491::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914453::DEBUG::2013-11-20 03:36:20,491::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914453::DEBUG::2013-11-20 03:36:20,491::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914453::DEBUG::2013-11-20 03:36:20,492::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914453::DEBUG::2013-11-20 03:36:20,492::task::495::TaskManager.Task::(_debug) Task 395a7bbb-f416-4769-adaa-b82283bf56ce: ref 0 aborting False >Thread-15914453::INFO::2013-11-20 03:36:20,493::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '53821308928', 'apparentsize': '53821308928'} >Thread-15914628::INFO::2013-11-20 03:36:21,214::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914628::DEBUG::2013-11-20 03:36:21,214::task::495::TaskManager.Task::(_debug) Task 161a4fe8-b5e0-4d2b-a642-02c606ed7707: moving from state init -> state preparing >Thread-15914628::DEBUG::2013-11-20 03:36:21,215::task::495::TaskManager.Task::(_debug) Task 161a4fe8-b5e0-4d2b-a642-02c606ed7707: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000280141830444', 'lastCheck': 1384918578.109441, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0189471244812', 'lastCheck': 1384918574.1225669, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00127506256104', 'lastCheck': 1384918577.6890521, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000320911407471', 'lastCheck': 1384918573.5659461, 'valid': True, 'code': 0}} >Thread-15914628::DEBUG::2013-11-20 03:36:21,216::task::495::TaskManager.Task::(_debug) Task 161a4fe8-b5e0-4d2b-a642-02c606ed7707: moving from state preparing -> state finished >Thread-15914628::DEBUG::2013-11-20 03:36:21,216::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914628::DEBUG::2013-11-20 03:36:21,217::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914628::DEBUG::2013-11-20 03:36:21,217::task::495::TaskManager.Task::(_debug) Task 161a4fe8-b5e0-4d2b-a642-02c606ed7707: ref 0 aborting False >Thread-15914628::INFO::2013-11-20 03:36:21,218::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000280141830444', 'lastCheck': 1384918578.109441, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0189471244812', 'lastCheck': 1384918574.1225669, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00127506256104', 'lastCheck': 1384918577.6890521, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000320911407471', 'lastCheck': 1384918573.5659461, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:36:21,767::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:36:22,175::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0509196 s, 20.1 MB/s\n'; <rc> = 0 >Thread-15914519::INFO::2013-11-20 03:36:23,980::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=5af747d8-95b1-4b7e-af53-cb77e4fcc896 volUUID=8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f) >Thread-15914519::DEBUG::2013-11-20 03:36:23,981::task::495::TaskManager.Task::(_debug) Task f31e806c-be39-4738-9aa8-790bcda6f703: moving from state init -> state preparing >Thread-15914519::DEBUG::2013-11-20 03:36:23,993::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`89472347-8f57-4e81-8b06-928048ad5432`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914519::DEBUG::2013-11-20 03:36:23,993::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914519::DEBUG::2013-11-20 03:36:23,994::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914519::DEBUG::2013-11-20 03:36:23,994::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`89472347-8f57-4e81-8b06-928048ad5432`::Granted request >Thread-15914519::DEBUG::2013-11-20 03:36:24,003::task::495::TaskManager.Task::(_debug) Task f31e806c-be39-4738-9aa8-790bcda6f703: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914519::DEBUG::2013-11-20 03:36:24,005::task::495::TaskManager.Task::(_debug) Task f31e806c-be39-4738-9aa8-790bcda6f703: ref 1 aborting False >Thread-15914519::DEBUG::2013-11-20 03:36:24,006::task::495::TaskManager.Task::(_debug) Task f31e806c-be39-4738-9aa8-790bcda6f703: finished: {'truesize': '86033563648', 'apparentsize': '86033563648'} >Thread-15914519::DEBUG::2013-11-20 03:36:24,006::task::495::TaskManager.Task::(_debug) Task f31e806c-be39-4738-9aa8-790bcda6f703: moving from state preparing -> state finished >Thread-15914519::DEBUG::2013-11-20 03:36:24,007::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914519::DEBUG::2013-11-20 03:36:24,007::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914519::DEBUG::2013-11-20 03:36:24,008::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914519::DEBUG::2013-11-20 03:36:24,008::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914519::DEBUG::2013-11-20 03:36:24,009::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914519::DEBUG::2013-11-20 03:36:24,009::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914519::DEBUG::2013-11-20 03:36:24,010::task::495::TaskManager.Task::(_debug) Task f31e806c-be39-4738-9aa8-790bcda6f703: ref 0 aborting False >Thread-15914519::INFO::2013-11-20 03:36:24,010::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '86033563648', 'apparentsize': '86033563648'} >Dummy-15912651::DEBUG::2013-11-20 03:36:24,186::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:36:24,712::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0582696 s, 17.6 MB/s\n'; <rc> = 0 >Thread-15914631::DEBUG::2013-11-20 03:36:26,548::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914631::INFO::2013-11-20 03:36:26,549::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914631::DEBUG::2013-11-20 03:36:26,549::task::495::TaskManager.Task::(_debug) Task 778046d5-8be7-4544-98a4-5684a3b33e92: moving from state init -> state preparing >Thread-15914631::DEBUG::2013-11-20 03:36:26,550::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914631::DEBUG::2013-11-20 03:36:26,550::task::495::TaskManager.Task::(_debug) Task 778046d5-8be7-4544-98a4-5684a3b33e92: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914631::DEBUG::2013-11-20 03:36:26,551::task::495::TaskManager.Task::(_debug) Task 778046d5-8be7-4544-98a4-5684a3b33e92: moving from state preparing -> state finished >Thread-15914631::DEBUG::2013-11-20 03:36:26,551::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914631::DEBUG::2013-11-20 03:36:26,552::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914631::DEBUG::2013-11-20 03:36:26,552::task::495::TaskManager.Task::(_debug) Task 778046d5-8be7-4544-98a4-5684a3b33e92: ref 0 aborting False >Thread-15914631::INFO::2013-11-20 03:36:26,556::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914632::DEBUG::2013-11-20 03:36:26,560::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914632::INFO::2013-11-20 03:36:26,561::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914632::DEBUG::2013-11-20 03:36:26,570::task::495::TaskManager.Task::(_debug) Task ade2c76e-f984-4e01-914c-d339f0a0fffc: moving from state init -> state preparing >Thread-15914632::DEBUG::2013-11-20 03:36:26,571::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f14fea54-9719-4556-bd91-4aef88a50a21`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914632::DEBUG::2013-11-20 03:36:26,571::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914632::DEBUG::2013-11-20 03:36:26,575::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914632::DEBUG::2013-11-20 03:36:26,575::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`f14fea54-9719-4556-bd91-4aef88a50a21`::Granted request >Thread-15914632::DEBUG::2013-11-20 03:36:26,576::task::495::TaskManager.Task::(_debug) Task ade2c76e-f984-4e01-914c-d339f0a0fffc: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914632::DEBUG::2013-11-20 03:36:26,577::task::495::TaskManager.Task::(_debug) Task ade2c76e-f984-4e01-914c-d339f0a0fffc: ref 1 aborting False >Thread-15914632::WARNING::2013-11-20 03:36:26,579::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914632::DEBUG::2013-11-20 03:36:26,579::task::495::TaskManager.Task::(_debug) Task ade2c76e-f984-4e01-914c-d339f0a0fffc: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914632::DEBUG::2013-11-20 03:36:26,580::task::495::TaskManager.Task::(_debug) Task ade2c76e-f984-4e01-914c-d339f0a0fffc: moving from state preparing -> state finished >Thread-15914632::DEBUG::2013-11-20 03:36:26,581::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914632::DEBUG::2013-11-20 03:36:26,581::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914632::DEBUG::2013-11-20 03:36:26,582::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914632::DEBUG::2013-11-20 03:36:26,582::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914632::DEBUG::2013-11-20 03:36:26,583::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914632::DEBUG::2013-11-20 03:36:26,583::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914632::DEBUG::2013-11-20 03:36:26,584::task::495::TaskManager.Task::(_debug) Task ade2c76e-f984-4e01-914c-d339f0a0fffc: ref 0 aborting False >Thread-15914632::INFO::2013-11-20 03:36:26,584::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:36:26,722::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:36:27,135::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.047749 s, 21.4 MB/s\n'; <rc> = 0 >Thread-15913518::INFO::2013-11-20 03:36:28,649::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:36:28,650::task::495::TaskManager.Task::(_debug) Task 6aa7eea4-7d93-4bbe-8bb0-a00f22adbdfd: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:36:28,651::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`358dc341-d058-4af6-b888-1ae76391fd55`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:36:28,652::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:36:28,658::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:36:28,659::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`358dc341-d058-4af6-b888-1ae76391fd55`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:36:28,659::task::495::TaskManager.Task::(_debug) Task 6aa7eea4-7d93-4bbe-8bb0-a00f22adbdfd: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:36:28,660::task::495::TaskManager.Task::(_debug) Task 6aa7eea4-7d93-4bbe-8bb0-a00f22adbdfd: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:36:28,660::task::495::TaskManager.Task::(_debug) Task 6aa7eea4-7d93-4bbe-8bb0-a00f22adbdfd: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:36:28,661::task::495::TaskManager.Task::(_debug) Task 6aa7eea4-7d93-4bbe-8bb0-a00f22adbdfd: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:36:28,661::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:36:28,662::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:36:28,662::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:36:28,662::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:36:28,663::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:36:28,663::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:36:28,663::task::495::TaskManager.Task::(_debug) Task 6aa7eea4-7d93-4bbe-8bb0-a00f22adbdfd: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:36:28,664::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:36:28,664::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:36:28,665::task::495::TaskManager.Task::(_debug) Task d1ba8b54-97da-4e4e-ac5e-7b8a6f2a9862: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:36:28,665::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`29f92141-feff-46a6-8c29-de77a97d3455`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:36:28,666::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:36:28,666::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:36:28,667::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`29f92141-feff-46a6-8c29-de77a97d3455`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:36:28,667::task::495::TaskManager.Task::(_debug) Task d1ba8b54-97da-4e4e-ac5e-7b8a6f2a9862: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:36:28,668::task::495::TaskManager.Task::(_debug) Task d1ba8b54-97da-4e4e-ac5e-7b8a6f2a9862: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:36:28,668::task::495::TaskManager.Task::(_debug) Task d1ba8b54-97da-4e4e-ac5e-7b8a6f2a9862: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:36:28,668::task::495::TaskManager.Task::(_debug) Task d1ba8b54-97da-4e4e-ac5e-7b8a6f2a9862: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:36:28,669::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:36:28,669::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:36:28,669::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:36:28,670::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:36:28,670::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:36:28,670::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:36:28,671::task::495::TaskManager.Task::(_debug) Task d1ba8b54-97da-4e4e-ac5e-7b8a6f2a9862: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:36:28,671::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Dummy-15912651::DEBUG::2013-11-20 03:36:29,145::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:36:30,058::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0510143 s, 20.1 MB/s\n'; <rc> = 0 >Thread-15913546::INFO::2013-11-20 03:36:30,642::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:36:30,658::task::495::TaskManager.Task::(_debug) Task 7216eb00-0bbd-4806-9b39-e1c65ddc7ce7: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:36:30,660::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`a9b9e01d-5c1e-4860-bca0-55af77fff4f3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:36:30,661::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:36:30,662::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:36:30,662::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`a9b9e01d-5c1e-4860-bca0-55af77fff4f3`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:36:30,664::task::495::TaskManager.Task::(_debug) Task 7216eb00-0bbd-4806-9b39-e1c65ddc7ce7: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:36:30,664::task::495::TaskManager.Task::(_debug) Task 7216eb00-0bbd-4806-9b39-e1c65ddc7ce7: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:36:30,665::task::495::TaskManager.Task::(_debug) Task 7216eb00-0bbd-4806-9b39-e1c65ddc7ce7: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:36:30,666::task::495::TaskManager.Task::(_debug) Task 7216eb00-0bbd-4806-9b39-e1c65ddc7ce7: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:36:30,666::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:36:30,667::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:36:30,667::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:36:30,668::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:36:30,668::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:36:30,669::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:36:30,669::task::495::TaskManager.Task::(_debug) Task 7216eb00-0bbd-4806-9b39-e1c65ddc7ce7: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:36:30,670::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:36:30,670::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:36:30,671::task::495::TaskManager.Task::(_debug) Task cf13df7c-01a8-4f17-9161-b1831900cb59: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:36:30,672::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`8a7269fe-46ae-4315-88a4-4be424fd0c44`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:36:30,672::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:36:30,673::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:36:30,673::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`8a7269fe-46ae-4315-88a4-4be424fd0c44`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:36:30,674::task::495::TaskManager.Task::(_debug) Task cf13df7c-01a8-4f17-9161-b1831900cb59: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:36:30,674::task::495::TaskManager.Task::(_debug) Task cf13df7c-01a8-4f17-9161-b1831900cb59: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:36:30,675::task::495::TaskManager.Task::(_debug) Task cf13df7c-01a8-4f17-9161-b1831900cb59: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:36:30,675::task::495::TaskManager.Task::(_debug) Task cf13df7c-01a8-4f17-9161-b1831900cb59: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:36:30,675::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:36:30,676::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:36:30,676::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:36:30,677::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:36:30,677::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:36:30,677::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:36:30,678::task::495::TaskManager.Task::(_debug) Task cf13df7c-01a8-4f17-9161-b1831900cb59: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:36:30,678::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:36:30,679::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:36:30,679::task::495::TaskManager.Task::(_debug) Task e23562e2-e055-44f0-a14b-5ba3515d2e58: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:36:30,680::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`cf141f97-2a97-4f7a-aff8-17281c04f835`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:36:30,680::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:36:30,681::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:36:30,681::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`cf141f97-2a97-4f7a-aff8-17281c04f835`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:36:30,682::task::495::TaskManager.Task::(_debug) Task e23562e2-e055-44f0-a14b-5ba3515d2e58: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:36:30,682::task::495::TaskManager.Task::(_debug) Task e23562e2-e055-44f0-a14b-5ba3515d2e58: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:36:30,683::task::495::TaskManager.Task::(_debug) Task e23562e2-e055-44f0-a14b-5ba3515d2e58: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:36:30,683::task::495::TaskManager.Task::(_debug) Task e23562e2-e055-44f0-a14b-5ba3515d2e58: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:36:30,683::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:36:30,684::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:36:30,684::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:36:30,685::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:36:30,685::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:36:30,685::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:36:30,686::task::495::TaskManager.Task::(_debug) Task e23562e2-e055-44f0-a14b-5ba3515d2e58: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:36:30,686::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:36:30,687::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:36:30,687::task::495::TaskManager.Task::(_debug) Task 4645e177-568a-436a-bff8-7eb2c9d05e91: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:36:30,688::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`16d26ba2-662c-4ae4-8728-4c0a444a90f5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:36:30,688::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:36:30,689::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:36:30,689::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`16d26ba2-662c-4ae4-8728-4c0a444a90f5`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:36:30,690::task::495::TaskManager.Task::(_debug) Task 4645e177-568a-436a-bff8-7eb2c9d05e91: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:36:30,690::task::495::TaskManager.Task::(_debug) Task 4645e177-568a-436a-bff8-7eb2c9d05e91: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:36:30,691::task::495::TaskManager.Task::(_debug) Task 4645e177-568a-436a-bff8-7eb2c9d05e91: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:36:30,691::task::495::TaskManager.Task::(_debug) Task 4645e177-568a-436a-bff8-7eb2c9d05e91: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:36:30,692::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:36:30,692::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:36:30,692::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:36:30,693::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:36:30,693::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:36:30,694::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:36:30,694::task::495::TaskManager.Task::(_debug) Task 4645e177-568a-436a-bff8-7eb2c9d05e91: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:36:30,694::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:36:32,068::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:36:34,723::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0615824 s, 16.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:36:36,738::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914636::DEBUG::2013-11-20 03:36:36,924::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914637::INFO::2013-11-20 03:36:36,928::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914636::INFO::2013-11-20 03:36:36,929::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914637::DEBUG::2013-11-20 03:36:36,930::task::495::TaskManager.Task::(_debug) Task 83e97da7-d2b7-496e-8b06-a1d0abf8ba5e: moving from state init -> state preparing >Thread-15914636::DEBUG::2013-11-20 03:36:36,930::task::495::TaskManager.Task::(_debug) Task 8bc76c5c-7804-454e-9fce-87fc8ecb7d53: moving from state init -> state preparing >Thread-15914637::DEBUG::2013-11-20 03:36:36,931::task::495::TaskManager.Task::(_debug) Task 83e97da7-d2b7-496e-8b06-a1d0abf8ba5e: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000470161437988', 'lastCheck': 1384918588.2784729, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00102710723877', 'lastCheck': 1384918594.3160679, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000917196273804', 'lastCheck': 1384918587.701437, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00043511390686', 'lastCheck': 1384918593.585772, 'valid': True, 'code': 0}} >Thread-15914636::DEBUG::2013-11-20 03:36:36,932::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914637::DEBUG::2013-11-20 03:36:36,933::task::495::TaskManager.Task::(_debug) Task 83e97da7-d2b7-496e-8b06-a1d0abf8ba5e: moving from state preparing -> state finished >Thread-15914636::DEBUG::2013-11-20 03:36:36,934::task::495::TaskManager.Task::(_debug) Task 8bc76c5c-7804-454e-9fce-87fc8ecb7d53: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914637::DEBUG::2013-11-20 03:36:36,934::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914636::DEBUG::2013-11-20 03:36:36,935::task::495::TaskManager.Task::(_debug) Task 8bc76c5c-7804-454e-9fce-87fc8ecb7d53: moving from state preparing -> state finished >Thread-15914637::DEBUG::2013-11-20 03:36:36,936::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914636::DEBUG::2013-11-20 03:36:36,936::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914637::DEBUG::2013-11-20 03:36:36,937::task::495::TaskManager.Task::(_debug) Task 83e97da7-d2b7-496e-8b06-a1d0abf8ba5e: ref 0 aborting False >Thread-15914636::DEBUG::2013-11-20 03:36:36,938::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914637::INFO::2013-11-20 03:36:36,938::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000470161437988', 'lastCheck': 1384918588.2784729, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00102710723877', 'lastCheck': 1384918594.3160679, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000917196273804', 'lastCheck': 1384918587.701437, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00043511390686', 'lastCheck': 1384918593.585772, 'valid': True, 'code': 0}} >Thread-15914636::DEBUG::2013-11-20 03:36:36,939::task::495::TaskManager.Task::(_debug) Task 8bc76c5c-7804-454e-9fce-87fc8ecb7d53: ref 0 aborting False >Thread-15914636::INFO::2013-11-20 03:36:36,940::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914638::DEBUG::2013-11-20 03:36:36,952::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914638::INFO::2013-11-20 03:36:36,953::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914638::DEBUG::2013-11-20 03:36:36,953::task::495::TaskManager.Task::(_debug) Task 983ea87a-845c-4b89-90c8-226d1ff1dda9: moving from state init -> state preparing >Thread-15914638::DEBUG::2013-11-20 03:36:36,955::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1cd9d01f-2944-4a1d-ae07-eea804e78834`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914638::DEBUG::2013-11-20 03:36:36,956::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914638::DEBUG::2013-11-20 03:36:36,956::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914638::DEBUG::2013-11-20 03:36:36,957::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`1cd9d01f-2944-4a1d-ae07-eea804e78834`::Granted request >Thread-15914638::DEBUG::2013-11-20 03:36:36,957::task::495::TaskManager.Task::(_debug) Task 983ea87a-845c-4b89-90c8-226d1ff1dda9: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914638::DEBUG::2013-11-20 03:36:36,958::task::495::TaskManager.Task::(_debug) Task 983ea87a-845c-4b89-90c8-226d1ff1dda9: ref 1 aborting False >Thread-15914638::WARNING::2013-11-20 03:36:36,959::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914638::DEBUG::2013-11-20 03:36:36,960::task::495::TaskManager.Task::(_debug) Task 983ea87a-845c-4b89-90c8-226d1ff1dda9: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914638::DEBUG::2013-11-20 03:36:36,961::task::495::TaskManager.Task::(_debug) Task 983ea87a-845c-4b89-90c8-226d1ff1dda9: moving from state preparing -> state finished >Thread-15914638::DEBUG::2013-11-20 03:36:36,961::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914638::DEBUG::2013-11-20 03:36:36,962::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914638::DEBUG::2013-11-20 03:36:36,962::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914638::DEBUG::2013-11-20 03:36:36,963::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914638::DEBUG::2013-11-20 03:36:36,963::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914638::DEBUG::2013-11-20 03:36:36,963::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914638::DEBUG::2013-11-20 03:36:36,964::task::495::TaskManager.Task::(_debug) Task 983ea87a-845c-4b89-90c8-226d1ff1dda9: ref 0 aborting False >Thread-15914638::INFO::2013-11-20 03:36:36,964::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:36:37,165::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0597052 s, 17.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:36:39,177::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:36:39,601::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0601218 s, 17.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:36:41,612::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:36:42,446::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0492968 s, 20.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:36:44,456::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:36:44,863::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0625085 s, 16.4 MB/s\n'; <rc> = 0 >Thread-15913639::INFO::2013-11-20 03:36:45,766::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:36:45,766::task::495::TaskManager.Task::(_debug) Task cd10ae1b-429f-4f91-be60-51d6266bb7e1: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:36:45,768::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`9b4c1794-eb20-44f7-b05f-723e872fa38c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:36:45,769::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:36:45,770::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:36:45,770::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`9b4c1794-eb20-44f7-b05f-723e872fa38c`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:36:45,771::task::495::TaskManager.Task::(_debug) Task cd10ae1b-429f-4f91-be60-51d6266bb7e1: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:36:45,772::task::495::TaskManager.Task::(_debug) Task cd10ae1b-429f-4f91-be60-51d6266bb7e1: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:36:45,772::task::495::TaskManager.Task::(_debug) Task cd10ae1b-429f-4f91-be60-51d6266bb7e1: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:36:45,773::task::495::TaskManager.Task::(_debug) Task cd10ae1b-429f-4f91-be60-51d6266bb7e1: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:36:45,773::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:36:45,774::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:36:45,774::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:36:45,775::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:36:45,775::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:36:45,776::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:36:45,776::task::495::TaskManager.Task::(_debug) Task cd10ae1b-429f-4f91-be60-51d6266bb7e1: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:36:45,777::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:36:46,873::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914643::DEBUG::2013-11-20 03:36:48,724::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914643::INFO::2013-11-20 03:36:48,728::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914643::DEBUG::2013-11-20 03:36:48,728::task::495::TaskManager.Task::(_debug) Task 80ddf458-9ce7-4294-822d-0ed94993b3d4: moving from state init -> state preparing >Thread-15914643::DEBUG::2013-11-20 03:36:48,735::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914643::DEBUG::2013-11-20 03:36:48,736::task::495::TaskManager.Task::(_debug) Task 80ddf458-9ce7-4294-822d-0ed94993b3d4: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914643::DEBUG::2013-11-20 03:36:48,736::task::495::TaskManager.Task::(_debug) Task 80ddf458-9ce7-4294-822d-0ed94993b3d4: moving from state preparing -> state finished >Thread-15914643::DEBUG::2013-11-20 03:36:48,737::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914643::DEBUG::2013-11-20 03:36:48,738::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914643::DEBUG::2013-11-20 03:36:48,738::task::495::TaskManager.Task::(_debug) Task 80ddf458-9ce7-4294-822d-0ed94993b3d4: ref 0 aborting False >Thread-15914643::INFO::2013-11-20 03:36:48,739::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914644::DEBUG::2013-11-20 03:36:48,745::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914644::INFO::2013-11-20 03:36:48,746::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914644::DEBUG::2013-11-20 03:36:48,746::task::495::TaskManager.Task::(_debug) Task 4fb3236b-d43c-4e9f-887e-de513f72277d: moving from state init -> state preparing >Thread-15914644::DEBUG::2013-11-20 03:36:48,748::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`75bdb7fb-4c1a-41f7-8b8f-a0de00de8b67`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914644::DEBUG::2013-11-20 03:36:48,748::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914644::DEBUG::2013-11-20 03:36:48,749::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914644::DEBUG::2013-11-20 03:36:48,750::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`75bdb7fb-4c1a-41f7-8b8f-a0de00de8b67`::Granted request >Thread-15914644::DEBUG::2013-11-20 03:36:48,751::task::495::TaskManager.Task::(_debug) Task 4fb3236b-d43c-4e9f-887e-de513f72277d: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914644::DEBUG::2013-11-20 03:36:48,752::task::495::TaskManager.Task::(_debug) Task 4fb3236b-d43c-4e9f-887e-de513f72277d: ref 1 aborting False >Thread-15914644::WARNING::2013-11-20 03:36:48,754::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914644::DEBUG::2013-11-20 03:36:48,755::task::495::TaskManager.Task::(_debug) Task 4fb3236b-d43c-4e9f-887e-de513f72277d: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914644::DEBUG::2013-11-20 03:36:48,757::task::495::TaskManager.Task::(_debug) Task 4fb3236b-d43c-4e9f-887e-de513f72277d: moving from state preparing -> state finished >Thread-15914644::DEBUG::2013-11-20 03:36:48,757::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914644::DEBUG::2013-11-20 03:36:48,758::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914644::DEBUG::2013-11-20 03:36:48,759::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914644::DEBUG::2013-11-20 03:36:48,759::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914644::DEBUG::2013-11-20 03:36:48,760::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914644::DEBUG::2013-11-20 03:36:48,760::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914644::DEBUG::2013-11-20 03:36:48,761::task::495::TaskManager.Task::(_debug) Task 4fb3236b-d43c-4e9f-887e-de513f72277d: ref 0 aborting False >Thread-15914644::INFO::2013-11-20 03:36:48,761::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:36:48,943::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0453946 s, 22.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:36:50,952::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914646::INFO::2013-11-20 03:36:53,173::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914646::DEBUG::2013-11-20 03:36:53,175::task::495::TaskManager.Task::(_debug) Task 9e7a09b8-59a5-4f48-9cc6-3cc6cff3ee13: moving from state init -> state preparing >Thread-15914646::DEBUG::2013-11-20 03:36:53,175::task::495::TaskManager.Task::(_debug) Task 9e7a09b8-59a5-4f48-9cc6-3cc6cff3ee13: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000593185424805', 'lastCheck': 1384918608.726352, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0191969871521', 'lastCheck': 1384918604.3605211, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00306797027588', 'lastCheck': 1384918608.7259099, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000400066375732', 'lastCheck': 1384918603.5963161, 'valid': True, 'code': 0}} >Thread-15914646::DEBUG::2013-11-20 03:36:53,185::task::495::TaskManager.Task::(_debug) Task 9e7a09b8-59a5-4f48-9cc6-3cc6cff3ee13: moving from state preparing -> state finished >Thread-15914646::DEBUG::2013-11-20 03:36:53,186::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914646::DEBUG::2013-11-20 03:36:53,187::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914646::DEBUG::2013-11-20 03:36:53,187::task::495::TaskManager.Task::(_debug) Task 9e7a09b8-59a5-4f48-9cc6-3cc6cff3ee13: ref 0 aborting False >Thread-15914646::INFO::2013-11-20 03:36:53,188::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000593185424805', 'lastCheck': 1384918608.726352, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0191969871521', 'lastCheck': 1384918604.3605211, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00306797027588', 'lastCheck': 1384918608.7259099, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000400066375732', 'lastCheck': 1384918603.5963161, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:36:53,873::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.049969 s, 20.5 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:36:55,884::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:36:56,395::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.054387 s, 18.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:36:58,405::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-42838::DEBUG::2013-11-20 03:36:58,736::lvm::374::Storage.Misc.excCmd::(cmd) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \\"a%3600508b1001c7159da8b28c482e8a07c|3600c0ff0001520c7414f605001000000|3600c0ff000152111c050605001000000%\\", \\"r%.*%\\" ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 } backup { retain_min = 50 retain_days = 0 } " 6b892067-ac7e-4039-a7be-5b7387525b16' (cwd None) >Thread-15914649::DEBUG::2013-11-20 03:36:59,066::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914649::INFO::2013-11-20 03:36:59,075::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914649::DEBUG::2013-11-20 03:36:59,076::task::495::TaskManager.Task::(_debug) Task 7085851e-105d-4b34-92d5-af29e9cd4213: moving from state init -> state preparing >Thread-15914649::DEBUG::2013-11-20 03:36:59,077::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914649::DEBUG::2013-11-20 03:36:59,077::task::495::TaskManager.Task::(_debug) Task 7085851e-105d-4b34-92d5-af29e9cd4213: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914649::DEBUG::2013-11-20 03:36:59,078::task::495::TaskManager.Task::(_debug) Task 7085851e-105d-4b34-92d5-af29e9cd4213: moving from state preparing -> state finished >Thread-15914649::DEBUG::2013-11-20 03:36:59,078::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914649::DEBUG::2013-11-20 03:36:59,079::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914649::DEBUG::2013-11-20 03:36:59,079::task::495::TaskManager.Task::(_debug) Task 7085851e-105d-4b34-92d5-af29e9cd4213: ref 0 aborting False >Thread-15914649::INFO::2013-11-20 03:36:59,080::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914650::DEBUG::2013-11-20 03:36:59,084::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914650::INFO::2013-11-20 03:36:59,085::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914650::DEBUG::2013-11-20 03:36:59,086::task::495::TaskManager.Task::(_debug) Task aeaf0456-9009-4ea2-9083-d6044914a871: moving from state init -> state preparing >Thread-15914650::DEBUG::2013-11-20 03:36:59,087::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`7befe63a-499f-4876-9a7f-30d57a1620b3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914650::DEBUG::2013-11-20 03:36:59,088::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914650::DEBUG::2013-11-20 03:36:59,090::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914650::DEBUG::2013-11-20 03:36:59,090::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`7befe63a-499f-4876-9a7f-30d57a1620b3`::Granted request >Thread-15914650::DEBUG::2013-11-20 03:36:59,092::task::495::TaskManager.Task::(_debug) Task aeaf0456-9009-4ea2-9083-d6044914a871: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914650::DEBUG::2013-11-20 03:36:59,092::task::495::TaskManager.Task::(_debug) Task aeaf0456-9009-4ea2-9083-d6044914a871: ref 1 aborting False >Thread-15914650::WARNING::2013-11-20 03:36:59,103::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914650::DEBUG::2013-11-20 03:36:59,107::task::495::TaskManager.Task::(_debug) Task aeaf0456-9009-4ea2-9083-d6044914a871: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914650::DEBUG::2013-11-20 03:36:59,110::task::495::TaskManager.Task::(_debug) Task aeaf0456-9009-4ea2-9083-d6044914a871: moving from state preparing -> state finished >Thread-15914650::DEBUG::2013-11-20 03:36:59,111::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914650::DEBUG::2013-11-20 03:36:59,111::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914650::DEBUG::2013-11-20 03:36:59,112::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914650::DEBUG::2013-11-20 03:36:59,112::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914650::DEBUG::2013-11-20 03:36:59,113::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914650::DEBUG::2013-11-20 03:36:59,113::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914650::DEBUG::2013-11-20 03:36:59,114::task::495::TaskManager.Task::(_debug) Task aeaf0456-9009-4ea2-9083-d6044914a871: ref 0 aborting False >Thread-15914650::INFO::2013-11-20 03:36:59,114::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-42838::DEBUG::2013-11-20 03:36:59,458::lvm::374::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = ''; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:37:00,404::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.052462 s, 19.5 MB/s\n'; <rc> = 0 >Thread-15913658::INFO::2013-11-20 03:37:00,903::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:37:00,912::task::495::TaskManager.Task::(_debug) Task abcab17a-b786-483e-b67b-d42283e9b3fd: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:37:00,913::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`65b9040a-dc83-4be1-8178-d4d7ba58b477`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:37:00,914::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:37:00,915::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:37:00,915::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`65b9040a-dc83-4be1-8178-d4d7ba58b477`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:37:00,916::task::495::TaskManager.Task::(_debug) Task abcab17a-b786-483e-b67b-d42283e9b3fd: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:37:00,917::task::495::TaskManager.Task::(_debug) Task abcab17a-b786-483e-b67b-d42283e9b3fd: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:37:00,918::task::495::TaskManager.Task::(_debug) Task abcab17a-b786-483e-b67b-d42283e9b3fd: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:37:00,919::task::495::TaskManager.Task::(_debug) Task abcab17a-b786-483e-b67b-d42283e9b3fd: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:37:00,919::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:37:00,920::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:37:00,920::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:37:00,921::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:37:00,922::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:37:00,922::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:37:00,923::task::495::TaskManager.Task::(_debug) Task abcab17a-b786-483e-b67b-d42283e9b3fd: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:37:00,923::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:37:02,414::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:37:02,891::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.076222 s, 13.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:37:04,907::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:37:05,381::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0572546 s, 17.9 MB/s\n'; <rc> = 0 >Thread-15912392::INFO::2013-11-20 03:37:06,728::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:37:06,729::task::495::TaskManager.Task::(_debug) Task f98d9a90-f142-4ca6-acb6-273b45e00a5a: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:37:06,730::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`5c948f74-7a77-4d31-b34f-e3601c862641`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:37:06,731::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:37:06,731::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:37:06,732::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`5c948f74-7a77-4d31-b34f-e3601c862641`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:37:06,733::task::495::TaskManager.Task::(_debug) Task f98d9a90-f142-4ca6-acb6-273b45e00a5a: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:37:06,733::task::495::TaskManager.Task::(_debug) Task f98d9a90-f142-4ca6-acb6-273b45e00a5a: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:37:06,734::task::495::TaskManager.Task::(_debug) Task f98d9a90-f142-4ca6-acb6-273b45e00a5a: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:37:06,734::task::495::TaskManager.Task::(_debug) Task f98d9a90-f142-4ca6-acb6-273b45e00a5a: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:37:06,735::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:37:06,735::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:37:06,736::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:37:06,736::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:37:06,736::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:37:06,737::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:37:06,737::task::495::TaskManager.Task::(_debug) Task f98d9a90-f142-4ca6-acb6-273b45e00a5a: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:37:06,738::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:37:07,393::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:37:07,883::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0538173 s, 19.0 MB/s\n'; <rc> = 0 >Thread-15914654::DEBUG::2013-11-20 03:37:08,089::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15914655::INFO::2013-11-20 03:37:08,795::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914655::DEBUG::2013-11-20 03:37:08,797::task::495::TaskManager.Task::(_debug) Task 152c3875-80c7-4f66-964e-70d836bab38f: moving from state init -> state preparing >Thread-15914655::DEBUG::2013-11-20 03:37:08,798::task::495::TaskManager.Task::(_debug) Task 152c3875-80c7-4f66-964e-70d836bab38f: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000318050384521', 'lastCheck': 1384918619.462513, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00131320953369', 'lastCheck': 1384918624.7372329, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.165114164352', 'lastCheck': 1384918618.9037509, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000554084777832', 'lastCheck': 1384918623.7183621, 'valid': True, 'code': 0}} >Thread-15914655::DEBUG::2013-11-20 03:37:08,800::task::495::TaskManager.Task::(_debug) Task 152c3875-80c7-4f66-964e-70d836bab38f: moving from state preparing -> state finished >Thread-15914655::DEBUG::2013-11-20 03:37:08,801::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914655::DEBUG::2013-11-20 03:37:08,802::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914655::DEBUG::2013-11-20 03:37:08,803::task::495::TaskManager.Task::(_debug) Task 152c3875-80c7-4f66-964e-70d836bab38f: ref 0 aborting False >Thread-15914655::INFO::2013-11-20 03:37:08,803::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000318050384521', 'lastCheck': 1384918619.462513, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00131320953369', 'lastCheck': 1384918624.7372329, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.165114164352', 'lastCheck': 1384918618.9037509, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000554084777832', 'lastCheck': 1384918623.7183621, 'valid': True, 'code': 0}} >Thread-15913695::INFO::2013-11-20 03:37:09,173::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:37:09,174::task::495::TaskManager.Task::(_debug) Task 5ba21f86-0c90-4457-b743-6db305fde2c0: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:37:09,176::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b08ce65b-7ed9-44e8-81f4-b413e1b9d923`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:37:09,176::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:37:09,178::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:37:09,179::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`b08ce65b-7ed9-44e8-81f4-b413e1b9d923`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:37:09,180::task::495::TaskManager.Task::(_debug) Task 5ba21f86-0c90-4457-b743-6db305fde2c0: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:37:09,181::task::495::TaskManager.Task::(_debug) Task 5ba21f86-0c90-4457-b743-6db305fde2c0: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:37:09,182::task::495::TaskManager.Task::(_debug) Task 5ba21f86-0c90-4457-b743-6db305fde2c0: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:37:09,183::task::495::TaskManager.Task::(_debug) Task 5ba21f86-0c90-4457-b743-6db305fde2c0: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:37:09,183::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:37:09,184::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:37:09,185::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:37:09,185::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:37:09,186::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:37:09,186::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:37:09,187::task::495::TaskManager.Task::(_debug) Task 5ba21f86-0c90-4457-b743-6db305fde2c0: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:37:09,188::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15914654::DEBUG::2013-11-20 03:37:09,300::clientIF::1219::vds::(fenceNode) rc 0 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = On >Done > err >Thread-15914657::DEBUG::2013-11-20 03:37:09,320::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=off,secure=,options=power_wait=4 >lanplus=1) >Thread-15914659::DEBUG::2013-11-20 03:37:09,702::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914659::INFO::2013-11-20 03:37:09,704::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914659::DEBUG::2013-11-20 03:37:09,705::task::495::TaskManager.Task::(_debug) Task 0b7aa4fe-2649-4f18-9dd8-23f628d1cb4b: moving from state init -> state preparing >Thread-15914659::DEBUG::2013-11-20 03:37:09,705::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914659::DEBUG::2013-11-20 03:37:09,706::task::495::TaskManager.Task::(_debug) Task 0b7aa4fe-2649-4f18-9dd8-23f628d1cb4b: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914659::DEBUG::2013-11-20 03:37:09,706::task::495::TaskManager.Task::(_debug) Task 0b7aa4fe-2649-4f18-9dd8-23f628d1cb4b: moving from state preparing -> state finished >Thread-15914659::DEBUG::2013-11-20 03:37:09,707::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914659::DEBUG::2013-11-20 03:37:09,707::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914659::DEBUG::2013-11-20 03:37:09,708::task::495::TaskManager.Task::(_debug) Task 0b7aa4fe-2649-4f18-9dd8-23f628d1cb4b: ref 0 aborting False >Thread-15914659::INFO::2013-11-20 03:37:09,708::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914660::DEBUG::2013-11-20 03:37:09,716::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914660::INFO::2013-11-20 03:37:09,718::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914660::DEBUG::2013-11-20 03:37:09,719::task::495::TaskManager.Task::(_debug) Task 20dc5815-4eac-4b57-ad67-a73debe75dc7: moving from state init -> state preparing >Thread-15914660::DEBUG::2013-11-20 03:37:09,720::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`81b4194a-4ad0-4ba4-bbb4-d60bd3b26338`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914660::DEBUG::2013-11-20 03:37:09,721::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914660::DEBUG::2013-11-20 03:37:09,721::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914660::DEBUG::2013-11-20 03:37:09,727::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`81b4194a-4ad0-4ba4-bbb4-d60bd3b26338`::Granted request >Thread-15914660::DEBUG::2013-11-20 03:37:09,727::task::495::TaskManager.Task::(_debug) Task 20dc5815-4eac-4b57-ad67-a73debe75dc7: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914660::DEBUG::2013-11-20 03:37:09,728::task::495::TaskManager.Task::(_debug) Task 20dc5815-4eac-4b57-ad67-a73debe75dc7: ref 1 aborting False >Thread-15914660::WARNING::2013-11-20 03:37:09,729::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914660::DEBUG::2013-11-20 03:37:09,730::task::495::TaskManager.Task::(_debug) Task 20dc5815-4eac-4b57-ad67-a73debe75dc7: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914660::DEBUG::2013-11-20 03:37:09,730::task::495::TaskManager.Task::(_debug) Task 20dc5815-4eac-4b57-ad67-a73debe75dc7: moving from state preparing -> state finished >Thread-15914660::DEBUG::2013-11-20 03:37:09,736::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914660::DEBUG::2013-11-20 03:37:09,742::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914660::DEBUG::2013-11-20 03:37:09,747::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914660::DEBUG::2013-11-20 03:37:09,748::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914660::DEBUG::2013-11-20 03:37:09,749::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914660::DEBUG::2013-11-20 03:37:09,749::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914660::DEBUG::2013-11-20 03:37:09,750::task::495::TaskManager.Task::(_debug) Task 20dc5815-4eac-4b57-ad67-a73debe75dc7: ref 0 aborting False >Thread-15914660::INFO::2013-11-20 03:37:09,751::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:37:09,895::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:37:10,990::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0508171 s, 20.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:37:13,001::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:37:13,402::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0497847 s, 20.6 MB/s\n'; <rc> = 0 >Thread-15914658::DEBUG::2013-11-20 03:37:14,627::clientIF::1179::vds::(waitForPid) rc 0 inp agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=off >passwd=XXXX >power_wait=4 >lanplus=1 out Powering off machine @ IPMI:192.168.100.119...Done > err >Thread-15914662::DEBUG::2013-11-20 03:37:14,883::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Dummy-15912651::DEBUG::2013-11-20 03:37:15,413::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914662::DEBUG::2013-11-20 03:37:15,616::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Thread-15914664::DEBUG::2013-11-20 03:37:15,623::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Dummy-15912651::DEBUG::2013-11-20 03:37:16,442::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.424345 s, 2.4 MB/s\n'; <rc> = 0 >Thread-15914664::DEBUG::2013-11-20 03:37:16,575::clientIF::1219::vds::(fenceNode) rc 2 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = Off >Done > err >Dummy-15912651::DEBUG::2013-11-20 03:37:18,453::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914435::INFO::2013-11-20 03:37:19,239::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=143cb26d-de08-4d30-89cf-aee10245b66a volUUID=d7028f7c-0551-4bbe-a907-6f804cc9bd54) >Thread-15914435::DEBUG::2013-11-20 03:37:19,240::task::495::TaskManager.Task::(_debug) Task 3c6450c9-f938-49af-a7a9-e1bf150559e5: moving from state init -> state preparing >Thread-15914435::DEBUG::2013-11-20 03:37:19,247::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`4313e9fe-ff8a-41ea-ae09-9a867515d4b6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914435::DEBUG::2013-11-20 03:37:19,248::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914435::DEBUG::2013-11-20 03:37:19,249::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914435::DEBUG::2013-11-20 03:37:19,249::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`4313e9fe-ff8a-41ea-ae09-9a867515d4b6`::Granted request >Thread-15914435::DEBUG::2013-11-20 03:37:19,250::task::495::TaskManager.Task::(_debug) Task 3c6450c9-f938-49af-a7a9-e1bf150559e5: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914435::DEBUG::2013-11-20 03:37:19,251::task::495::TaskManager.Task::(_debug) Task 3c6450c9-f938-49af-a7a9-e1bf150559e5: ref 1 aborting False >Thread-15914435::DEBUG::2013-11-20 03:37:19,252::task::495::TaskManager.Task::(_debug) Task 3c6450c9-f938-49af-a7a9-e1bf150559e5: finished: {'truesize': '56908316672', 'apparentsize': '56908316672'} >Thread-15914435::DEBUG::2013-11-20 03:37:19,253::task::495::TaskManager.Task::(_debug) Task 3c6450c9-f938-49af-a7a9-e1bf150559e5: moving from state preparing -> state finished >Thread-15914435::DEBUG::2013-11-20 03:37:19,253::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914435::DEBUG::2013-11-20 03:37:19,254::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914435::DEBUG::2013-11-20 03:37:19,254::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914435::DEBUG::2013-11-20 03:37:19,254::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914435::DEBUG::2013-11-20 03:37:19,255::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914435::DEBUG::2013-11-20 03:37:19,255::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914435::DEBUG::2013-11-20 03:37:19,256::task::495::TaskManager.Task::(_debug) Task 3c6450c9-f938-49af-a7a9-e1bf150559e5: ref 0 aborting False >Thread-15914435::INFO::2013-11-20 03:37:19,256::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '56908316672', 'apparentsize': '56908316672'} >Thread-15914666::DEBUG::2013-11-20 03:37:20,642::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914666::INFO::2013-11-20 03:37:20,643::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914666::DEBUG::2013-11-20 03:37:20,643::task::495::TaskManager.Task::(_debug) Task f92ea81e-f4d5-4142-a973-909ff81a1188: moving from state init -> state preparing >Thread-15914666::DEBUG::2013-11-20 03:37:20,644::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914666::DEBUG::2013-11-20 03:37:20,644::task::495::TaskManager.Task::(_debug) Task f92ea81e-f4d5-4142-a973-909ff81a1188: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914666::DEBUG::2013-11-20 03:37:20,644::task::495::TaskManager.Task::(_debug) Task f92ea81e-f4d5-4142-a973-909ff81a1188: moving from state preparing -> state finished >Thread-15914666::DEBUG::2013-11-20 03:37:20,645::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914666::DEBUG::2013-11-20 03:37:20,645::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914666::DEBUG::2013-11-20 03:37:20,646::task::495::TaskManager.Task::(_debug) Task f92ea81e-f4d5-4142-a973-909ff81a1188: ref 0 aborting False >Thread-15914666::INFO::2013-11-20 03:37:20,646::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914667::DEBUG::2013-11-20 03:37:20,650::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914667::INFO::2013-11-20 03:37:20,651::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914667::DEBUG::2013-11-20 03:37:20,652::task::495::TaskManager.Task::(_debug) Task 2b6a1bab-f3b6-405c-8d00-efd0c062e2c2: moving from state init -> state preparing >Thread-15914667::DEBUG::2013-11-20 03:37:20,655::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`486502a0-dc9f-470c-9404-7ade9acde5e3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914667::DEBUG::2013-11-20 03:37:20,656::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914667::DEBUG::2013-11-20 03:37:20,657::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914667::DEBUG::2013-11-20 03:37:20,657::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`486502a0-dc9f-470c-9404-7ade9acde5e3`::Granted request >Thread-15914667::DEBUG::2013-11-20 03:37:20,658::task::495::TaskManager.Task::(_debug) Task 2b6a1bab-f3b6-405c-8d00-efd0c062e2c2: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914667::DEBUG::2013-11-20 03:37:20,658::task::495::TaskManager.Task::(_debug) Task 2b6a1bab-f3b6-405c-8d00-efd0c062e2c2: ref 1 aborting False >Thread-15914667::WARNING::2013-11-20 03:37:20,660::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914667::DEBUG::2013-11-20 03:37:20,661::task::495::TaskManager.Task::(_debug) Task 2b6a1bab-f3b6-405c-8d00-efd0c062e2c2: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914667::DEBUG::2013-11-20 03:37:20,662::task::495::TaskManager.Task::(_debug) Task 2b6a1bab-f3b6-405c-8d00-efd0c062e2c2: moving from state preparing -> state finished >Thread-15914667::DEBUG::2013-11-20 03:37:20,663::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914667::DEBUG::2013-11-20 03:37:20,663::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914667::DEBUG::2013-11-20 03:37:20,664::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914667::DEBUG::2013-11-20 03:37:20,664::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914667::DEBUG::2013-11-20 03:37:20,665::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914667::DEBUG::2013-11-20 03:37:20,665::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914667::DEBUG::2013-11-20 03:37:20,666::task::495::TaskManager.Task::(_debug) Task 2b6a1bab-f3b6-405c-8d00-efd0c062e2c2: ref 0 aborting False >Thread-15914667::INFO::2013-11-20 03:37:20,666::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:37:21,183::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0486728 s, 21.0 MB/s\n'; <rc> = 0 >Thread-15914453::INFO::2013-11-20 03:37:22,392::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=5f930ef3-763f-4294-a62a-6d52df220f8c volUUID=b9cffdbe-152e-41ce-ae5d-08db7c08454e) >Thread-15914453::DEBUG::2013-11-20 03:37:22,393::task::495::TaskManager.Task::(_debug) Task 6722cfd3-d347-4538-bc88-363051a9b2f5: moving from state init -> state preparing >Thread-15914453::DEBUG::2013-11-20 03:37:22,394::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`6c7fc913-1c59-4ade-a44c-487a4fcb3b1e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914453::DEBUG::2013-11-20 03:37:22,395::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914453::DEBUG::2013-11-20 03:37:22,395::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914453::DEBUG::2013-11-20 03:37:22,396::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`6c7fc913-1c59-4ade-a44c-487a4fcb3b1e`::Granted request >Thread-15914453::DEBUG::2013-11-20 03:37:22,396::task::495::TaskManager.Task::(_debug) Task 6722cfd3-d347-4538-bc88-363051a9b2f5: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914453::DEBUG::2013-11-20 03:37:22,397::task::495::TaskManager.Task::(_debug) Task 6722cfd3-d347-4538-bc88-363051a9b2f5: ref 1 aborting False >Thread-15914453::DEBUG::2013-11-20 03:37:22,397::task::495::TaskManager.Task::(_debug) Task 6722cfd3-d347-4538-bc88-363051a9b2f5: finished: {'truesize': '53821308928', 'apparentsize': '53821308928'} >Thread-15914453::DEBUG::2013-11-20 03:37:22,397::task::495::TaskManager.Task::(_debug) Task 6722cfd3-d347-4538-bc88-363051a9b2f5: moving from state preparing -> state finished >Thread-15914453::DEBUG::2013-11-20 03:37:22,398::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914453::DEBUG::2013-11-20 03:37:22,398::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914453::DEBUG::2013-11-20 03:37:22,399::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914453::DEBUG::2013-11-20 03:37:22,399::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914453::DEBUG::2013-11-20 03:37:22,399::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914453::DEBUG::2013-11-20 03:37:22,400::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914453::DEBUG::2013-11-20 03:37:22,400::task::495::TaskManager.Task::(_debug) Task 6722cfd3-d347-4538-bc88-363051a9b2f5: ref 0 aborting False >Thread-15914453::INFO::2013-11-20 03:37:22,400::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '53821308928', 'apparentsize': '53821308928'} >Dummy-15912651::DEBUG::2013-11-20 03:37:23,194::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:37:24,161::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.052298 s, 19.6 MB/s\n'; <rc> = 0 >Thread-15914669::INFO::2013-11-20 03:37:24,340::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914669::DEBUG::2013-11-20 03:37:24,341::task::495::TaskManager.Task::(_debug) Task 09830d34-887d-402c-aaa0-c7e4b62be6c6: moving from state init -> state preparing >Thread-15914669::DEBUG::2013-11-20 03:37:24,342::task::495::TaskManager.Task::(_debug) Task 09830d34-887d-402c-aaa0-c7e4b62be6c6: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00108313560486', 'lastCheck': 1384918640.4566481, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0227658748627', 'lastCheck': 1384918634.7731681, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00116491317749', 'lastCheck': 1384918640.4568701, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000399827957153', 'lastCheck': 1384918633.726419, 'valid': True, 'code': 0}} >Thread-15914669::DEBUG::2013-11-20 03:37:24,342::task::495::TaskManager.Task::(_debug) Task 09830d34-887d-402c-aaa0-c7e4b62be6c6: moving from state preparing -> state finished >Thread-15914669::DEBUG::2013-11-20 03:37:24,343::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914669::DEBUG::2013-11-20 03:37:24,343::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914669::DEBUG::2013-11-20 03:37:24,344::task::495::TaskManager.Task::(_debug) Task 09830d34-887d-402c-aaa0-c7e4b62be6c6: ref 0 aborting False >Thread-15914669::INFO::2013-11-20 03:37:24,344::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00108313560486', 'lastCheck': 1384918640.4566481, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.0227658748627', 'lastCheck': 1384918634.7731681, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00116491317749', 'lastCheck': 1384918640.4568701, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000399827957153', 'lastCheck': 1384918633.726419, 'valid': True, 'code': 0}} >Thread-15914519::INFO::2013-11-20 03:37:25,450::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=5af747d8-95b1-4b7e-af53-cb77e4fcc896 volUUID=8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f) >Thread-15914519::DEBUG::2013-11-20 03:37:25,451::task::495::TaskManager.Task::(_debug) Task 1f76def7-e4f1-4024-9cb0-f7ce8ec3028d: moving from state init -> state preparing >Thread-15914519::DEBUG::2013-11-20 03:37:25,454::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`3b94cb75-0024-4790-9165-540e78890c29`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914519::DEBUG::2013-11-20 03:37:25,455::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914519::DEBUG::2013-11-20 03:37:25,456::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914519::DEBUG::2013-11-20 03:37:25,457::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`3b94cb75-0024-4790-9165-540e78890c29`::Granted request >Thread-15914519::DEBUG::2013-11-20 03:37:25,467::task::495::TaskManager.Task::(_debug) Task 1f76def7-e4f1-4024-9cb0-f7ce8ec3028d: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914519::DEBUG::2013-11-20 03:37:25,468::task::495::TaskManager.Task::(_debug) Task 1f76def7-e4f1-4024-9cb0-f7ce8ec3028d: ref 1 aborting False >Thread-15914519::DEBUG::2013-11-20 03:37:25,469::task::495::TaskManager.Task::(_debug) Task 1f76def7-e4f1-4024-9cb0-f7ce8ec3028d: finished: {'truesize': '86033563648', 'apparentsize': '86033563648'} >Thread-15914519::DEBUG::2013-11-20 03:37:25,470::task::495::TaskManager.Task::(_debug) Task 1f76def7-e4f1-4024-9cb0-f7ce8ec3028d: moving from state preparing -> state finished >Thread-15914519::DEBUG::2013-11-20 03:37:25,471::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914519::DEBUG::2013-11-20 03:37:25,471::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914519::DEBUG::2013-11-20 03:37:25,472::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914519::DEBUG::2013-11-20 03:37:25,473::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914519::DEBUG::2013-11-20 03:37:25,479::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914519::DEBUG::2013-11-20 03:37:25,480::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914519::DEBUG::2013-11-20 03:37:25,481::task::495::TaskManager.Task::(_debug) Task 1f76def7-e4f1-4024-9cb0-f7ce8ec3028d: ref 0 aborting False >Thread-15914519::INFO::2013-11-20 03:37:25,483::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '86033563648', 'apparentsize': '86033563648'} >Dummy-15912651::DEBUG::2013-11-20 03:37:26,172::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:37:26,611::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0855587 s, 12.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:37:28,635::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:37:29,554::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0880683 s, 11.6 MB/s\n'; <rc> = 0 >Thread-15913518::INFO::2013-11-20 03:37:30,569::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:37:30,570::task::495::TaskManager.Task::(_debug) Task 3ac35c31-8cd3-4928-a236-18acf3646bd6: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:37:30,571::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`6a33f83b-9887-4a77-b872-0a50e235e29f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:37:30,571::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:37:30,572::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:37:30,572::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`6a33f83b-9887-4a77-b872-0a50e235e29f`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:37:30,573::task::495::TaskManager.Task::(_debug) Task 3ac35c31-8cd3-4928-a236-18acf3646bd6: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:37:30,573::task::495::TaskManager.Task::(_debug) Task 3ac35c31-8cd3-4928-a236-18acf3646bd6: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:37:30,574::task::495::TaskManager.Task::(_debug) Task 3ac35c31-8cd3-4928-a236-18acf3646bd6: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:37:30,574::task::495::TaskManager.Task::(_debug) Task 3ac35c31-8cd3-4928-a236-18acf3646bd6: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:37:30,575::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:37:30,575::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:37:30,576::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:37:30,576::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:37:30,576::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:37:30,577::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:37:30,577::task::495::TaskManager.Task::(_debug) Task 3ac35c31-8cd3-4928-a236-18acf3646bd6: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:37:30,578::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:37:30,578::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:37:30,579::task::495::TaskManager.Task::(_debug) Task adee1d79-45af-48b6-8385-608906be0e60: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:37:30,580::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c0b92e6d-b6be-4d6e-b746-d4cbf83afb8f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:37:30,580::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:37:30,581::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:37:30,581::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`c0b92e6d-b6be-4d6e-b746-d4cbf83afb8f`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:37:30,582::task::495::TaskManager.Task::(_debug) Task adee1d79-45af-48b6-8385-608906be0e60: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:37:30,582::task::495::TaskManager.Task::(_debug) Task adee1d79-45af-48b6-8385-608906be0e60: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:37:30,582::task::495::TaskManager.Task::(_debug) Task adee1d79-45af-48b6-8385-608906be0e60: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:37:30,583::task::495::TaskManager.Task::(_debug) Task adee1d79-45af-48b6-8385-608906be0e60: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:37:30,583::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:37:30,583::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:37:30,584::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:37:30,584::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:37:30,584::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:37:30,585::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:37:30,585::task::495::TaskManager.Task::(_debug) Task adee1d79-45af-48b6-8385-608906be0e60: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:37:30,586::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15914672::DEBUG::2013-11-20 03:37:30,935::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914672::INFO::2013-11-20 03:37:30,936::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914672::DEBUG::2013-11-20 03:37:30,937::task::495::TaskManager.Task::(_debug) Task 6800d690-8ce3-4000-9ca7-227f8e682a9e: moving from state init -> state preparing >Thread-15914672::DEBUG::2013-11-20 03:37:30,938::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914672::DEBUG::2013-11-20 03:37:30,938::task::495::TaskManager.Task::(_debug) Task 6800d690-8ce3-4000-9ca7-227f8e682a9e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914672::DEBUG::2013-11-20 03:37:30,939::task::495::TaskManager.Task::(_debug) Task 6800d690-8ce3-4000-9ca7-227f8e682a9e: moving from state preparing -> state finished >Thread-15914672::DEBUG::2013-11-20 03:37:30,941::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914672::DEBUG::2013-11-20 03:37:30,941::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914672::DEBUG::2013-11-20 03:37:30,942::task::495::TaskManager.Task::(_debug) Task 6800d690-8ce3-4000-9ca7-227f8e682a9e: ref 0 aborting False >Thread-15914672::INFO::2013-11-20 03:37:30,942::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914674::DEBUG::2013-11-20 03:37:30,946::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914674::INFO::2013-11-20 03:37:30,946::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914674::DEBUG::2013-11-20 03:37:30,947::task::495::TaskManager.Task::(_debug) Task e7fb3ea1-acd2-4918-9042-59f699c7e69b: moving from state init -> state preparing >Thread-15914674::DEBUG::2013-11-20 03:37:30,950::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a01d1ee2-2d06-4ac1-a191-a3b698be2ad6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914674::DEBUG::2013-11-20 03:37:30,950::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914674::DEBUG::2013-11-20 03:37:30,951::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914674::DEBUG::2013-11-20 03:37:30,951::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`a01d1ee2-2d06-4ac1-a191-a3b698be2ad6`::Granted request >Thread-15914674::DEBUG::2013-11-20 03:37:30,951::task::495::TaskManager.Task::(_debug) Task e7fb3ea1-acd2-4918-9042-59f699c7e69b: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914674::DEBUG::2013-11-20 03:37:30,952::task::495::TaskManager.Task::(_debug) Task e7fb3ea1-acd2-4918-9042-59f699c7e69b: ref 1 aborting False >Thread-15914674::WARNING::2013-11-20 03:37:30,953::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914674::DEBUG::2013-11-20 03:37:30,954::task::495::TaskManager.Task::(_debug) Task e7fb3ea1-acd2-4918-9042-59f699c7e69b: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914674::DEBUG::2013-11-20 03:37:30,954::task::495::TaskManager.Task::(_debug) Task e7fb3ea1-acd2-4918-9042-59f699c7e69b: moving from state preparing -> state finished >Thread-15914674::DEBUG::2013-11-20 03:37:30,955::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914674::DEBUG::2013-11-20 03:37:30,955::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914674::DEBUG::2013-11-20 03:37:30,955::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914674::DEBUG::2013-11-20 03:37:30,956::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914674::DEBUG::2013-11-20 03:37:30,956::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914674::DEBUG::2013-11-20 03:37:30,956::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914674::DEBUG::2013-11-20 03:37:30,957::task::495::TaskManager.Task::(_debug) Task e7fb3ea1-acd2-4918-9042-59f699c7e69b: ref 0 aborting False >Thread-15914674::INFO::2013-11-20 03:37:30,957::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:37:31,564::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:37:31,987::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0525519 s, 19.5 MB/s\n'; <rc> = 0 >Thread-15913546::INFO::2013-11-20 03:37:32,421::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:37:32,423::task::495::TaskManager.Task::(_debug) Task 8c14a2a8-ec58-4dd7-ad2c-f7bf5f2cdaf7: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:37:32,426::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`25aabcbc-e0aa-44f0-9b38-11237885da59`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:37:32,427::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:37:32,429::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:37:32,429::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`25aabcbc-e0aa-44f0-9b38-11237885da59`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:37:32,430::task::495::TaskManager.Task::(_debug) Task 8c14a2a8-ec58-4dd7-ad2c-f7bf5f2cdaf7: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:37:32,431::task::495::TaskManager.Task::(_debug) Task 8c14a2a8-ec58-4dd7-ad2c-f7bf5f2cdaf7: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:37:32,431::task::495::TaskManager.Task::(_debug) Task 8c14a2a8-ec58-4dd7-ad2c-f7bf5f2cdaf7: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:37:32,432::task::495::TaskManager.Task::(_debug) Task 8c14a2a8-ec58-4dd7-ad2c-f7bf5f2cdaf7: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:37:32,433::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:37:32,434::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:37:32,436::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:37:32,437::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:37:32,438::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:37:32,439::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:37:32,440::task::495::TaskManager.Task::(_debug) Task 8c14a2a8-ec58-4dd7-ad2c-f7bf5f2cdaf7: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:37:32,441::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:37:32,443::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:37:32,444::task::495::TaskManager.Task::(_debug) Task b3d8503f-7f7f-4191-ae99-9f65669affd8: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:37:32,445::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`4b602ec5-640e-45e2-9316-8d944d676b95`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:37:32,446::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:37:32,447::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:37:32,447::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`4b602ec5-640e-45e2-9316-8d944d676b95`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:37:32,448::task::495::TaskManager.Task::(_debug) Task b3d8503f-7f7f-4191-ae99-9f65669affd8: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:37:32,449::task::495::TaskManager.Task::(_debug) Task b3d8503f-7f7f-4191-ae99-9f65669affd8: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:37:32,449::task::495::TaskManager.Task::(_debug) Task b3d8503f-7f7f-4191-ae99-9f65669affd8: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:37:32,450::task::495::TaskManager.Task::(_debug) Task b3d8503f-7f7f-4191-ae99-9f65669affd8: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:37:32,451::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:37:32,452::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:37:32,452::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:37:32,453::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:37:32,453::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:37:32,454::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:37:32,454::task::495::TaskManager.Task::(_debug) Task b3d8503f-7f7f-4191-ae99-9f65669affd8: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:37:32,455::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:37:32,456::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:37:32,456::task::495::TaskManager.Task::(_debug) Task 1381adb3-0b93-42ef-8d72-f2c0255c6f5f: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:37:32,457::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`5962d5ab-4214-4504-b2b4-ba8ddfb47350`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:37:32,457::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:37:32,458::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:37:32,458::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`5962d5ab-4214-4504-b2b4-ba8ddfb47350`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:37:32,459::task::495::TaskManager.Task::(_debug) Task 1381adb3-0b93-42ef-8d72-f2c0255c6f5f: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:37:32,460::task::495::TaskManager.Task::(_debug) Task 1381adb3-0b93-42ef-8d72-f2c0255c6f5f: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:37:32,461::task::495::TaskManager.Task::(_debug) Task 1381adb3-0b93-42ef-8d72-f2c0255c6f5f: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:37:32,461::task::495::TaskManager.Task::(_debug) Task 1381adb3-0b93-42ef-8d72-f2c0255c6f5f: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:37:32,462::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:37:32,462::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:37:32,462::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:37:32,463::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:37:32,464::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:37:32,464::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:37:32,465::task::495::TaskManager.Task::(_debug) Task 1381adb3-0b93-42ef-8d72-f2c0255c6f5f: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:37:32,465::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:37:32,466::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:37:32,467::task::495::TaskManager.Task::(_debug) Task e729767f-9d61-4fd8-98c3-13cd0a44bafd: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:37:32,468::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`7308e1da-dd1d-4130-a479-5c7486f9dce0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:37:32,469::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:37:32,470::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:37:32,471::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`7308e1da-dd1d-4130-a479-5c7486f9dce0`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:37:32,474::task::495::TaskManager.Task::(_debug) Task e729767f-9d61-4fd8-98c3-13cd0a44bafd: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:37:32,475::task::495::TaskManager.Task::(_debug) Task e729767f-9d61-4fd8-98c3-13cd0a44bafd: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:37:32,476::task::495::TaskManager.Task::(_debug) Task e729767f-9d61-4fd8-98c3-13cd0a44bafd: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:37:32,477::task::495::TaskManager.Task::(_debug) Task e729767f-9d61-4fd8-98c3-13cd0a44bafd: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:37:32,478::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:37:32,479::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:37:32,480::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:37:32,482::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:37:32,483::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:37:32,485::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:37:32,487::task::495::TaskManager.Task::(_debug) Task e729767f-9d61-4fd8-98c3-13cd0a44bafd: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:37:32,487::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:37:33,997::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:37:34,396::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0629641 s, 16.3 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:37:36,406::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:37:39,378::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0467262 s, 21.9 MB/s\n'; <rc> = 0 >Thread-15914677::INFO::2013-11-20 03:37:40,126::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914677::DEBUG::2013-11-20 03:37:40,127::task::495::TaskManager.Task::(_debug) Task 97856968-fcde-41f3-b7ae-69970386484b: moving from state init -> state preparing >Thread-15914677::DEBUG::2013-11-20 03:37:40,127::task::495::TaskManager.Task::(_debug) Task 97856968-fcde-41f3-b7ae-69970386484b: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000234127044678', 'lastCheck': 1384918650.4769681, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000960111618042', 'lastCheck': 1384918655.4306259, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000908136367798', 'lastCheck': 1384918650.477262, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000332117080688', 'lastCheck': 1384918654.448143, 'valid': True, 'code': 0}} >Thread-15914677::DEBUG::2013-11-20 03:37:40,183::task::495::TaskManager.Task::(_debug) Task 97856968-fcde-41f3-b7ae-69970386484b: moving from state preparing -> state finished >Thread-15914677::DEBUG::2013-11-20 03:37:40,184::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914677::DEBUG::2013-11-20 03:37:40,184::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914677::DEBUG::2013-11-20 03:37:40,185::task::495::TaskManager.Task::(_debug) Task 97856968-fcde-41f3-b7ae-69970386484b: ref 0 aborting False >Thread-15914677::INFO::2013-11-20 03:37:40,185::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000234127044678', 'lastCheck': 1384918650.4769681, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000960111618042', 'lastCheck': 1384918655.4306259, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.000908136367798', 'lastCheck': 1384918650.477262, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000332117080688', 'lastCheck': 1384918654.448143, 'valid': True, 'code': 0}} >Thread-15914679::DEBUG::2013-11-20 03:37:41,254::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914679::INFO::2013-11-20 03:37:41,255::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914679::DEBUG::2013-11-20 03:37:41,256::task::495::TaskManager.Task::(_debug) Task aee4e481-04b9-41ea-97fc-5ecd4d7bae4e: moving from state init -> state preparing >Thread-15914679::DEBUG::2013-11-20 03:37:41,256::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914679::DEBUG::2013-11-20 03:37:41,257::task::495::TaskManager.Task::(_debug) Task aee4e481-04b9-41ea-97fc-5ecd4d7bae4e: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914679::DEBUG::2013-11-20 03:37:41,258::task::495::TaskManager.Task::(_debug) Task aee4e481-04b9-41ea-97fc-5ecd4d7bae4e: moving from state preparing -> state finished >Thread-15914679::DEBUG::2013-11-20 03:37:41,258::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914679::DEBUG::2013-11-20 03:37:41,259::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914679::DEBUG::2013-11-20 03:37:41,259::task::495::TaskManager.Task::(_debug) Task aee4e481-04b9-41ea-97fc-5ecd4d7bae4e: ref 0 aborting False >Thread-15914679::INFO::2013-11-20 03:37:41,260::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914680::DEBUG::2013-11-20 03:37:41,268::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914680::INFO::2013-11-20 03:37:41,269::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914680::DEBUG::2013-11-20 03:37:41,270::task::495::TaskManager.Task::(_debug) Task a1e6a3b9-0478-4d07-ad24-b30f72ae5e1b: moving from state init -> state preparing >Thread-15914680::DEBUG::2013-11-20 03:37:41,276::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`21262ab0-2d5a-46d2-a618-37f288b1f950`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914680::DEBUG::2013-11-20 03:37:41,277::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914680::DEBUG::2013-11-20 03:37:41,278::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914680::DEBUG::2013-11-20 03:37:41,278::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`21262ab0-2d5a-46d2-a618-37f288b1f950`::Granted request >Thread-15914680::DEBUG::2013-11-20 03:37:41,279::task::495::TaskManager.Task::(_debug) Task a1e6a3b9-0478-4d07-ad24-b30f72ae5e1b: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914680::DEBUG::2013-11-20 03:37:41,279::task::495::TaskManager.Task::(_debug) Task a1e6a3b9-0478-4d07-ad24-b30f72ae5e1b: ref 1 aborting False >Thread-15914680::WARNING::2013-11-20 03:37:41,281::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914680::DEBUG::2013-11-20 03:37:41,287::task::495::TaskManager.Task::(_debug) Task a1e6a3b9-0478-4d07-ad24-b30f72ae5e1b: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914680::DEBUG::2013-11-20 03:37:41,288::task::495::TaskManager.Task::(_debug) Task a1e6a3b9-0478-4d07-ad24-b30f72ae5e1b: moving from state preparing -> state finished >Thread-15914680::DEBUG::2013-11-20 03:37:41,289::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914680::DEBUG::2013-11-20 03:37:41,289::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914680::DEBUG::2013-11-20 03:37:41,290::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914680::DEBUG::2013-11-20 03:37:41,290::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914680::DEBUG::2013-11-20 03:37:41,291::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914680::DEBUG::2013-11-20 03:37:41,291::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914680::DEBUG::2013-11-20 03:37:41,292::task::495::TaskManager.Task::(_debug) Task a1e6a3b9-0478-4d07-ad24-b30f72ae5e1b: ref 0 aborting False >Thread-15914680::INFO::2013-11-20 03:37:41,292::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:37:41,388::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:37:43,507::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.048719 s, 21.0 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:37:45,517::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15913639::INFO::2013-11-20 03:37:46,832::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=6f97dfa5-ee35-4f37-8495-1e6b3136b92c volUUID=5eaa8dde-10d8-45b5-9e10-5417c297c4be) >Thread-15913639::DEBUG::2013-11-20 03:37:49,517::task::495::TaskManager.Task::(_debug) Task 12c492b5-0277-47f1-b735-fe824ca55e12: moving from state init -> state preparing >Thread-15913639::DEBUG::2013-11-20 03:37:49,569::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`edf08044-816b-444b-8162-dd003adbd497`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913639::DEBUG::2013-11-20 03:37:49,586::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913639::DEBUG::2013-11-20 03:37:49,589::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913639::DEBUG::2013-11-20 03:37:49,589::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`edf08044-816b-444b-8162-dd003adbd497`::Granted request >Thread-15913639::DEBUG::2013-11-20 03:37:49,591::task::495::TaskManager.Task::(_debug) Task 12c492b5-0277-47f1-b735-fe824ca55e12: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913639::DEBUG::2013-11-20 03:37:49,591::task::495::TaskManager.Task::(_debug) Task 12c492b5-0277-47f1-b735-fe824ca55e12: ref 1 aborting False >Thread-15913639::DEBUG::2013-11-20 03:37:49,592::task::495::TaskManager.Task::(_debug) Task 12c492b5-0277-47f1-b735-fe824ca55e12: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913639::DEBUG::2013-11-20 03:37:49,593::task::495::TaskManager.Task::(_debug) Task 12c492b5-0277-47f1-b735-fe824ca55e12: moving from state preparing -> state finished >Thread-15913639::DEBUG::2013-11-20 03:37:49,593::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913639::DEBUG::2013-11-20 03:37:49,594::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913639::DEBUG::2013-11-20 03:37:49,595::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913639::DEBUG::2013-11-20 03:37:49,596::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913639::DEBUG::2013-11-20 03:37:49,597::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913639::DEBUG::2013-11-20 03:37:49,597::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913639::DEBUG::2013-11-20 03:37:49,598::task::495::TaskManager.Task::(_debug) Task 12c492b5-0277-47f1-b735-fe824ca55e12: ref 0 aborting False >Thread-15913639::INFO::2013-11-20 03:37:49,598::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:37:49,763::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0584681 s, 17.5 MB/s\n'; <rc> = 0 >Thread-15914683::DEBUG::2013-11-20 03:37:51,712::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914683::INFO::2013-11-20 03:37:51,712::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914683::DEBUG::2013-11-20 03:37:51,713::task::495::TaskManager.Task::(_debug) Task 0ab6d987-1007-41ed-aac9-d0573aee3fab: moving from state init -> state preparing >Thread-15914683::DEBUG::2013-11-20 03:37:51,713::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914683::DEBUG::2013-11-20 03:37:51,714::task::495::TaskManager.Task::(_debug) Task 0ab6d987-1007-41ed-aac9-d0573aee3fab: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914683::DEBUG::2013-11-20 03:37:51,714::task::495::TaskManager.Task::(_debug) Task 0ab6d987-1007-41ed-aac9-d0573aee3fab: moving from state preparing -> state finished >Thread-15914683::DEBUG::2013-11-20 03:37:51,714::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914683::DEBUG::2013-11-20 03:37:51,715::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914683::DEBUG::2013-11-20 03:37:51,715::task::495::TaskManager.Task::(_debug) Task 0ab6d987-1007-41ed-aac9-d0573aee3fab: ref 0 aborting False >Thread-15914683::INFO::2013-11-20 03:37:51,715::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914684::DEBUG::2013-11-20 03:37:51,722::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914684::INFO::2013-11-20 03:37:51,723::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914684::DEBUG::2013-11-20 03:37:51,723::task::495::TaskManager.Task::(_debug) Task e5d8a42b-543e-40fc-9739-32136decf7c0: moving from state init -> state preparing >Thread-15914684::DEBUG::2013-11-20 03:37:51,724::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fc6b2417-787c-4309-8b88-d09897c322a6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914684::DEBUG::2013-11-20 03:37:51,725::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914684::DEBUG::2013-11-20 03:37:51,725::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914684::DEBUG::2013-11-20 03:37:51,726::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`fc6b2417-787c-4309-8b88-d09897c322a6`::Granted request >Thread-15914684::DEBUG::2013-11-20 03:37:51,730::task::495::TaskManager.Task::(_debug) Task e5d8a42b-543e-40fc-9739-32136decf7c0: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914684::DEBUG::2013-11-20 03:37:51,733::task::495::TaskManager.Task::(_debug) Task e5d8a42b-543e-40fc-9739-32136decf7c0: ref 1 aborting False >Thread-15914684::WARNING::2013-11-20 03:37:51,735::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914684::DEBUG::2013-11-20 03:37:51,736::task::495::TaskManager.Task::(_debug) Task e5d8a42b-543e-40fc-9739-32136decf7c0: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914684::DEBUG::2013-11-20 03:37:51,736::task::495::TaskManager.Task::(_debug) Task e5d8a42b-543e-40fc-9739-32136decf7c0: moving from state preparing -> state finished >Thread-15914684::DEBUG::2013-11-20 03:37:51,737::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914684::DEBUG::2013-11-20 03:37:51,737::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914684::DEBUG::2013-11-20 03:37:51,738::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914684::DEBUG::2013-11-20 03:37:51,738::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914684::DEBUG::2013-11-20 03:37:51,738::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914684::DEBUG::2013-11-20 03:37:51,739::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914684::DEBUG::2013-11-20 03:37:51,739::task::495::TaskManager.Task::(_debug) Task e5d8a42b-543e-40fc-9739-32136decf7c0: ref 0 aborting False >Thread-15914684::INFO::2013-11-20 03:37:51,740::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:37:51,773::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914685::DEBUG::2013-11-20 03:37:52,053::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15914685::DEBUG::2013-11-20 03:37:54,245::clientIF::1219::vds::(fenceNode) rc 0 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = On >Done > err >Dummy-15912651::DEBUG::2013-11-20 03:37:54,481::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0544322 s, 18.8 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:37:56,492::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:37:56,908::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0519747 s, 19.7 MB/s\n'; <rc> = 0 >Thread-15914688::INFO::2013-11-20 03:37:58,689::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914688::DEBUG::2013-11-20 03:37:58,690::task::495::TaskManager.Task::(_debug) Task 0e035cda-4ebe-4bbd-805c-44f6edf2345e: moving from state init -> state preparing >Thread-15914688::DEBUG::2013-11-20 03:37:58,703::task::495::TaskManager.Task::(_debug) Task 0e035cda-4ebe-4bbd-805c-44f6edf2345e: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00072193145752', 'lastCheck': 1384918672.5010259, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00947713851929', 'lastCheck': 1384918677.6737339, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00101804733276', 'lastCheck': 1384918672.5023389, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00279402732849', 'lastCheck': 1384918676.6652191, 'valid': True, 'code': 0}} >Thread-15914688::DEBUG::2013-11-20 03:37:58,704::task::495::TaskManager.Task::(_debug) Task 0e035cda-4ebe-4bbd-805c-44f6edf2345e: moving from state preparing -> state finished >Thread-15914688::DEBUG::2013-11-20 03:37:58,704::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914688::DEBUG::2013-11-20 03:37:58,704::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914688::DEBUG::2013-11-20 03:37:58,705::task::495::TaskManager.Task::(_debug) Task 0e035cda-4ebe-4bbd-805c-44f6edf2345e: ref 0 aborting False >Thread-15914688::INFO::2013-11-20 03:37:58,705::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.00072193145752', 'lastCheck': 1384918672.5010259, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00947713851929', 'lastCheck': 1384918677.6737339, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00101804733276', 'lastCheck': 1384918672.5023389, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.00279402732849', 'lastCheck': 1384918676.6652191, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:37:58,959::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914690::DEBUG::2013-11-20 03:37:59,732::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Dummy-15912651::DEBUG::2013-11-20 03:37:59,965::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.563991 s, 1.8 MB/s\n'; <rc> = 0 >Thread-15914690::DEBUG::2013-11-20 03:38:00,481::clientIF::1219::vds::(fenceNode) rc 0 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = On >Done > err >Thread-15914691::DEBUG::2013-11-20 03:38:00,489::clientIF::1193::vds::(fenceNode) fenceNode(addr=192.168.100.119,port=,agent=ipmilan,user=Administrator,passwd=XXXX,action=status,secure=,options=power_wait=4 >lanplus=1) >Thread-15914691::DEBUG::2013-11-20 03:38:00,973::clientIF::1219::vds::(fenceNode) rc 0 in agent=fence_ipmilan >ipaddr=192.168.100.119 >login=Administrator >option=status >passwd=XXXX >power_wait=4 >lanplus=1 out Getting status of IPMI:192.168.100.119...Chassis power = On >Done > err >Dummy-15912651::DEBUG::2013-11-20 03:38:01,975::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914693::DEBUG::2013-11-20 03:38:02,251::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914693::INFO::2013-11-20 03:38:02,254::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914693::DEBUG::2013-11-20 03:38:02,254::task::495::TaskManager.Task::(_debug) Task 02a7aea7-1adc-42bd-b821-3fde9e145ef5: moving from state init -> state preparing >Thread-15914693::DEBUG::2013-11-20 03:38:02,255::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914693::DEBUG::2013-11-20 03:38:02,256::task::495::TaskManager.Task::(_debug) Task 02a7aea7-1adc-42bd-b821-3fde9e145ef5: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914693::DEBUG::2013-11-20 03:38:02,256::task::495::TaskManager.Task::(_debug) Task 02a7aea7-1adc-42bd-b821-3fde9e145ef5: moving from state preparing -> state finished >Thread-15914693::DEBUG::2013-11-20 03:38:02,257::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914693::DEBUG::2013-11-20 03:38:02,257::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914693::DEBUG::2013-11-20 03:38:02,258::task::495::TaskManager.Task::(_debug) Task 02a7aea7-1adc-42bd-b821-3fde9e145ef5: ref 0 aborting False >Thread-15914693::INFO::2013-11-20 03:38:02,258::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914694::DEBUG::2013-11-20 03:38:02,273::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914694::INFO::2013-11-20 03:38:02,274::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914694::DEBUG::2013-11-20 03:38:02,274::task::495::TaskManager.Task::(_debug) Task eee8f5e0-ae26-466c-925d-a7549eb7029c: moving from state init -> state preparing >Thread-15914694::DEBUG::2013-11-20 03:38:02,276::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`22499dd0-9abb-4fb7-8889-a7f95c4eae84`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914694::DEBUG::2013-11-20 03:38:02,276::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914694::DEBUG::2013-11-20 03:38:02,277::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914694::DEBUG::2013-11-20 03:38:02,277::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`22499dd0-9abb-4fb7-8889-a7f95c4eae84`::Granted request >Thread-15914694::DEBUG::2013-11-20 03:38:02,278::task::495::TaskManager.Task::(_debug) Task eee8f5e0-ae26-466c-925d-a7549eb7029c: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914694::DEBUG::2013-11-20 03:38:02,283::task::495::TaskManager.Task::(_debug) Task eee8f5e0-ae26-466c-925d-a7549eb7029c: ref 1 aborting False >Thread-15914694::WARNING::2013-11-20 03:38:02,284::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914694::DEBUG::2013-11-20 03:38:02,285::task::495::TaskManager.Task::(_debug) Task eee8f5e0-ae26-466c-925d-a7549eb7029c: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914694::DEBUG::2013-11-20 03:38:02,286::task::495::TaskManager.Task::(_debug) Task eee8f5e0-ae26-466c-925d-a7549eb7029c: moving from state preparing -> state finished >Thread-15914694::DEBUG::2013-11-20 03:38:02,286::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914694::DEBUG::2013-11-20 03:38:02,287::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914694::DEBUG::2013-11-20 03:38:02,287::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914694::DEBUG::2013-11-20 03:38:02,288::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914694::DEBUG::2013-11-20 03:38:02,288::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914694::DEBUG::2013-11-20 03:38:02,288::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914694::DEBUG::2013-11-20 03:38:02,289::task::495::TaskManager.Task::(_debug) Task eee8f5e0-ae26-466c-925d-a7549eb7029c: ref 0 aborting False >Thread-15914694::INFO::2013-11-20 03:38:02,289::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:38:05,209::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 2.26363 s, 452 kB/s\n'; <rc> = 0 >Thread-15913658::INFO::2013-11-20 03:38:07,062::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b74dd33b-748d-48df-83f2-ebcc3523e117 volUUID=8506c15b-61ae-4830-84ed-c830955c25a3) >Thread-15913658::DEBUG::2013-11-20 03:38:07,063::task::495::TaskManager.Task::(_debug) Task d9548475-7c36-4439-9b1e-4b84ef37e3a6: moving from state init -> state preparing >Thread-15913658::DEBUG::2013-11-20 03:38:07,064::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ce556582-d87c-43ed-bfc4-c458fc0bf8ec`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913658::DEBUG::2013-11-20 03:38:07,064::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913658::DEBUG::2013-11-20 03:38:07,065::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913658::DEBUG::2013-11-20 03:38:07,066::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`ce556582-d87c-43ed-bfc4-c458fc0bf8ec`::Granted request >Thread-15913658::DEBUG::2013-11-20 03:38:07,066::task::495::TaskManager.Task::(_debug) Task d9548475-7c36-4439-9b1e-4b84ef37e3a6: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913658::DEBUG::2013-11-20 03:38:07,067::task::495::TaskManager.Task::(_debug) Task d9548475-7c36-4439-9b1e-4b84ef37e3a6: ref 1 aborting False >Thread-15913658::DEBUG::2013-11-20 03:38:07,067::task::495::TaskManager.Task::(_debug) Task d9548475-7c36-4439-9b1e-4b84ef37e3a6: finished: {'truesize': '85899345920', 'apparentsize': '85899345920'} >Thread-15913658::DEBUG::2013-11-20 03:38:07,068::task::495::TaskManager.Task::(_debug) Task d9548475-7c36-4439-9b1e-4b84ef37e3a6: moving from state preparing -> state finished >Thread-15913658::DEBUG::2013-11-20 03:38:07,068::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913658::DEBUG::2013-11-20 03:38:07,068::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913658::DEBUG::2013-11-20 03:38:07,069::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913658::DEBUG::2013-11-20 03:38:07,069::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913658::DEBUG::2013-11-20 03:38:07,070::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913658::DEBUG::2013-11-20 03:38:07,070::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913658::DEBUG::2013-11-20 03:38:07,071::task::495::TaskManager.Task::(_debug) Task d9548475-7c36-4439-9b1e-4b84ef37e3a6: ref 0 aborting False >Thread-15913658::INFO::2013-11-20 03:38:07,071::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '85899345920', 'apparentsize': '85899345920'} >Dummy-15912651::DEBUG::2013-11-20 03:38:07,219::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:38:07,634::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0474537 s, 21.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:38:09,644::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:38:10,565::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0498078 s, 20.6 MB/s\n'; <rc> = 0 >Thread-15912392::INFO::2013-11-20 03:38:11,028::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d8be6e76-2f7f-4b56-9ada-c3fd543a9794 volUUID=038d75ec-9421-4d79-9865-0dbebd5271d9) >Thread-15912392::DEBUG::2013-11-20 03:38:11,029::task::495::TaskManager.Task::(_debug) Task 97aa7a64-2a88-4680-b7c8-a220e8912a93: moving from state init -> state preparing >Thread-15912392::DEBUG::2013-11-20 03:38:11,030::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e9c6f406-75e7-400c-89f8-5477845e3f37`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15912392::DEBUG::2013-11-20 03:38:11,030::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15912392::DEBUG::2013-11-20 03:38:11,031::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15912392::DEBUG::2013-11-20 03:38:11,032::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`e9c6f406-75e7-400c-89f8-5477845e3f37`::Granted request >Thread-15912392::DEBUG::2013-11-20 03:38:11,032::task::495::TaskManager.Task::(_debug) Task 97aa7a64-2a88-4680-b7c8-a220e8912a93: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15912392::DEBUG::2013-11-20 03:38:11,033::task::495::TaskManager.Task::(_debug) Task 97aa7a64-2a88-4680-b7c8-a220e8912a93: ref 1 aborting False >Thread-15912392::DEBUG::2013-11-20 03:38:11,034::task::495::TaskManager.Task::(_debug) Task 97aa7a64-2a88-4680-b7c8-a220e8912a93: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15912392::DEBUG::2013-11-20 03:38:11,034::task::495::TaskManager.Task::(_debug) Task 97aa7a64-2a88-4680-b7c8-a220e8912a93: moving from state preparing -> state finished >Thread-15912392::DEBUG::2013-11-20 03:38:11,034::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15912392::DEBUG::2013-11-20 03:38:11,035::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15912392::DEBUG::2013-11-20 03:38:11,035::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15912392::DEBUG::2013-11-20 03:38:11,036::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15912392::DEBUG::2013-11-20 03:38:11,036::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15912392::DEBUG::2013-11-20 03:38:11,037::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15912392::DEBUG::2013-11-20 03:38:11,037::task::495::TaskManager.Task::(_debug) Task 97aa7a64-2a88-4680-b7c8-a220e8912a93: ref 0 aborting False >Thread-15912392::INFO::2013-11-20 03:38:11,038::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Dummy-15912651::DEBUG::2013-11-20 03:38:12,575::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914698::DEBUG::2013-11-20 03:38:12,794::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914698::INFO::2013-11-20 03:38:12,796::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914698::DEBUG::2013-11-20 03:38:12,797::task::495::TaskManager.Task::(_debug) Task da789efb-02d0-4125-9953-79ac1de9705c: moving from state init -> state preparing >Thread-15914698::DEBUG::2013-11-20 03:38:12,798::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914698::DEBUG::2013-11-20 03:38:12,798::task::495::TaskManager.Task::(_debug) Task da789efb-02d0-4125-9953-79ac1de9705c: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914698::DEBUG::2013-11-20 03:38:12,799::task::495::TaskManager.Task::(_debug) Task da789efb-02d0-4125-9953-79ac1de9705c: moving from state preparing -> state finished >Thread-15914698::DEBUG::2013-11-20 03:38:12,799::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914698::DEBUG::2013-11-20 03:38:12,800::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914698::DEBUG::2013-11-20 03:38:12,800::task::495::TaskManager.Task::(_debug) Task da789efb-02d0-4125-9953-79ac1de9705c: ref 0 aborting False >Thread-15914698::INFO::2013-11-20 03:38:12,801::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914699::DEBUG::2013-11-20 03:38:12,807::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914699::INFO::2013-11-20 03:38:12,808::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914699::DEBUG::2013-11-20 03:38:12,808::task::495::TaskManager.Task::(_debug) Task dfb99a07-6d7f-4bfe-8977-62a2c675cd74: moving from state init -> state preparing >Thread-15914699::DEBUG::2013-11-20 03:38:12,810::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`13c44c6c-6f6d-4025-89a6-eef2b1538f76`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914699::DEBUG::2013-11-20 03:38:12,810::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914699::DEBUG::2013-11-20 03:38:12,811::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914699::DEBUG::2013-11-20 03:38:12,812::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`13c44c6c-6f6d-4025-89a6-eef2b1538f76`::Granted request >Thread-15914699::DEBUG::2013-11-20 03:38:12,813::task::495::TaskManager.Task::(_debug) Task dfb99a07-6d7f-4bfe-8977-62a2c675cd74: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914699::DEBUG::2013-11-20 03:38:12,813::task::495::TaskManager.Task::(_debug) Task dfb99a07-6d7f-4bfe-8977-62a2c675cd74: ref 1 aborting False >Thread-15914699::WARNING::2013-11-20 03:38:12,815::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914699::DEBUG::2013-11-20 03:38:12,816::task::495::TaskManager.Task::(_debug) Task dfb99a07-6d7f-4bfe-8977-62a2c675cd74: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914699::DEBUG::2013-11-20 03:38:12,816::task::495::TaskManager.Task::(_debug) Task dfb99a07-6d7f-4bfe-8977-62a2c675cd74: moving from state preparing -> state finished >Thread-15914699::DEBUG::2013-11-20 03:38:12,817::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914699::DEBUG::2013-11-20 03:38:12,817::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914699::DEBUG::2013-11-20 03:38:12,818::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914699::DEBUG::2013-11-20 03:38:12,818::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914699::DEBUG::2013-11-20 03:38:12,819::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914699::DEBUG::2013-11-20 03:38:12,819::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914699::DEBUG::2013-11-20 03:38:12,819::task::495::TaskManager.Task::(_debug) Task dfb99a07-6d7f-4bfe-8977-62a2c675cd74: ref 0 aborting False >Thread-15914699::INFO::2013-11-20 03:38:12,820::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:38:13,036::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0485796 s, 21.1 MB/s\n'; <rc> = 0 >Thread-15913695::INFO::2013-11-20 03:38:13,147::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=cc863eeb-33db-4227-b4eb-9f4347299cf3 volUUID=ff5e3845-0d0d-4037-a0de-a3f584a4aef5) >Thread-15913695::DEBUG::2013-11-20 03:38:13,148::task::495::TaskManager.Task::(_debug) Task 126e657f-156b-4b85-9897-28cf2e09e43c: moving from state init -> state preparing >Thread-15913695::DEBUG::2013-11-20 03:38:13,149::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`9b17a1e2-a8e9-45f9-8e40-9fdc47f37fe0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913695::DEBUG::2013-11-20 03:38:13,149::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913695::DEBUG::2013-11-20 03:38:13,150::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913695::DEBUG::2013-11-20 03:38:13,150::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`9b17a1e2-a8e9-45f9-8e40-9fdc47f37fe0`::Granted request >Thread-15913695::DEBUG::2013-11-20 03:38:13,151::task::495::TaskManager.Task::(_debug) Task 126e657f-156b-4b85-9897-28cf2e09e43c: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913695::DEBUG::2013-11-20 03:38:13,151::task::495::TaskManager.Task::(_debug) Task 126e657f-156b-4b85-9897-28cf2e09e43c: ref 1 aborting False >Thread-15913695::DEBUG::2013-11-20 03:38:13,151::task::495::TaskManager.Task::(_debug) Task 126e657f-156b-4b85-9897-28cf2e09e43c: finished: {'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15913695::DEBUG::2013-11-20 03:38:13,152::task::495::TaskManager.Task::(_debug) Task 126e657f-156b-4b85-9897-28cf2e09e43c: moving from state preparing -> state finished >Thread-15913695::DEBUG::2013-11-20 03:38:13,152::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913695::DEBUG::2013-11-20 03:38:13,152::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913695::DEBUG::2013-11-20 03:38:13,153::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913695::DEBUG::2013-11-20 03:38:13,153::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913695::DEBUG::2013-11-20 03:38:13,154::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913695::DEBUG::2013-11-20 03:38:13,154::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913695::DEBUG::2013-11-20 03:38:13,154::task::495::TaskManager.Task::(_debug) Task 126e657f-156b-4b85-9897-28cf2e09e43c: ref 0 aborting False >Thread-15913695::INFO::2013-11-20 03:38:13,155::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '20401094656', 'apparentsize': '20401094656'} >Thread-15914700::INFO::2013-11-20 03:38:14,287::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914700::DEBUG::2013-11-20 03:38:14,287::task::495::TaskManager.Task::(_debug) Task 41002617-db92-4684-9272-4f16f9396d3d: moving from state init -> state preparing >Thread-15914700::DEBUG::2013-11-20 03:38:14,288::task::495::TaskManager.Task::(_debug) Task 41002617-db92-4684-9272-4f16f9396d3d: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000248908996582', 'lastCheck': 1384918694.0153489, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000996112823486', 'lastCheck': 1384918689.0189869, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00234699249268', 'lastCheck': 1384918685.1095481, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000396966934204', 'lastCheck': 1384918688.007751, 'valid': True, 'code': 0}} >Thread-15914700::DEBUG::2013-11-20 03:38:14,289::task::495::TaskManager.Task::(_debug) Task 41002617-db92-4684-9272-4f16f9396d3d: moving from state preparing -> state finished >Thread-15914700::DEBUG::2013-11-20 03:38:14,289::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914700::DEBUG::2013-11-20 03:38:14,289::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914700::DEBUG::2013-11-20 03:38:14,290::task::495::TaskManager.Task::(_debug) Task 41002617-db92-4684-9272-4f16f9396d3d: ref 0 aborting False >Thread-15914700::INFO::2013-11-20 03:38:14,290::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000248908996582', 'lastCheck': 1384918694.0153489, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.000996112823486', 'lastCheck': 1384918689.0189869, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00234699249268', 'lastCheck': 1384918685.1095481, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000396966934204', 'lastCheck': 1384918688.007751, 'valid': True, 'code': 0}} >Dummy-15912651::DEBUG::2013-11-20 03:38:15,047::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:38:15,560::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0727395 s, 14.1 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:38:17,573::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:38:18,775::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0482866 s, 21.2 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:38:20,786::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:38:22,561::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0631438 s, 16.2 MB/s\n'; <rc> = 0 >Thread-15914435::INFO::2013-11-20 03:38:23,297::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=143cb26d-de08-4d30-89cf-aee10245b66a volUUID=d7028f7c-0551-4bbe-a907-6f804cc9bd54) >Thread-15914435::DEBUG::2013-11-20 03:38:23,298::task::495::TaskManager.Task::(_debug) Task bc80af7a-ee68-4d42-9058-db6c05b3da83: moving from state init -> state preparing >Thread-15914435::DEBUG::2013-11-20 03:38:23,300::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f473ba6e-59eb-4bf8-a23c-b6834065bcf1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914435::DEBUG::2013-11-20 03:38:23,301::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914435::DEBUG::2013-11-20 03:38:23,302::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914435::DEBUG::2013-11-20 03:38:23,303::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`f473ba6e-59eb-4bf8-a23c-b6834065bcf1`::Granted request >Thread-15914435::DEBUG::2013-11-20 03:38:23,304::task::495::TaskManager.Task::(_debug) Task bc80af7a-ee68-4d42-9058-db6c05b3da83: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914435::DEBUG::2013-11-20 03:38:23,305::task::495::TaskManager.Task::(_debug) Task bc80af7a-ee68-4d42-9058-db6c05b3da83: ref 1 aborting False >Thread-15914435::DEBUG::2013-11-20 03:38:23,307::task::495::TaskManager.Task::(_debug) Task bc80af7a-ee68-4d42-9058-db6c05b3da83: finished: {'truesize': '56908316672', 'apparentsize': '56908316672'} >Thread-15914704::DEBUG::2013-11-20 03:38:23,308::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914435::DEBUG::2013-11-20 03:38:23,309::task::495::TaskManager.Task::(_debug) Task bc80af7a-ee68-4d42-9058-db6c05b3da83: moving from state preparing -> state finished >Thread-15914704::INFO::2013-11-20 03:38:23,310::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914435::DEBUG::2013-11-20 03:38:23,310::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914704::DEBUG::2013-11-20 03:38:23,311::task::495::TaskManager.Task::(_debug) Task e280666f-8b11-4129-aed5-20edc6dcc35a: moving from state init -> state preparing >Thread-15914435::DEBUG::2013-11-20 03:38:23,312::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914704::DEBUG::2013-11-20 03:38:23,312::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914435::DEBUG::2013-11-20 03:38:23,314::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914704::DEBUG::2013-11-20 03:38:23,314::task::495::TaskManager.Task::(_debug) Task e280666f-8b11-4129-aed5-20edc6dcc35a: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914704::DEBUG::2013-11-20 03:38:23,315::task::495::TaskManager.Task::(_debug) Task e280666f-8b11-4129-aed5-20edc6dcc35a: moving from state preparing -> state finished >Thread-15914435::DEBUG::2013-11-20 03:38:23,316::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914704::DEBUG::2013-11-20 03:38:23,316::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914435::DEBUG::2013-11-20 03:38:23,317::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914704::DEBUG::2013-11-20 03:38:23,317::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914435::DEBUG::2013-11-20 03:38:23,318::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914704::DEBUG::2013-11-20 03:38:23,319::task::495::TaskManager.Task::(_debug) Task e280666f-8b11-4129-aed5-20edc6dcc35a: ref 0 aborting False >Thread-15914435::DEBUG::2013-11-20 03:38:23,320::task::495::TaskManager.Task::(_debug) Task bc80af7a-ee68-4d42-9058-db6c05b3da83: ref 0 aborting False >Thread-15914704::INFO::2013-11-20 03:38:23,320::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914435::INFO::2013-11-20 03:38:23,321::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '56908316672', 'apparentsize': '56908316672'} >Thread-15914705::DEBUG::2013-11-20 03:38:23,329::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914705::INFO::2013-11-20 03:38:23,330::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914705::DEBUG::2013-11-20 03:38:23,330::task::495::TaskManager.Task::(_debug) Task c6078c9a-9a22-408f-b1f9-1a1c3f23636a: moving from state init -> state preparing >Thread-15914705::DEBUG::2013-11-20 03:38:23,331::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2af81856-186e-4dee-bd46-d6dca27c12a6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914705::DEBUG::2013-11-20 03:38:23,332::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914705::DEBUG::2013-11-20 03:38:23,332::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914705::DEBUG::2013-11-20 03:38:23,332::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`2af81856-186e-4dee-bd46-d6dca27c12a6`::Granted request >Thread-15914705::DEBUG::2013-11-20 03:38:23,333::task::495::TaskManager.Task::(_debug) Task c6078c9a-9a22-408f-b1f9-1a1c3f23636a: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914705::DEBUG::2013-11-20 03:38:23,333::task::495::TaskManager.Task::(_debug) Task c6078c9a-9a22-408f-b1f9-1a1c3f23636a: ref 1 aborting False >Thread-15914705::WARNING::2013-11-20 03:38:23,335::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914705::DEBUG::2013-11-20 03:38:23,335::task::495::TaskManager.Task::(_debug) Task c6078c9a-9a22-408f-b1f9-1a1c3f23636a: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914705::DEBUG::2013-11-20 03:38:23,336::task::495::TaskManager.Task::(_debug) Task c6078c9a-9a22-408f-b1f9-1a1c3f23636a: moving from state preparing -> state finished >Thread-15914705::DEBUG::2013-11-20 03:38:23,336::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914705::DEBUG::2013-11-20 03:38:23,337::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914705::DEBUG::2013-11-20 03:38:23,337::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914705::DEBUG::2013-11-20 03:38:23,338::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914705::DEBUG::2013-11-20 03:38:23,338::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914705::DEBUG::2013-11-20 03:38:23,338::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914705::DEBUG::2013-11-20 03:38:23,339::task::495::TaskManager.Task::(_debug) Task c6078c9a-9a22-408f-b1f9-1a1c3f23636a: ref 0 aborting False >Thread-15914705::INFO::2013-11-20 03:38:23,339::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:38:24,570::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:38:24,976::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0468105 s, 21.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:38:26,984::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Thread-15914453::INFO::2013-11-20 03:38:27,143::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=5f930ef3-763f-4294-a62a-6d52df220f8c volUUID=b9cffdbe-152e-41ce-ae5d-08db7c08454e) >Thread-15914453::DEBUG::2013-11-20 03:38:27,145::task::495::TaskManager.Task::(_debug) Task 108e17bb-a003-478a-b1da-ebcba5176179: moving from state init -> state preparing >Thread-15914453::DEBUG::2013-11-20 03:38:27,147::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`1ef6eb4a-f5f0-4e4d-a6c1-e04ddaf4438a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914453::DEBUG::2013-11-20 03:38:27,147::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914453::DEBUG::2013-11-20 03:38:27,148::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914453::DEBUG::2013-11-20 03:38:27,148::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`1ef6eb4a-f5f0-4e4d-a6c1-e04ddaf4438a`::Granted request >Thread-15914453::DEBUG::2013-11-20 03:38:27,149::task::495::TaskManager.Task::(_debug) Task 108e17bb-a003-478a-b1da-ebcba5176179: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914453::DEBUG::2013-11-20 03:38:27,150::task::495::TaskManager.Task::(_debug) Task 108e17bb-a003-478a-b1da-ebcba5176179: ref 1 aborting False >Thread-15914453::DEBUG::2013-11-20 03:38:27,150::task::495::TaskManager.Task::(_debug) Task 108e17bb-a003-478a-b1da-ebcba5176179: finished: {'truesize': '53821308928', 'apparentsize': '53821308928'} >Thread-15914453::DEBUG::2013-11-20 03:38:27,151::task::495::TaskManager.Task::(_debug) Task 108e17bb-a003-478a-b1da-ebcba5176179: moving from state preparing -> state finished >Thread-15914453::DEBUG::2013-11-20 03:38:27,151::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914453::DEBUG::2013-11-20 03:38:27,152::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914453::DEBUG::2013-11-20 03:38:27,152::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914453::DEBUG::2013-11-20 03:38:27,152::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914453::DEBUG::2013-11-20 03:38:27,153::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914453::DEBUG::2013-11-20 03:38:27,153::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914453::DEBUG::2013-11-20 03:38:27,154::task::495::TaskManager.Task::(_debug) Task 108e17bb-a003-478a-b1da-ebcba5176179: ref 0 aborting False >Thread-15914453::INFO::2013-11-20 03:38:27,154::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '53821308928', 'apparentsize': '53821308928'} >Dummy-15912651::DEBUG::2013-11-20 03:38:27,427::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0710322 s, 14.4 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:38:29,448::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:38:29,998::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0867894 s, 11.8 MB/s\n'; <rc> = 0 >Thread-15914708::INFO::2013-11-20 03:38:30,243::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, args: () >Thread-15914708::DEBUG::2013-11-20 03:38:30,244::task::495::TaskManager.Task::(_debug) Task 721bc26a-517c-4bda-9fde-4f2ba425f5e3: moving from state init -> state preparing >Thread-15914708::DEBUG::2013-11-20 03:38:30,245::task::495::TaskManager.Task::(_debug) Task 721bc26a-517c-4bda-9fde-4f2ba425f5e3: finished: {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000270128250122', 'lastCheck': 1384918704.0276289, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00129294395447', 'lastCheck': 1384918709.153007, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00109100341797', 'lastCheck': 1384918705.285018, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000608205795288', 'lastCheck': 1384918708.142421, 'valid': True, 'code': 0}} >Thread-15914708::DEBUG::2013-11-20 03:38:30,246::task::495::TaskManager.Task::(_debug) Task 721bc26a-517c-4bda-9fde-4f2ba425f5e3: moving from state preparing -> state finished >Thread-15914708::DEBUG::2013-11-20 03:38:30,247::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914708::DEBUG::2013-11-20 03:38:30,247::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914708::DEBUG::2013-11-20 03:38:30,248::task::495::TaskManager.Task::(_debug) Task 721bc26a-517c-4bda-9fde-4f2ba425f5e3: ref 0 aborting False >Thread-15914708::INFO::2013-11-20 03:38:30,249::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: repoStats, Return response: {'status': {'message': 'OK', 'code': 0}, u'6b892067-ac7e-4039-a7be-5b7387525b16': {'delay': '0.000270128250122', 'lastCheck': 1384918704.0276289, 'valid': True, 'code': 0}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'delay': '0.00129294395447', 'lastCheck': 1384918709.153007, 'valid': True, 'code': 0}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'delay': '0.00109100341797', 'lastCheck': 1384918705.285018, 'valid': True, 'code': 0}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'delay': '0.000608205795288', 'lastCheck': 1384918708.142421, 'valid': True, 'code': 0}} >Thread-15914519::INFO::2013-11-20 03:38:30,281::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=5af747d8-95b1-4b7e-af53-cb77e4fcc896 volUUID=8d8f8dca-fbb0-4f15-9a32-28fc0176bd7f) >Thread-15914519::DEBUG::2013-11-20 03:38:30,283::task::495::TaskManager.Task::(_debug) Task 7643b6ab-c4fb-41f6-baab-b993f1ed5da7: moving from state init -> state preparing >Thread-15914519::DEBUG::2013-11-20 03:38:30,290::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`fbe5869d-12b4-4c6d-8dd8-54f68c56a4d3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15914519::DEBUG::2013-11-20 03:38:30,291::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15914519::DEBUG::2013-11-20 03:38:30,292::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15914519::DEBUG::2013-11-20 03:38:30,292::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`fbe5869d-12b4-4c6d-8dd8-54f68c56a4d3`::Granted request >Thread-15914519::DEBUG::2013-11-20 03:38:30,294::task::495::TaskManager.Task::(_debug) Task 7643b6ab-c4fb-41f6-baab-b993f1ed5da7: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15914519::DEBUG::2013-11-20 03:38:30,294::task::495::TaskManager.Task::(_debug) Task 7643b6ab-c4fb-41f6-baab-b993f1ed5da7: ref 1 aborting False >Thread-15914519::DEBUG::2013-11-20 03:38:30,295::task::495::TaskManager.Task::(_debug) Task 7643b6ab-c4fb-41f6-baab-b993f1ed5da7: finished: {'truesize': '86033563648', 'apparentsize': '86033563648'} >Thread-15914519::DEBUG::2013-11-20 03:38:30,296::task::495::TaskManager.Task::(_debug) Task 7643b6ab-c4fb-41f6-baab-b993f1ed5da7: moving from state preparing -> state finished >Thread-15914519::DEBUG::2013-11-20 03:38:30,297::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15914519::DEBUG::2013-11-20 03:38:30,298::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914519::DEBUG::2013-11-20 03:38:30,298::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15914519::DEBUG::2013-11-20 03:38:30,299::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15914519::DEBUG::2013-11-20 03:38:30,300::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15914519::DEBUG::2013-11-20 03:38:30,301::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15914519::DEBUG::2013-11-20 03:38:30,301::task::495::TaskManager.Task::(_debug) Task 7643b6ab-c4fb-41f6-baab-b993f1ed5da7: ref 0 aborting False >Thread-15914519::INFO::2013-11-20 03:38:30,302::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '86033563648', 'apparentsize': '86033563648'} >Dummy-15912651::DEBUG::2013-11-20 03:38:32,013::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:38:32,593::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0868053 s, 11.8 MB/s\n'; <rc> = 0 >Thread-15913518::INFO::2013-11-20 03:38:33,461::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=b0980185-7de1-4e3b-bd76-4495169ab08b volUUID=0ecd1db7-48b0-4663-a038-e40e891b9424) >Thread-15913518::DEBUG::2013-11-20 03:38:33,462::task::495::TaskManager.Task::(_debug) Task b2be4377-b186-4fa1-92d7-4661f07e444e: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:38:33,464::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`07c826c4-40e6-47f9-ad47-b8572f43a1c7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:38:33,465::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:38:33,466::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:38:33,466::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`07c826c4-40e6-47f9-ad47-b8572f43a1c7`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:38:33,468::task::495::TaskManager.Task::(_debug) Task b2be4377-b186-4fa1-92d7-4661f07e444e: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:38:33,469::task::495::TaskManager.Task::(_debug) Task b2be4377-b186-4fa1-92d7-4661f07e444e: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:38:33,470::task::495::TaskManager.Task::(_debug) Task b2be4377-b186-4fa1-92d7-4661f07e444e: finished: {'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::DEBUG::2013-11-20 03:38:33,470::task::495::TaskManager.Task::(_debug) Task b2be4377-b186-4fa1-92d7-4661f07e444e: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:38:33,471::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:38:33,472::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:38:33,473::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:38:33,474::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:38:33,474::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:38:33,475::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:38:33,475::task::495::TaskManager.Task::(_debug) Task b2be4377-b186-4fa1-92d7-4661f07e444e: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:38:33,476::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '30064771072', 'apparentsize': '30064771072'} >Thread-15913518::INFO::2013-11-20 03:38:33,477::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=6b892067-ac7e-4039-a7be-5b7387525b16 spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=0b89347c-4654-42bb-a8d7-1a627130b7bd volUUID=6d82661d-9502-4cb3-a0a0-91cd82dd2a80) >Thread-15913518::DEBUG::2013-11-20 03:38:33,477::task::495::TaskManager.Task::(_debug) Task bf1e648f-039d-4220-a93e-3264fb943f0f: moving from state init -> state preparing >Thread-15913518::DEBUG::2013-11-20 03:38:33,478::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`45a0219b-dd21-4565-8b84-dbaa5d84caf3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913518::DEBUG::2013-11-20 03:38:33,479::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' for lock type 'shared' >Thread-15913518::DEBUG::2013-11-20 03:38:33,480::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free. Now locking as 'shared' (1 active user) >Thread-15913518::DEBUG::2013-11-20 03:38:33,480::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.6b892067-ac7e-4039-a7be-5b7387525b16`ReqID=`45a0219b-dd21-4565-8b84-dbaa5d84caf3`::Granted request >Thread-15913518::DEBUG::2013-11-20 03:38:33,481::task::495::TaskManager.Task::(_debug) Task bf1e648f-039d-4220-a93e-3264fb943f0f: _resourcesAcquired: Storage.6b892067-ac7e-4039-a7be-5b7387525b16 (shared) >Thread-15913518::DEBUG::2013-11-20 03:38:33,482::task::495::TaskManager.Task::(_debug) Task bf1e648f-039d-4220-a93e-3264fb943f0f: ref 1 aborting False >Thread-15913518::DEBUG::2013-11-20 03:38:33,482::task::495::TaskManager.Task::(_debug) Task bf1e648f-039d-4220-a93e-3264fb943f0f: finished: {'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15913518::DEBUG::2013-11-20 03:38:33,483::task::495::TaskManager.Task::(_debug) Task bf1e648f-039d-4220-a93e-3264fb943f0f: moving from state preparing -> state finished >Thread-15913518::DEBUG::2013-11-20 03:38:33,483::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.6b892067-ac7e-4039-a7be-5b7387525b16': < ResourceRef 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', isValid: 'True' obj: 'None'>} >Thread-15913518::DEBUG::2013-11-20 03:38:33,484::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913518::DEBUG::2013-11-20 03:38:33,485::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' >Thread-15913518::DEBUG::2013-11-20 03:38:33,486::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' (0 active users) >Thread-15913518::DEBUG::2013-11-20 03:38:33,486::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16' is free, finding out if anyone is waiting for it. >Thread-15913518::DEBUG::2013-11-20 03:38:33,487::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.6b892067-ac7e-4039-a7be-5b7387525b16', Clearing records. >Thread-15913518::DEBUG::2013-11-20 03:38:33,487::task::495::TaskManager.Task::(_debug) Task bf1e648f-039d-4220-a93e-3264fb943f0f: ref 0 aborting False >Thread-15913518::INFO::2013-11-20 03:38:33,488::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '826781204480', 'apparentsize': '826781204480'} >Thread-15914710::DEBUG::2013-11-20 03:38:33,760::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914710::INFO::2013-11-20 03:38:33,761::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914710::DEBUG::2013-11-20 03:38:33,762::task::495::TaskManager.Task::(_debug) Task d71e556d-e47e-460d-be8c-32b5dfd9362b: moving from state init -> state preparing >Thread-15914710::DEBUG::2013-11-20 03:38:33,763::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914710::DEBUG::2013-11-20 03:38:33,764::task::495::TaskManager.Task::(_debug) Task d71e556d-e47e-460d-be8c-32b5dfd9362b: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914710::DEBUG::2013-11-20 03:38:33,764::task::495::TaskManager.Task::(_debug) Task d71e556d-e47e-460d-be8c-32b5dfd9362b: moving from state preparing -> state finished >Thread-15914710::DEBUG::2013-11-20 03:38:33,765::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914710::DEBUG::2013-11-20 03:38:33,766::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914710::DEBUG::2013-11-20 03:38:33,767::task::495::TaskManager.Task::(_debug) Task d71e556d-e47e-460d-be8c-32b5dfd9362b: ref 0 aborting False >Thread-15914710::INFO::2013-11-20 03:38:33,767::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914711::DEBUG::2013-11-20 03:38:33,782::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914711::INFO::2013-11-20 03:38:33,783::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914711::DEBUG::2013-11-20 03:38:33,784::task::495::TaskManager.Task::(_debug) Task a0e12ebd-6f4b-46c2-a27b-6f0d244454a4: moving from state init -> state preparing >Thread-15914711::DEBUG::2013-11-20 03:38:33,785::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`558cbd46-a7cc-4f8f-b046-84331b37d342`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914711::DEBUG::2013-11-20 03:38:33,786::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914711::DEBUG::2013-11-20 03:38:33,786::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914711::DEBUG::2013-11-20 03:38:33,787::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`558cbd46-a7cc-4f8f-b046-84331b37d342`::Granted request >Thread-15914711::DEBUG::2013-11-20 03:38:33,788::task::495::TaskManager.Task::(_debug) Task a0e12ebd-6f4b-46c2-a27b-6f0d244454a4: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914711::DEBUG::2013-11-20 03:38:33,789::task::495::TaskManager.Task::(_debug) Task a0e12ebd-6f4b-46c2-a27b-6f0d244454a4: ref 1 aborting False >Thread-15914711::WARNING::2013-11-20 03:38:33,791::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >Thread-15914711::DEBUG::2013-11-20 03:38:33,792::task::495::TaskManager.Task::(_debug) Task a0e12ebd-6f4b-46c2-a27b-6f0d244454a4: finished: {'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Thread-15914711::DEBUG::2013-11-20 03:38:33,793::task::495::TaskManager.Task::(_debug) Task a0e12ebd-6f4b-46c2-a27b-6f0d244454a4: moving from state preparing -> state finished >Thread-15914711::DEBUG::2013-11-20 03:38:33,793::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30': < ResourceRef 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', isValid: 'True' obj: 'None'>} >Thread-15914711::DEBUG::2013-11-20 03:38:33,794::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914711::DEBUG::2013-11-20 03:38:33,794::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' >Thread-15914711::DEBUG::2013-11-20 03:38:33,795::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' (0 active users) >Thread-15914711::DEBUG::2013-11-20 03:38:33,796::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free, finding out if anyone is waiting for it. >Thread-15914711::DEBUG::2013-11-20 03:38:33,796::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30', Clearing records. >Thread-15914711::DEBUG::2013-11-20 03:38:33,797::task::495::TaskManager.Task::(_debug) Task a0e12ebd-6f4b-46c2-a27b-6f0d244454a4: ref 0 aborting False >Thread-15914711::INFO::2013-11-20 03:38:33,798::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, Return response: {'status': {'message': 'OK', 'code': 0}, 'info': {'spm_id': 2, 'master_uuid': '6b892067-ac7e-4039-a7be-5b7387525b16', 'name': 'Default', 'version': '2', 'domains': u'6b892067-ac7e-4039-a7be-5b7387525b16:Active,45cdff8b-89c0-4926-a8c7-80354064bfa6:Active,772472cb-2429-466d-96cd-00b3301ef78a:Active,5d7b5178-3790-472c-a13b-b481eb27a61e:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/45cdff8b-89c0-4926-a8c7-80354064bfa6/images/11111111-1111-1111-1111-111111111111', 'type': 'FCP', 'master_ver': 1, 'lver': 287}, 'dominfo': {u'6b892067-ac7e-4039-a7be-5b7387525b16': {'status': u'Active', 'diskfree': '1142595518464', 'alerts': [], 'disktotal': '3595827150848'}, u'45cdff8b-89c0-4926-a8c7-80354064bfa6': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}, u'772472cb-2429-466d-96cd-00b3301ef78a': {'status': u'Active', 'diskfree': '442650066944', 'alerts': [], 'disktotal': '1198430093312'}, u'5d7b5178-3790-472c-a13b-b481eb27a61e': {'status': u'Active', 'diskfree': '115248988160', 'alerts': [], 'disktotal': '240964861952'}}} >Dummy-15912651::DEBUG::2013-11-20 03:38:34,604::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:38:35,469::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0951538 s, 10.8 MB/s\n'; <rc> = 0 >Thread-15913546::INFO::2013-11-20 03:38:37,280::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=c4460480-32d3-469f-b0f4-c03bb6d74e41 volUUID=e6984064-e09e-4c0c-8b51-a8a1711994cf) >Thread-15913546::DEBUG::2013-11-20 03:38:37,281::task::495::TaskManager.Task::(_debug) Task fa7784a6-9b95-4c44-9dce-7b7399b64f08: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:38:37,282::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`d1dbcd93-2770-49f0-a59a-2a9803470ec5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:38:37,283::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:38:37,284::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:38:37,285::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`d1dbcd93-2770-49f0-a59a-2a9803470ec5`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:38:37,286::task::495::TaskManager.Task::(_debug) Task fa7784a6-9b95-4c44-9dce-7b7399b64f08: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:38:37,287::task::495::TaskManager.Task::(_debug) Task fa7784a6-9b95-4c44-9dce-7b7399b64f08: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:38:37,288::task::495::TaskManager.Task::(_debug) Task fa7784a6-9b95-4c44-9dce-7b7399b64f08: finished: {'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::DEBUG::2013-11-20 03:38:37,289::task::495::TaskManager.Task::(_debug) Task fa7784a6-9b95-4c44-9dce-7b7399b64f08: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:38:37,290::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:38:37,290::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:38:37,292::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:38:37,293::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:38:37,293::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:38:37,294::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:38:37,295::task::495::TaskManager.Task::(_debug) Task fa7784a6-9b95-4c44-9dce-7b7399b64f08: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:38:37,296::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '64424509440', 'apparentsize': '64424509440'} >Thread-15913546::INFO::2013-11-20 03:38:37,297::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=df8b790c-c762-4feb-9378-d64a321ce43c volUUID=2e903e43-1016-4339-a7f3-c5723a995725) >Thread-15913546::DEBUG::2013-11-20 03:38:37,298::task::495::TaskManager.Task::(_debug) Task 61cb8d8a-795d-4732-9ec7-7f2a414c4ff8: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:38:37,299::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`2b3dc324-fea7-4db9-8855-a6143db04384`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:38:37,300::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:38:37,301::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:38:37,301::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`2b3dc324-fea7-4db9-8855-a6143db04384`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:38:37,302::task::495::TaskManager.Task::(_debug) Task 61cb8d8a-795d-4732-9ec7-7f2a414c4ff8: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:38:37,303::task::495::TaskManager.Task::(_debug) Task 61cb8d8a-795d-4732-9ec7-7f2a414c4ff8: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:38:37,304::task::495::TaskManager.Task::(_debug) Task 61cb8d8a-795d-4732-9ec7-7f2a414c4ff8: finished: {'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::DEBUG::2013-11-20 03:38:37,304::task::495::TaskManager.Task::(_debug) Task 61cb8d8a-795d-4732-9ec7-7f2a414c4ff8: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:38:37,305::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:38:37,305::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:38:37,306::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:38:37,307::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:38:37,307::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:38:37,308::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:38:37,309::task::495::TaskManager.Task::(_debug) Task 61cb8d8a-795d-4732-9ec7-7f2a414c4ff8: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:38:37,309::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '150323855360', 'apparentsize': '150323855360'} >Thread-15913546::INFO::2013-11-20 03:38:37,310::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=13efb45b-f4d2-4c94-bbdb-91ce9b9b7321 volUUID=088640f7-316c-4e2b-bfdc-0f64cf2913c8) >Thread-15913546::DEBUG::2013-11-20 03:38:37,311::task::495::TaskManager.Task::(_debug) Task f1dfd8c4-6fa6-4633-ad05-776620a45f3c: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:38:37,312::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`541a4af3-90fb-4f75-b946-ed760cb10d58`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:38:37,312::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:38:37,313::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:38:37,314::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`541a4af3-90fb-4f75-b946-ed760cb10d58`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:38:37,315::task::495::TaskManager.Task::(_debug) Task f1dfd8c4-6fa6-4633-ad05-776620a45f3c: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:38:37,315::task::495::TaskManager.Task::(_debug) Task f1dfd8c4-6fa6-4633-ad05-776620a45f3c: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:38:37,316::task::495::TaskManager.Task::(_debug) Task f1dfd8c4-6fa6-4633-ad05-776620a45f3c: finished: {'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::DEBUG::2013-11-20 03:38:37,316::task::495::TaskManager.Task::(_debug) Task f1dfd8c4-6fa6-4633-ad05-776620a45f3c: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:38:37,317::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:38:37,318::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:38:37,318::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:38:37,319::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:38:37,319::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:38:37,320::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:38:37,320::task::495::TaskManager.Task::(_debug) Task f1dfd8c4-6fa6-4633-ad05-776620a45f3c: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:38:37,321::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '107374182400', 'apparentsize': '107374182400'} >Thread-15913546::INFO::2013-11-20 03:38:37,322::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, args: ( sdUUID=772472cb-2429-466d-96cd-00b3301ef78a spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30 imgUUID=d368307e-425e-423a-85b7-266c1e4a99ec volUUID=fdc91117-476f-4b82-8fc4-d682fa283ecc) >Thread-15913546::DEBUG::2013-11-20 03:38:37,322::task::495::TaskManager.Task::(_debug) Task f5798a30-6e69-4911-ac4d-72e83af0e37b: moving from state init -> state preparing >Thread-15913546::DEBUG::2013-11-20 03:38:37,323::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`af0cec48-3c28-4140-a131-dbe7a3d41dde`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1615' at 'public_getVolumeSize' >Thread-15913546::DEBUG::2013-11-20 03:38:37,324::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' for lock type 'shared' >Thread-15913546::DEBUG::2013-11-20 03:38:37,324::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free. Now locking as 'shared' (1 active user) >Thread-15913546::DEBUG::2013-11-20 03:38:37,325::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.772472cb-2429-466d-96cd-00b3301ef78a`ReqID=`af0cec48-3c28-4140-a131-dbe7a3d41dde`::Granted request >Thread-15913546::DEBUG::2013-11-20 03:38:37,326::task::495::TaskManager.Task::(_debug) Task f5798a30-6e69-4911-ac4d-72e83af0e37b: _resourcesAcquired: Storage.772472cb-2429-466d-96cd-00b3301ef78a (shared) >Thread-15913546::DEBUG::2013-11-20 03:38:37,326::task::495::TaskManager.Task::(_debug) Task f5798a30-6e69-4911-ac4d-72e83af0e37b: ref 1 aborting False >Thread-15913546::DEBUG::2013-11-20 03:38:37,327::task::495::TaskManager.Task::(_debug) Task f5798a30-6e69-4911-ac4d-72e83af0e37b: finished: {'truesize': '429496729600', 'apparentsize': '429496729600'} >Thread-15913546::DEBUG::2013-11-20 03:38:37,328::task::495::TaskManager.Task::(_debug) Task f5798a30-6e69-4911-ac4d-72e83af0e37b: moving from state preparing -> state finished >Thread-15913546::DEBUG::2013-11-20 03:38:37,328::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.772472cb-2429-466d-96cd-00b3301ef78a': < ResourceRef 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', isValid: 'True' obj: 'None'>} >Thread-15913546::DEBUG::2013-11-20 03:38:37,329::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15913546::DEBUG::2013-11-20 03:38:37,330::resourceManager::515::ResourceManager::(releaseResource) Trying to release resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' >Thread-15913546::DEBUG::2013-11-20 03:38:37,330::resourceManager::530::ResourceManager::(releaseResource) Released resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' (0 active users) >Thread-15913546::DEBUG::2013-11-20 03:38:37,331::resourceManager::535::ResourceManager::(releaseResource) Resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a' is free, finding out if anyone is waiting for it. >Thread-15913546::DEBUG::2013-11-20 03:38:37,331::resourceManager::542::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.772472cb-2429-466d-96cd-00b3301ef78a', Clearing records. >Thread-15913546::DEBUG::2013-11-20 03:38:37,332::task::495::TaskManager.Task::(_debug) Task f5798a30-6e69-4911-ac4d-72e83af0e37b: ref 0 aborting False >Thread-15913546::INFO::2013-11-20 03:38:37,332::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getVolumeSize, Return response: {'status': {'message': 'OK', 'code': 0}, 'truesize': '429496729600', 'apparentsize': '429496729600'} >Dummy-15912651::DEBUG::2013-11-20 03:38:37,486::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:38:38,058::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0881193 s, 11.6 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:38:40,078::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:38:40,618::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0736225 s, 13.9 MB/s\n'; <rc> = 0 >Dummy-15912651::DEBUG::2013-11-20 03:38:42,631::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) 'dd if=/rhev/data-center/c4e5889a-348d-11e2-83fe-9f72369eab30/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None) >Dummy-15912651::DEBUG::2013-11-20 03:38:43,216::storage_mailbox::623::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.098209 s, 10.4 MB/s\n'; <rc> = 0 >Thread-15914716::DEBUG::2013-11-20 03:38:44,335::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914716::INFO::2013-11-20 03:38:44,342::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914716::DEBUG::2013-11-20 03:38:44,344::task::495::TaskManager.Task::(_debug) Task d2a6611e-6a08-40b9-8eeb-fe5213854e05: moving from state init -> state preparing >Thread-15914716::DEBUG::2013-11-20 03:38:44,346::spm::688::Storage.SPM::(public_getSpmStatus) spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30: spmStatus=SPM spmLver=287 spmId=2 >Thread-15914716::DEBUG::2013-11-20 03:38:44,347::task::495::TaskManager.Task::(_debug) Task d2a6611e-6a08-40b9-8eeb-fe5213854e05: finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914716::DEBUG::2013-11-20 03:38:44,350::task::495::TaskManager.Task::(_debug) Task d2a6611e-6a08-40b9-8eeb-fe5213854e05: moving from state preparing -> state finished >Thread-15914716::DEBUG::2013-11-20 03:38:44,352::resourceManager::786::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-15914716::DEBUG::2013-11-20 03:38:44,355::resourceManager::821::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-15914716::DEBUG::2013-11-20 03:38:44,358::task::495::TaskManager.Task::(_debug) Task d2a6611e-6a08-40b9-8eeb-fe5213854e05: ref 0 aborting False >Thread-15914716::INFO::2013-11-20 03:38:44,359::dispatcher::100::Storage.Dispatcher.Protect::(run) Run and protect: getSpmStatus, Return response: {'status': {'message': 'OK', 'code': 0}, 'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 287}} >Thread-15914717::DEBUG::2013-11-20 03:38:44,368::clientIF::239::Storage.Dispatcher.Protect::(wrapper) [192.168.100.135] >Thread-15914717::INFO::2013-11-20 03:38:44,370::dispatcher::94::Storage.Dispatcher.Protect::(run) Run and protect: getStoragePoolInfo, args: ( spUUID=c4e5889a-348d-11e2-83fe-9f72369eab30) >Thread-15914717::DEBUG::2013-11-20 03:38:44,372::task::495::TaskManager.Task::(_debug) Task 9680edfd-8e21-41f6-beca-d68ef4dd433b: moving from state init -> state preparing >Thread-15914717::DEBUG::2013-11-20 03:38:44,375::resourceManager::155::ResourceManager.Request::(__init__) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d2bcbbae-4363-49de-a23b-be71fa143bac`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1162' at 'public_getStoragePoolInfo' >Thread-15914717::DEBUG::2013-11-20 03:38:44,378::resourceManager::463::ResourceManager::(registerResource) Trying to register resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' for lock type 'shared' >Thread-15914717::DEBUG::2013-11-20 03:38:44,383::resourceManager::505::ResourceManager::(registerResource) Resource 'Storage.c4e5889a-348d-11e2-83fe-9f72369eab30' is free. Now locking as 'shared' (1 active user) >Thread-15914717::DEBUG::2013-11-20 03:38:44,384::resourceManager::192::ResourceManager.Request::(grant) ResName=`Storage.c4e5889a-348d-11e2-83fe-9f72369eab30`ReqID=`d2bcbbae-4363-49de-a23b-be71fa143bac`::Granted request >Thread-15914717::DEBUG::2013-11-20 03:38:44,388::task::495::TaskManager.Task::(_debug) Task 9680edfd-8e21-41f6-beca-d68ef4dd433b: _resourcesAcquired: Storage.c4e5889a-348d-11e2-83fe-9f72369eab30 (shared) >Thread-15914717::DEBUG::2013-11-20 03:38:44,391::task::495::TaskManager.Task::(_debug) Task 9680edfd-8e21-41f6-beca-d68ef4dd433b: ref 1 aborting False >Thread-15914717::WARNING::2013-11-20 03:38:44,399::sp::1065::Storage.StoragePool::(getInfo) Could not get full domain information, it is probably unavailable >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sp.py", line 1061, in getInfo > File "/usr/share/vdsm/storage/sd.py", line 663, in isISO > File "/usr/share/vdsm/storage/sd.py", line 620, in getMetaParam > File "/usr/share/vdsm/storage/persistentDict.py", line 63, in __getitem__ > File "/usr/share/vdsm/storage/persistentDict.py", line 173, in __getitem__ >KeyError: 'CLASS' >